Panic Mode On (113) Server Problems?

Message boards : Number crunching : Panic Mode On (113) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 25 · 26 · 27 · 28 · 29 · 30 · 31 . . . 37 · Next

AuthorMessage
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13398
Credit: 208,696,464
RAC: 304
Australia
Message 1962392 - Posted: 29 Oct 2018, 10:48:20 UTC - in response to Message 1962389.  
Last modified: 29 Oct 2018, 10:51:02 UTC

221k now.

Make that 233k.
And the splitter output is diving, along with the Ready-to-send buffer.
If the load doesn't taper off soon, it's all going to come to a grinding halt till the returned work load drops off significantly and the backlogs clear.
A big batch of Arecibo VLARs & AP work would be helpful right about now.

Looking at my transfers the upload server is struggling at times as well.

I've noticed a few downloads taking a while to get going.
Grant
Darwin NT
ID: 1962392 · Report as offensive
Cruncher-American Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor

Send message
Joined: 25 Mar 02
Posts: 1513
Credit: 370,893,186
RAC: 340
United States
Message 1962402 - Posted: 29 Oct 2018, 12:19:22 UTC

Hey - I have noticed many WUs recently that process in < 10 secs. Like dozens or hundreds/day, both on GPU and CPU. Are these the "noisy" WUs referred to above? Are they the variety that has 30 spikes detected and then dropped by the app(s)?
ID: 1962402 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1962414 - Posted: 29 Oct 2018, 14:03:59 UTC

Wowie Zowie !!!

Results Returned in Last Hour has hit 237,593 and the RTS is practically empty (30K). The system is still up though. It is 7am in California, so hopefully soon someone will notice and throw us an Aricebo dataset , if they have one.
ID: 1962414 · Report as offensive
Sirius B Project Donor
Volunteer tester
Avatar

Send message
Joined: 26 Dec 00
Posts: 24613
Credit: 3,081,182
RAC: 7
Ireland
Message 1962417 - Posted: 29 Oct 2018, 14:52:03 UTC - in response to Message 1962170.  

I realised that when Grant mentioned Beta earlier. Been too used to seeing 50 Beta & 100 Main. It seems that AP flood upset the applecart. All the MB will be completed by this evening & the AP by the weekend.
ID: 1962417 · Report as offensive
Cruncher-American Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor

Send message
Joined: 25 Mar 02
Posts: 1513
Credit: 370,893,186
RAC: 340
United States
Message 1962419 - Posted: 29 Oct 2018, 15:14:01 UTC - in response to Message 1962414.  

Results Returned in Last Hour has hit 237,593 and the RTS is practically empty (30K). The system is still up though. It is 7am in California, so hopefully soon someone will notice and throw us an Aricebo dataset , if they have one.


Yeah. This is consistent with my personal surge of 100s of 10 sec WUs (apparently too many spikes) that I have been getting on both GPU and CPU on 2 crunchers (out of 2). So maybe NOT just me...

My pendings and validated have gone way high in the last couple of days. More than doubled.

Watch for many more WUs being sent to users possibly causing congestion on the outgoing data path from SETI????
ID: 1962419 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11172
Credit: 29,581,041
RAC: 66
United States
Message 1962422 - Posted: 29 Oct 2018, 15:36:26 UTC

Results ready to send 0 0 205 0m
This is a problem
ID: 1962422 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1962427 - Posted: 29 Oct 2018, 15:52:04 UTC - in response to Message 1962422.  

Results ready to send 0 0 205 0m
This is a problem


Definitely a problem. On the bright side, the splitters are still splitting, but just aren't keeping up with demand. The big question is how many of the files to split are garbage. Once we work through the junk and get some good WUs then the system can recover and we can all refill our caches. So far the Results out in the Field is falling slowly.
ID: 1962427 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13159
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1962428 - Posted: 29 Oct 2018, 15:52:26 UTC - in response to Message 1962422.  

Just looked at my hosts and found the big iron empty. Checked the RTS and it is empty. Seems like all we've had to crunch for several hours are noise bombs.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1962428 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5516
Credit: 528,817,460
RAC: 242
United States
Message 1962430 - Posted: 29 Oct 2018, 16:01:09 UTC - in response to Message 1962428.  

Just looked at my hosts and found the big iron empty. Checked the RTS and it is empty. Seems like all we've had to crunch for several hours are noise bombs.


GPUs are dry, CPU is happy.... Einstein here come the GPUs...
ID: 1962430 · Report as offensive
Kevin Olley

Send message
Joined: 3 Aug 99
Posts: 906
Credit: 261,085,289
RAC: 572
United Kingdom
Message 1962431 - Posted: 29 Oct 2018, 16:02:21 UTC - in response to Message 1962428.  

Just looked at my hosts and found the big iron empty. Checked the RTS and it is empty. Seems like all we've had to crunch for several hours are noise bombs.


Still got a few on the CPU but the 1080's are now running Einstein, it stops them getting cold;-)
Kevin


ID: 1962431 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13159
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1962446 - Posted: 29 Oct 2018, 17:24:52 UTC

It looks like the elves in the lab loaded some new BLC01 files. Hope the splitters move to them soonest and flush out the noise bombs.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1962446 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1962453 - Posted: 29 Oct 2018, 19:01:49 UTC - in response to Message 1962446.  

It looks like the elves in the lab loaded some new BLC01 files. Hope the splitters move to them soonest and flush out the noise bombs.


and now they have added an Aricebo file 28oc18aa. The creation rate is up to 90/sec and we are still taking them as fast as created.
ID: 1962453 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1962465 - Posted: 29 Oct 2018, 21:09:00 UTC - in response to Message 1962402.  

Hey - I have noticed many WUs recently that process in < 10 secs. Like dozens or hundreds/day, both on GPU and CPU. Are these the "noisy" WUs referred to above? Are they the variety that has 30 spikes detected and then dropped by the app(s)?


. . Yes!

Stephen

.
ID: 1962465 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13159
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1962490 - Posted: 30 Oct 2018, 0:54:28 UTC

Still getting noise bombs.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1962490 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1962499 - Posted: 30 Oct 2018, 1:59:33 UTC - in response to Message 1962490.  

Still getting noise bombs.


. . About 80% to 90% it seems ... need to preserve Arecibo VLARs to have any hope for the outage ...

Stephen

:(
ID: 1962499 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13159
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1962500 - Posted: 30 Oct 2018, 2:01:07 UTC - in response to Message 1962499.  

I'm hoping the BLC23 clear out of the RTS buffer overnight and we can chew on a steady diet of BLC22, BLC01 and Arecibo tasks.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1962500 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1962501 - Posted: 30 Oct 2018, 2:24:44 UTC - in response to Message 1962500.  

I'm hoping the BLC23 clear out of the RTS buffer overnight and we can chew on a steady diet of BLC22, BLC01 and Arecibo tasks.


. . the Blc22 tasks are from the same date/time and are just as noisy, they are all noise bombs. We have to hope there is a good supply of Arecibo tapes to keep the Arecibo VLARs coming to slow things down and let the RTS refill. Hopefully the remaining blc22/blc23 tapes will split and clear before the outage so that maybe the Blc01 tapes can start to come out and are less noise prone.

Stephen

:(
ID: 1962501 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13159
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1962502 - Posted: 30 Oct 2018, 2:41:55 UTC - in response to Message 1962501.  

I'm hoping the BLC23 clear out of the RTS buffer overnight and we can chew on a steady diet of BLC22, BLC01 and Arecibo tasks.


. . the Blc22 tasks are from the same date/time and are just as noisy, they are all noise bombs. We have to hope there is a good supply of Arecibo tapes to keep the Arecibo VLARs coming to slow things down and let the RTS refill. Hopefully the remaining blc22/blc23 tapes will split and clear before the outage so that maybe the Blc01 tapes can start to come out and are less noise prone.

Stephen

:(

I'm not finding many BLC22 noise bombs at all. I'd say 90% are good.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1962502 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11172
Credit: 29,581,041
RAC: 66
United States
Message 1962506 - Posted: 30 Oct 2018, 3:41:51 UTC - in response to Message 1962502.  

Well Green Banks should not have any noise. IMO the noise is probably a hardware issue. We will never know.
ID: 1962506 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13159
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1962509 - Posted: 30 Oct 2018, 3:45:30 UTC - in response to Message 1962506.  
Last modified: 30 Oct 2018, 3:46:50 UTC

Well Green Banks should not have any noise. IMO the noise is probably a hardware issue. We will never know.

The Arecibo tasks almost always have noise. It's in the form of the massive radar pulses from the dish when active. At least they get blanked automatically and we never see them except for the AP tasks.

Agree for Green Bank being in a noise free zone, it must have been hardware failure of some kind.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1962509 · Report as offensive
Previous · 1 . . . 25 · 26 · 27 · 28 · 29 · 30 · 31 . . . 37 · Next

Message boards : Number crunching : Panic Mode On (113) Server Problems?


 
©2022 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.