Panic Mode On (113) Server Problems?

Message boards : Number crunching : Panic Mode On (113) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 25 · 26 · 27 · 28 · 29 · 30 · 31 . . . 37 · Next

AuthorMessage
Sirius B Project Donor
Volunteer tester
Avatar

Send message
Joined: 26 Dec 00
Posts: 24631
Credit: 3,081,182
RAC: 7
Ireland
Message 1962417 - Posted: 29 Oct 2018, 14:52:03 UTC - in response to Message 1962170.  

I realised that when Grant mentioned Beta earlier. Been too used to seeing 50 Beta & 100 Main. It seems that AP flood upset the applecart. All the MB will be completed by this evening & the AP by the weekend.
ID: 1962417 · Report as offensive
Cruncher-American Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor

Send message
Joined: 25 Mar 02
Posts: 1513
Credit: 370,893,186
RAC: 340
United States
Message 1962419 - Posted: 29 Oct 2018, 15:14:01 UTC - in response to Message 1962414.  

Results Returned in Last Hour has hit 237,593 and the RTS is practically empty (30K). The system is still up though. It is 7am in California, so hopefully soon someone will notice and throw us an Aricebo dataset , if they have one.


Yeah. This is consistent with my personal surge of 100s of 10 sec WUs (apparently too many spikes) that I have been getting on both GPU and CPU on 2 crunchers (out of 2). So maybe NOT just me...

My pendings and validated have gone way high in the last couple of days. More than doubled.

Watch for many more WUs being sent to users possibly causing congestion on the outgoing data path from SETI????
ID: 1962419 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11191
Credit: 29,581,041
RAC: 66
United States
Message 1962422 - Posted: 29 Oct 2018, 15:36:26 UTC

Results ready to send 0 0 205 0m
This is a problem
ID: 1962422 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1962427 - Posted: 29 Oct 2018, 15:52:04 UTC - in response to Message 1962422.  

Results ready to send 0 0 205 0m
This is a problem


Definitely a problem. On the bright side, the splitters are still splitting, but just aren't keeping up with demand. The big question is how many of the files to split are garbage. Once we work through the junk and get some good WUs then the system can recover and we can all refill our caches. So far the Results out in the Field is falling slowly.
ID: 1962427 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13159
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1962428 - Posted: 29 Oct 2018, 15:52:26 UTC - in response to Message 1962422.  

Just looked at my hosts and found the big iron empty. Checked the RTS and it is empty. Seems like all we've had to crunch for several hours are noise bombs.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1962428 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5516
Credit: 528,817,460
RAC: 242
United States
Message 1962430 - Posted: 29 Oct 2018, 16:01:09 UTC - in response to Message 1962428.  

Just looked at my hosts and found the big iron empty. Checked the RTS and it is empty. Seems like all we've had to crunch for several hours are noise bombs.


GPUs are dry, CPU is happy.... Einstein here come the GPUs...
ID: 1962430 · Report as offensive
Kevin Olley

Send message
Joined: 3 Aug 99
Posts: 906
Credit: 261,085,289
RAC: 572
United Kingdom
Message 1962431 - Posted: 29 Oct 2018, 16:02:21 UTC - in response to Message 1962428.  

Just looked at my hosts and found the big iron empty. Checked the RTS and it is empty. Seems like all we've had to crunch for several hours are noise bombs.


Still got a few on the CPU but the 1080's are now running Einstein, it stops them getting cold;-)
Kevin


ID: 1962431 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13159
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1962446 - Posted: 29 Oct 2018, 17:24:52 UTC

It looks like the elves in the lab loaded some new BLC01 files. Hope the splitters move to them soonest and flush out the noise bombs.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1962446 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1962453 - Posted: 29 Oct 2018, 19:01:49 UTC - in response to Message 1962446.  

It looks like the elves in the lab loaded some new BLC01 files. Hope the splitters move to them soonest and flush out the noise bombs.


and now they have added an Aricebo file 28oc18aa. The creation rate is up to 90/sec and we are still taking them as fast as created.
ID: 1962453 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1962465 - Posted: 29 Oct 2018, 21:09:00 UTC - in response to Message 1962402.  

Hey - I have noticed many WUs recently that process in < 10 secs. Like dozens or hundreds/day, both on GPU and CPU. Are these the "noisy" WUs referred to above? Are they the variety that has 30 spikes detected and then dropped by the app(s)?


. . Yes!

Stephen

.
ID: 1962465 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13159
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1962490 - Posted: 30 Oct 2018, 0:54:28 UTC

Still getting noise bombs.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1962490 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1962499 - Posted: 30 Oct 2018, 1:59:33 UTC - in response to Message 1962490.  

Still getting noise bombs.


. . About 80% to 90% it seems ... need to preserve Arecibo VLARs to have any hope for the outage ...

Stephen

:(
ID: 1962499 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13159
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1962500 - Posted: 30 Oct 2018, 2:01:07 UTC - in response to Message 1962499.  

I'm hoping the BLC23 clear out of the RTS buffer overnight and we can chew on a steady diet of BLC22, BLC01 and Arecibo tasks.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1962500 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1962501 - Posted: 30 Oct 2018, 2:24:44 UTC - in response to Message 1962500.  

I'm hoping the BLC23 clear out of the RTS buffer overnight and we can chew on a steady diet of BLC22, BLC01 and Arecibo tasks.


. . the Blc22 tasks are from the same date/time and are just as noisy, they are all noise bombs. We have to hope there is a good supply of Arecibo tapes to keep the Arecibo VLARs coming to slow things down and let the RTS refill. Hopefully the remaining blc22/blc23 tapes will split and clear before the outage so that maybe the Blc01 tapes can start to come out and are less noise prone.

Stephen

:(
ID: 1962501 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13159
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1962502 - Posted: 30 Oct 2018, 2:41:55 UTC - in response to Message 1962501.  

I'm hoping the BLC23 clear out of the RTS buffer overnight and we can chew on a steady diet of BLC22, BLC01 and Arecibo tasks.


. . the Blc22 tasks are from the same date/time and are just as noisy, they are all noise bombs. We have to hope there is a good supply of Arecibo tapes to keep the Arecibo VLARs coming to slow things down and let the RTS refill. Hopefully the remaining blc22/blc23 tapes will split and clear before the outage so that maybe the Blc01 tapes can start to come out and are less noise prone.

Stephen

:(

I'm not finding many BLC22 noise bombs at all. I'd say 90% are good.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1962502 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11191
Credit: 29,581,041
RAC: 66
United States
Message 1962506 - Posted: 30 Oct 2018, 3:41:51 UTC - in response to Message 1962502.  

Well Green Banks should not have any noise. IMO the noise is probably a hardware issue. We will never know.
ID: 1962506 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13159
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1962509 - Posted: 30 Oct 2018, 3:45:30 UTC - in response to Message 1962506.  
Last modified: 30 Oct 2018, 3:46:50 UTC

Well Green Banks should not have any noise. IMO the noise is probably a hardware issue. We will never know.

The Arecibo tasks almost always have noise. It's in the form of the massive radar pulses from the dish when active. At least they get blanked automatically and we never see them except for the AP tasks.

Agree for Green Bank being in a noise free zone, it must have been hardware failure of some kind.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1962509 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13414
Credit: 208,696,464
RAC: 304
Australia
Message 1962537 - Posted: 30 Oct 2018, 6:34:30 UTC

Well, things did get interesting overnight.
Looks like the return rate just missed out on a sustained 300k/s, at least it's now dropped to "only" 190k and the splitters managed to get going again & are mostly keeping up with the present demand- occasionally getting ahead & re-building the Ready-to-send buffer, then dropping behind & the buffer drains again.
And while the deleters managed to catch up & get back on top of the work load, the Assimilators appear to have given up- the present backlog is just short of 1 million.

Interesting that after all the VLAR Arecibo work we had been getting, now we've got lots of quick GBT jobs, we're also getting lots of shorter running Arecibo work. The perversity of nature.
Grant
Darwin NT
ID: 1962537 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1962540 - Posted: 30 Oct 2018, 7:55:25 UTC - in response to Message 1962502.  

. . the Blc22 tasks are from the same date/time and are just as noisy, they are all noise bombs. We have to hope there is a good supply of Arecibo tapes to keep the Arecibo VLARs coming to slow things down and let the RTS refill. Hopefully the remaining blc22/blc23 tapes will split and clear before the outage so that maybe the Blc01 tapes can start to come out and are less noise prone.
Stephen

I'm not finding many BLC22 noise bombs at all. I'd say 90% are good.


. . Then I suggest you take the opportunity to buy a lottery ticket :). I am having the opposite results here. On all 4 machines.

Stephen

:(
ID: 1962540 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1962541 - Posted: 30 Oct 2018, 7:57:57 UTC - in response to Message 1962506.  

Well Green Banks should not have any noise. IMO the noise is probably a hardware issue. We will never know.


. . Even the best locations can experience RFI at some time. The earlier tapes in the current two series (which cover the same observing period) were pretty good and noise free, but at the tail end of the series both series seem to have experienced a rise in RFI.

Stephen

:(
ID: 1962541 · Report as offensive
Previous · 1 . . . 25 · 26 · 27 · 28 · 29 · 30 · 31 . . . 37 · Next

Message boards : Number crunching : Panic Mode On (113) Server Problems?


 
©2023 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.