Panic Mode On (109) Server Problems?

Message boards : Number crunching : Panic Mode On (109) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 28 · 29 · 30 · 31 · 32 · 33 · 34 . . . 35 · Next

AuthorMessage
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22720
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1913333 - Posted: 16 Jan 2018, 10:50:28 UTC

Thanks Richard - I was on my phone and looking at two things at once on that is "problematic" - hopefully the new SERVER binaries will address some of the issues we've seen on main of late. (And Eric's note post dates the "new splitters broke Beta" announcement by a few weeks, and the more recent Beta stuff has flowed fairly smoothly....
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1913333 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1913371 - Posted: 17 Jan 2018, 0:44:41 UTC

and we are back
ID: 1913371 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1913376 - Posted: 17 Jan 2018, 1:07:04 UTC

But the splitters are offline. If Eric did deploy new server code, I hope we are not seeing the same issue that Beta had with the new server code.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1913376 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1913383 - Posted: 17 Jan 2018, 1:38:43 UTC
Last modified: 17 Jan 2018, 1:46:19 UTC

And the splitters are back online. But the SSP has not updated in 15 minutes. So who knows?
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1913383 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11450
Credit: 29,581,041
RAC: 66
United States
Message 1913387 - Posted: 17 Jan 2018, 2:06:12 UTC

Current result creation rate ** 0/sec 0.0148/sec 2.5690/sec 6m

This is not gud.
ID: 1913387 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1913389 - Posted: 17 Jan 2018, 2:17:25 UTC - in response to Message 1913298.  

Is it just me, or are these new 02 WUs processing even faster than the previous 05s? Around 10sec on the GPUs and 4min on the CPU faster.
The servers weren't doing too well as it was, these new WUs will push them even harder.


. . Yes, they are processing slightly faster than those Blc05 units were. Not quite so dramatic here though they are around 30 secs faster on the 950 GPU and about one min on the i5-6400, maybe 10 secs on the 970s and 2 mins on the i5-6600, and finally about 10 to 12 secs faster on the 1050ti, the C2D doesn't crunch normally but during the outage it ran a couple of Blc02s and they were about 2 to 5 mins quicker (compared to 81 to 84 mins).

Stephen

PS:- the post outage hourly return nearly hit the 200,000 mark :(
ID: 1913389 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1913390 - Posted: 17 Jan 2018, 2:19:42 UTC - in response to Message 1913315.  

Looks like we've reached that critical point of returned-last-hour, WU-waiting-deletion, in-progress and splitter load.
Waiting-deletion is going up, splitter output has gone down, and Ready-to-send buffer is rapidly emptying. And with the shorter WU run times, channels left to be split is rapidly diminishing also.


. . On that subject (channels to be split) I am left pondering why it is that all the "tapes" (Disks) show as 52.24 GB but while some show 80 to 120 channels others show only one or two. What causes that level of discrepancy?

Stephen

??
ID: 1913390 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13904
Credit: 208,696,464
RAC: 304
Australia
Message 1913400 - Posted: 17 Jan 2018, 2:54:20 UTC - in response to Message 1913383.  
Last modified: 17 Jan 2018, 3:22:48 UTC

And the splitters are back online. But the SSP has not updated in 15 minutes. So who knows?

Splitters showing Green, and status page has updated. But still no work actually coming from the splitters.
Server code update broke our splitters as well as Betas?


Edit- over 2 hours since the project came back up, but still no splitter output.
Grant
Darwin NT
ID: 1913400 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13904
Credit: 208,696,464
RAC: 304
Australia
Message 1913402 - Posted: 17 Jan 2018, 2:56:27 UTC - in response to Message 1913390.  

. . On that subject (channels to be split) I am left pondering why it is that all the "tapes" (Disks) show as 52.24 GB but while some show 80 to 120 channels others show only one or two. What causes that level of discrepancy?

Only those in progress (dark green), or have been processed (light green), show up.
Grant
Darwin NT
ID: 1913402 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1913411 - Posted: 17 Jan 2018, 3:45:40 UTC
Last modified: 17 Jan 2018, 3:46:25 UTC

Apparently the splitters are splitting as showed by the Splitter status but where the splitted data is going?
ID: 1913411 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11450
Credit: 29,581,041
RAC: 66
United States
Message 1913412 - Posted: 17 Jan 2018, 3:50:37 UTC

I made it thru todays outrage but it looks like I will go dry this evening unless something gets fixed.
ID: 1913412 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13904
Credit: 208,696,464
RAC: 304
Australia
Message 1913414 - Posted: 17 Jan 2018, 4:09:37 UTC - in response to Message 1913411.  

Apparently the splitters are splitting as showed by the Splitter status but where the splitted data is going?

I think it's a case of the splitters are running, but they're not actually doing anything.

Apparently, the updated server code broke the splitters on Beta.
They got that fixed, but it would appear there's some differences between the splitters here & at Beta that didn't cope too well with the new code.
It could be a while before work is flowing again.
Grant
Darwin NT
ID: 1913414 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1913428 - Posted: 17 Jan 2018, 5:06:12 UTC

Nothing new on the graphs, the In Progress is still headed Down;


about a couple hours left, then my machines will be going cold...like the weather outside.
ID: 1913428 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13904
Credit: 208,696,464
RAC: 304
Australia
Message 1913429 - Posted: 17 Jan 2018, 5:12:32 UTC - in response to Message 1913428.  

Nothing new on the graphs, the In Progress is still headed Down;

Even if they get the splitters fixed quickly, and they then split like never before, and maintain that level of output like never before, it's going to be a long and tough recovery.
Grant
Darwin NT
ID: 1913429 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1913430 - Posted: 17 Jan 2018, 5:13:54 UTC - in response to Message 1913412.  

I made it thru todays outrage but it looks like I will go dry this evening unless something gets fixed.


. . I have about 3 to 5 hours for all machines to be empty. Considering it is after 9pm in Berkeley I guess the guys (computers) will get to sleep tonight...

Stephen

:(
ID: 1913430 · Report as offensive
Profile KWSN THE Holy Hand Grenade!
Volunteer tester
Avatar

Send message
Joined: 20 Dec 05
Posts: 3187
Credit: 57,163,290
RAC: 0
United States
Message 1913444 - Posted: 17 Jan 2018, 8:15:48 UTC
Last modified: 17 Jan 2018, 8:18:45 UTC

someone needs to give the splitters a (re-) boot in the bum... 0 WU ready to send and only .7584 WU per second created (with 0 WU available, that should be in the 49-50 range!)

It's time to give your alternate projects some love... (IF they aren't also down [hello, GPUgrid?])
.

Hello, from Albany, CA!...
ID: 1913444 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1858
Credit: 268,616,081
RAC: 1,349
United States
Message 1913450 - Posted: 17 Jan 2018, 8:46:12 UTC - in response to Message 1913444.  

someone needs to give the splitters a (re-) boot in the bum... 0 WU ready to send and only .7584 WU per second created (with 0 WU available, that should be in the 49-50 range!)
Apparently a change was made that didn't go well. Splitters crashed shortly after the outage.

It's time to give your alternate projects some love... (IF they aren't also down [hello, GPUgrid?])
Yeah, only good news here is I finally got a config I'm happy with that lets Einstein take over when SETI goes sideways without having to dump a bunch of work when SETI resurfaces.
ID: 1913450 · Report as offensive
Profile Stargate (SA)
Volunteer tester
Avatar

Send message
Joined: 4 Mar 10
Posts: 1854
Credit: 2,258,721
RAC: 0
Australia
Message 1913451 - Posted: 17 Jan 2018, 8:53:04 UTC

I don't personally think so IMO maybe the project is in disarray after what happened at Arecibo, they would be still finding there feet one would imagine..
ID: 1913451 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13904
Credit: 208,696,464
RAC: 304
Australia
Message 1913452 - Posted: 17 Jan 2018, 8:54:16 UTC - in response to Message 1913449.  
Last modified: 17 Jan 2018, 8:54:46 UTC

I bet it is the patches for Meltdown and Spectre that's the reason for the server slowdowns.

If we're lucky.
If not, it's the increased load (of all those short running GBT WUs) showing the present systems limits, and the patches will just make things even worse when they are applied.

Present lack of work- most likely the server update that worked on Beta (after it was fixed for breaking the splitters there), broke the splitters here.
Grant
Darwin NT
ID: 1913452 · Report as offensive
Cruncher-American Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor

Send message
Joined: 25 Mar 02
Posts: 1513
Credit: 370,893,186
RAC: 340
United States
Message 1913459 - Posted: 17 Jan 2018, 9:58:28 UTC

Currently, the server status page is in an inconsistent state vis a vis the splitters for BLC.

It shows 14 splitters running, but 17 tapes being processed, one of which appears to have > 1 splitter associated with it.

With all the things that have happened recently, I don't recall ever seeing ghost splitters before.

Did I miss something?
ID: 1913459 · Report as offensive
Previous · 1 . . . 28 · 29 · 30 · 31 · 32 · 33 · 34 . . . 35 · Next

Message boards : Number crunching : Panic Mode On (109) Server Problems?


 
©2025 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.