Panic Mode On (116) Server Problems?

Message boards : Number crunching : Panic Mode On (116) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 35 · 36 · 37 · 38 · 39 · 40 · 41 . . . 48 · Next

AuthorMessage
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 11780
Credit: 179,886,095
RAC: 218,150
Australia
Message 1999651 - Posted: 26 Jun 2019, 4:10:13 UTC - in response to Message 1999572.  
Last modified: 26 Jun 2019, 4:37:05 UTC

We are definitely in a shorty storm. We are returning 145K results per hour right now, when we have been closer to 100k/hr for these data files. We need a shorty storm scale like we have for hurricanes.

Around 187k for a while now (down from a peak of over 200k).

Edit-
Make that over 190k for a while now.
Grant
Darwin NT
ID: 1999651 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 11780
Credit: 179,886,095
RAC: 218,150
Australia
Message 1999657 - Posted: 26 Jun 2019, 6:00:44 UTC - in response to Message 1999651.  
Last modified: 26 Jun 2019, 6:09:34 UTC

Edit-
Make that over 190k for a while now.

Just hit 197k. Can we make the 200k mark? (And when will the servers fall in a heap?)

Edit-
That didn't take long, over 200k and still climbing.
Grant
Darwin NT
ID: 1999657 · Report as offensive
Profile Keith Myers Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 10135
Credit: 981,840,193
RAC: 1,521,683
United States
Message 1999658 - Posted: 26 Jun 2019, 6:04:14 UTC - in response to Message 1999613.  

18dc09aa remains stuck ...

I don't know why they haven't kicked that file to the curb long ago.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1999658 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 11780
Credit: 179,886,095
RAC: 218,150
Australia
Message 1999659 - Posted: 26 Jun 2019, 6:08:18 UTC - in response to Message 1999658.  

18dc09aa remains stuck ...

I don't know why they haven't kicked that file to the curb long ago.

Or at least used it for debugging just what it is that causes the splitters to jam up on it.
Grant
Darwin NT
ID: 1999659 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 596
Credit: 1,981,841
RAC: 1,079
United States
Message 1999660 - Posted: 26 Jun 2019, 6:32:11 UTC - in response to Message 1999657.  

Edit-
Make that over 190k for a while now.

Just hit 197k. Can we make the 200k mark? (And when will the servers fall in a heap?)

Edit-
That didn't take long, over 200k and still climbing.


says 211k/ hr for returned results - bad shorty storm. I must say that I'm amazed how well the machine is handling the load. Hopefully it won't crash. The results out in the field is falling, so it can't split fast enough to keep up with demand. It's 11:30pm in California, so I doubt much will be done about the situation until morning.
ID: 1999660 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1488
Credit: 185,363,642
RAC: 464,971
United States
Message 1999662 - Posted: 26 Jun 2019, 6:42:41 UTC - in response to Message 1999572.  

We are definitely in a shorty storm.
Not getting any better, at least here ...
ID: 1999662 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2773
Credit: 605,472,164
RAC: 961,830
Canada
Message 1999663 - Posted: 26 Jun 2019, 6:44:50 UTC - in response to Message 1999660.  

Yes the servers are handling it quite well considering the load.
Once the database lag and assimilation catch up (from maintenance), performance should improve with them having less load as well.

I wouldn't expect it to get any better for overflows until the next tape series - there are a million slower CPUs out there with 24h plus cache, so they are still working on normal runtimes. As they get into these tasks ... more and more returns.
ID: 1999663 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 11780
Credit: 179,886,095
RAC: 218,150
Australia
Message 1999664 - Posted: 26 Jun 2019, 7:09:34 UTC - in response to Message 1999663.  

I wouldn't expect it to get any better for overflows until the next tape series - there are a million slower CPUs out there with 24h plus cache, so they are still working on normal runtimes. As they get into these tasks ... more and more returns.

Yeah, I had been getting 1 or 2 noise bombs here & there, but just had a batch of 9 in a row bomb out.
Grant
Darwin NT
ID: 1999664 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 11780
Credit: 179,886,095
RAC: 218,150
Australia
Message 1999665 - Posted: 26 Jun 2019, 7:23:35 UTC - in response to Message 1999572.  

We are definitely in a shorty storm.

Not shortie, Noise bombs.
Shorties are Arecibo WUs that take less than half the usual time to process, but at least do take time process.
Noise bombs are what we've got at the moment- noisy data that that finishes almost as soon as it starts.

And to add to the fun, the splitters are showing signs of problems. Pumping out the work at a good rate, then not producing any for several minutes. Then pumping it out again for a hour or two, then nothing for a while.
Grant
Darwin NT
ID: 1999665 · Report as offensive
Profile Wiggo "Democratic Socialist"
Avatar

Send message
Joined: 24 Jan 00
Posts: 17035
Credit: 236,878,284
RAC: 181,649
Australia
Message 1999667 - Posted: 26 Jun 2019, 7:42:51 UTC

Hundreds of Noise Bombs have been through here (probably well over a thousand) and now I'm having trouble downloading enough work fast enough to cover the mongrels.

Cheers.
ID: 1999667 · Report as offensive
Profile Stargate (SA)
Volunteer tester
Avatar

Send message
Joined: 4 Mar 10
Posts: 1850
Credit: 2,256,965
RAC: 379
Australia
Message 1999669 - Posted: 26 Jun 2019, 7:50:04 UTC - in response to Message 1999667.  

Getting loads myself Boss
ID: 1999669 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 17900
Credit: 408,279,262
RAC: 36,150
United Kingdom
Message 1999675 - Posted: 26 Jun 2019, 8:11:46 UTC

Looks as though the splitters have spluttered - creation rate zero, RTS zero

Ah well, I'm just entering my day time shut-down, so I will just sit back for 12 hours and hope they've had the kick of life by then....
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1999675 · Report as offensive
Profile Stargate (SA)
Volunteer tester
Avatar

Send message
Joined: 4 Mar 10
Posts: 1850
Credit: 2,256,965
RAC: 379
Australia
Message 1999677 - Posted: 26 Jun 2019, 8:16:11 UTC

A bottle of vitamin c will help..
ID: 1999677 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 11780
Credit: 179,886,095
RAC: 218,150
Australia
Message 1999680 - Posted: 26 Jun 2019, 8:46:02 UTC - in response to Message 1999677.  
Last modified: 26 Jun 2019, 8:57:24 UTC

A bottle of vitamin c will help..

Gone beyond a splutter, they've been down and out for over an hour now.

Which is odd as the usual signs of impending doom (a increasing backlog of WUs awaiting validation, deletion etc) weren't there. While the load was huge, everything was chugging along OK, till the splitters stopped splitting (after an initial pause in output).
Could be return rates over 200k are the limit for the servers, although the cessation of splitting actually occurred when the return rate was less than it had been.

Present return rate of 237k will only last till people's caches run dry.


Edit-
Q. What is the best way to fix an issue with the Seti servers.
A. Post a comment about it .

The splitters are splitting again. Not a lot, but it's still better than nothing.
Grant
Darwin NT
ID: 1999680 · Report as offensive
Stephen "Heretic" Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 4720
Credit: 159,727,897
RAC: 257,202
Australia
Message 1999681 - Posted: 26 Jun 2019, 8:48:27 UTC - in response to Message 1999664.  

I wouldn't expect it to get any better for overflows until the next tape series - there are a million slower CPUs out there with 24h plus cache, so they are still working on normal runtimes. As they get into these tasks ... more and more returns.

Yeah, I had been getting 1 or 2 noise bombs here & there, but just had a batch of 9 in a row bomb out.

. . I just returned 140 tasks, 91 were noise bombs :(

Stephen

:(
ID: 1999681 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 11780
Credit: 179,886,095
RAC: 218,150
Australia
Message 1999682 - Posted: 26 Jun 2019, 8:49:42 UTC - in response to Message 1999681.  

. . I just returned 140 tasks, 91 were noise bombs :(

And nothing to replace them.
Grant
Darwin NT
ID: 1999682 · Report as offensive
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 26 May 99
Posts: 9869
Credit: 87,759,811
RAC: 92,281
United Kingdom
Message 1999688 - Posted: 26 Jun 2019, 9:26:58 UTC

I put a new machine on line yesterday and was looking this morning at it's RAC and wondering why it was so low compared to the number of validated tasks, then I looked.

First page 15 noise bombs vs 5 actual tasks.

Bad day to fire up a new cruncher :-(
ID: 1999688 · Report as offensive
Profile Wiggo "Democratic Socialist"
Avatar

Send message
Joined: 24 Jan 00
Posts: 17035
Credit: 236,878,284
RAC: 181,649
Australia
Message 1999691 - Posted: 26 Jun 2019, 9:54:55 UTC

Well I've suspended GPU processing on my 2 rigs until I can get a bit of a buffer downloaded, but it's not fun downloading all this work when most of it is just garbage. :-(

Cheers.
ID: 1999691 · Report as offensive
Profile Freewill Project Donor
Avatar

Send message
Joined: 19 May 99
Posts: 95
Credit: 222,787,102
RAC: 639,171
United States
Message 1999692 - Posted: 26 Jun 2019, 10:07:00 UTC - in response to Message 1999691.  

Are "noise bombs" the 2-3 second, 2-3 point tasks I'm processing?
ID: 1999692 · Report as offensive
Profile Wiggo "Democratic Socialist"
Avatar

Send message
Joined: 24 Jan 00
Posts: 17035
Credit: 236,878,284
RAC: 181,649
Australia
Message 1999693 - Posted: 26 Jun 2019, 10:09:20 UTC - in response to Message 1999692.  

Are "noise bombs" the 2-3 second, 2-3 point tasks I'm processing?
That's them, instant overflows.

Cheers.
ID: 1999693 · Report as offensive
Previous · 1 . . . 35 · 36 · 37 · 38 · 39 · 40 · 41 . . . 48 · Next

Message boards : Number crunching : Panic Mode On (116) Server Problems?


 
©2019 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.