Panic Mode On (114) Server Problems?

Message boards : Number crunching : Panic Mode On (114) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 11 · 12 · 13 · 14 · 15 · 16 · 17 . . . 45 · Next

AuthorMessage
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1971302 - Posted: 21 Dec 2018, 3:42:29 UTC - in response to Message 1971300.  

My Team is in the middle of a challenge, so definitely want the computers running at their best. Yes, in most cases the servers sort themselves out on the daily glitches. Occasionally they are severe enough that a notification of staff is necessary to alert them to an upset.

Einstein is currently having major upload troubles now. Hundreds of tasks in backoff and can't clear the hosts because of the server having comms issues.


. . Next week they are going down for some maintenance event so that will be sorted out too :)

Stephen

:)
ID: 1971302 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 38673
Credit: 261,360,520
RAC: 489
Australia
Message 1971310 - Posted: 21 Dec 2018, 5:53:51 UTC - in response to Message 1971301.  

Ok people, get a bit of life into ya's and stop staring at the SSP and your rigs every waking moment, it may surprise some, but there are other things in life and the servers (plus your rigs) will eventually sort themselves out. ;-)

I'm sorry, but I just had to say that. :-D

Cheers.


. . But I don't drink beer ... :)

Stephen

:)
Spirits?

I've needed the sugar hit a few time today just to get the energy going again in this silly heat, even if it's half strength cordial added to the rum. :-)

A person would dehydrate in the shed here today quickly on beer. ;-)

About the only thing that I'm keeping my eye on ATM is the Grafton Rain Radar.

Cheers.
ID: 1971310 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1971320 - Posted: 21 Dec 2018, 7:49:14 UTC - in response to Message 1971310.  


. . But I don't drink beer ... :)
Stephen
:)
Spirits?

I've needed the sugar hit a few time today just to get the energy going again in this silly heat, even if it's half strength cordial added to the rum. :-)

A person would dehydrate in the shed here today quickly on beer. ;-)

About the only thing that I'm keeping my eye on ATM is the Grafton Rain Radar.

Cheers.


. . Yeah, the weather has been giving me curry this week as well. A failed A/C unit means I am running with the windows wide open and a storm caught me out earlier in the week so I spent hours mopping up pools of rainwater the winds drove through the windows. Yesterday I locked everything up when the storms came but it got too hot and the rigs were overheating even with the wicks wound down so I had to shut everything down until I got home later. Lost a lot of productive hours there. I need to address the A/C issue :(

Stephen

:(
ID: 1971320 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 14015
Credit: 208,696,464
RAC: 304
Australia
Message 1971542 - Posted: 22 Dec 2018, 21:45:23 UTC

Looks like the Validators have issues, yet again. Their backlog continues to grow.
Grant
Darwin NT
ID: 1971542 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1861
Credit: 268,616,081
RAC: 1,349
United States
Message 1971548 - Posted: 22 Dec 2018, 22:17:24 UTC - in response to Message 1971542.  
Last modified: 22 Dec 2018, 22:36:54 UTC

Looks like the Validators have issues, yet again. Their backlog continues to grow.
Grant,
Are you referring to:
Item                                            sah7        ap          sah8            as of
Results returned and awaiting validation	0	    32,614	5,737,593	0m

?
ID: 1971548 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1649
Credit: 12,921,799
RAC: 89
New Zealand
Message 1971555 - Posted: 22 Dec 2018, 22:51:40 UTC

I am just curious has anybody noticed any ghost work units? I have 79. Not sure how I got them. I will try and clean them up if I am not successful I will complete the work I have and set them free.
ID: 1971555 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 14015
Credit: 208,696,464
RAC: 304
Australia
Message 1971556 - Posted: 22 Dec 2018, 22:56:42 UTC - in response to Message 1971548.  

Looks like the Validators have issues, yet again. Their backlog continues to grow.
Grant,
Are you referring to:
Item                                            sah7        ap          sah8            as of
Results returned and awaiting validation	0	    32,614	5,737,593	0m

?

Yep.
When you look at the graphs, you can see how backlogged it is- usually around 4 million, now almost 5.8 million. And of course with them backlogged, that causes the Assimilaators to become backlogged, and purging can't occur either.
Grant
Darwin NT
ID: 1971556 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 14015
Credit: 208,696,464
RAC: 304
Australia
Message 1971557 - Posted: 22 Dec 2018, 22:59:23 UTC - in response to Message 1971555.  

I am just curious has anybody noticed any ghost work units? I have 79. Not sure how I got them. I will try and clean them up if I am not successful I will complete the work I have and set them free.

No ghosts here.
Ghosts tend to be the result of a network glitch during a Scheduler request (whether it's a internet issue, or the result of network issues on a system that's struggling).

I wouldn't bother mucking around with them. Just let them be, and they'll time out & get re-issued.
Grant
Darwin NT
ID: 1971557 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1971559 - Posted: 22 Dec 2018, 23:19:38 UTC - in response to Message 1971555.  

I am just curious has anybody noticed any ghost work units? I have 79. Not sure how I got them. I will try and clean them up if I am not successful I will complete the work I have and set them free.


. . The ghost recovery should sort that out, if you have the patience to do it 4 or 5 times as you only get 20 on each attempt.

Stephen

:)
ID: 1971559 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1649
Credit: 12,921,799
RAC: 89
New Zealand
Message 1971561 - Posted: 22 Dec 2018, 23:37:01 UTC - in response to Message 1971559.  

I am just curious has anybody noticed any ghost work units? I have 79. Not sure how I got them. I will try and clean them up if I am not successful I will complete the work I have and set them free.


. . The ghost recovery should sort that out, if you have the patience to do it 4 or 5 times as you only get 20 on each attempt.

Stephen

:)

I agree Stephen, timing is everything and sometimes I find it tricky to trigger the double resend I will give it a go
ID: 1971561 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 14015
Credit: 208,696,464
RAC: 304
Australia
Message 1971590 - Posted: 23 Dec 2018, 1:49:32 UTC
Last modified: 23 Dec 2018, 1:56:44 UTC

Web site & forums have been slower than a month of Sundays for a good 10 minutes now.

Edit- and the Scheduler has been unresponsive for at least 20min.
23/12/2018 11:08:30 | SETI@home | Scheduler request failed: Couldn't connect to server

And several of the Server stats haven't been updated for an hour.
Grant
Darwin NT
ID: 1971590 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1971600 - Posted: 23 Dec 2018, 2:06:15 UTC
Last modified: 23 Dec 2018, 2:06:52 UTC

Yes the servers have pretty much fallen over. If you look at the Haveland graphs it is repeating almost exactly the pattern from 5 days ago with a severe climb and peak to over 6 million for the task/WU validations. Takes another day for it to fall back down.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1971600 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1971604 - Posted: 23 Dec 2018, 2:48:25 UTC

I think the servers ebb and flow with either cron jobs or if then type triggers. I think there will be times with somethings get backed up (assimilation/validation) and there will be times when we won't get work handed out, and it might just be part of a new normal.
ID: 1971604 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1971609 - Posted: 23 Dec 2018, 3:13:19 UTC - in response to Message 1971604.  
Last modified: 23 Dec 2018, 3:18:42 UTC

I think the servers ebb and flow with either cron jobs or if then type triggers. I think there will be times with somethings get backed up (assimilation/validation) and there will be times when we won't get work handed out, and it might just be part of a new normal.


. . Well I'm hoping it is just the now infamous 'daily glitch'. Otherwise my rigs may get Christmas off, "no tasks available" and all that.

. . Still maybe there will be someone in the lab on Christmas Eve ... otherwise they should be left in peace for the duration ...

{Edit}
. . OK! In true SETI " na na nah na, na na nah na ... " (imagine the Twilight Zone theme) fashion, while I was typing this message I got more work downloaded. What are the odds the servers have developed AI that monitors this thread ...

Stephen

? ?
ID: 1971609 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1971610 - Posted: 23 Dec 2018, 3:17:02 UTC - in response to Message 1971609.  

I was down about half of my cache allotment earlier but they picked themselves back up on their own without any intervention on my part. I think the servers sort it out eventually. As long as the splitters have files to work on and they keep the RTS full, I hope that we can get through the holidays without any drama.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1971610 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1971611 - Posted: 23 Dec 2018, 3:20:31 UTC - in response to Message 1971610.  

I was down about half of my cache allotment earlier but they picked themselves back up on their own without any intervention on my part. I think the servers sort it out eventually. As long as the splitters have files to work on and they keep the RTS full, I hope that we can get through the holidays without any drama.


. . Here! Here!

Stephen

(or is that Hear! Hear!)
:)
ID: 1971611 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1971692 - Posted: 23 Dec 2018, 18:43:34 UTC

I see the validators have reached a new peak for pendings. Sure hope the servers don't fall over under the increased database load.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1971692 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1971697 - Posted: 23 Dec 2018, 19:13:17 UTC - in response to Message 1971692.  

I see the validators have reached a new peak for pendings. Sure hope the servers don't fall over under the increased database load.


I think I've seen the validators go up to 10 million before the system died in the past. I'm hoping that someone will do the Tuesday maintenance and it will help. I hope the Tuesday maintenance is mainly automated. It started really early the past couple of weeks.
ID: 1971697 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14690
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1971864 - Posted: 24 Dec 2018, 20:08:10 UTC - in response to Message 1971697.  

I'm hoping that someone will do the Tuesday maintenance and it will help. I hope the Tuesday maintenance is mainly automated. It started really early the past couple of weeks.
I fear that Jeff has just proved that maintenance isn't completely automated. This just appeared on the front page:

Jeff Cobb wrote:
Because of the Christmas and New Years Day holidays, our weekly outage will be on Wednesday rather than the normal Tuesday both this week and next.
ID: 1971864 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1971868 - Posted: 24 Dec 2018, 20:52:17 UTC

The validators are validating, the assimilators are assimilating, so the system is looking good until the Wednesday outage. I'm glad they let us know. I hope everyone has a good holiday!!!
ID: 1971868 · Report as offensive
Previous · 1 . . . 11 · 12 · 13 · 14 · 15 · 16 · 17 . . . 45 · Next

Message boards : Number crunching : Panic Mode On (114) Server Problems?


 
©2026 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.