The Server Issues / Outages Thread - Panic Mode On! (118)

Message boards : Number crunching : The Server Issues / Outages Thread - Panic Mode On! (118)
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 42 · 43 · 44 · 45 · 46 · 47 · 48 . . . 94 · Next

AuthorMessage
Boiler Paul

Send message
Joined: 4 May 00
Posts: 232
Credit: 4,965,771
RAC: 64
United States
Message 2028696 - Posted: 21 Jan 2020, 1:57:48 UTC

I guess that they took it off line for tomorrows outage FWIW
ID: 2028696 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2028700 - Posted: 21 Jan 2020, 2:15:43 UTC - in response to Message 2028696.  

I guess that they took it off line for tomorrows outage FWIW

I assume it will help the servers recover because I am sure it has an I/O impact on the main database. Anything that can help the servers clear the validations and purge/delete backlogs will be welcome.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2028700 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 2028707 - Posted: 21 Jan 2020, 3:32:43 UTC
Last modified: 21 Jan 2020, 3:32:58 UTC

had a good run for about half the day. now back to no tasks available
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 2028707 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2028711 - Posted: 21 Jan 2020, 4:01:24 UTC

Think the splitters have commandeered the I/O and the schedulers are getting short-changed in responding to work requests.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2028711 · Report as offensive
Ville Saari
Avatar

Send message
Joined: 30 Nov 00
Posts: 1158
Credit: 49,177,052
RAC: 82,530
Finland
Message 2028713 - Posted: 21 Jan 2020, 4:18:58 UTC - in response to Message 2028711.  

Think the splitters have commandeered the I/O and the schedulers are getting short-changed in responding to work requests.
Or all the stuff that hit the replica before is now bombing the master and commandeering the I/O.
ID: 2028713 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2028716 - Posted: 21 Jan 2020, 4:54:23 UTC - in response to Message 2028713.  

We'll see if the splitter eventually throttles down after 1.2M or so. At least the replica is back in sync after being taken offline.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2028716 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11366
Credit: 29,581,041
RAC: 66
United States
Message 2028725 - Posted: 21 Jan 2020, 6:50:08 UTC

Tuesday's outage will be epic IMO
ID: 2028725 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13765
Credit: 208,696,464
RAC: 304
Australia
Message 2028731 - Posted: 21 Jan 2020, 8:56:58 UTC - in response to Message 2028725.  

Tuesday's outage will be epic IMO
Already promised to be so by Eric.
Grant
Darwin NT
ID: 2028731 · Report as offensive
Miklos M.

Send message
Joined: 5 May 99
Posts: 955
Credit: 136,115,648
RAC: 73
Hungary
Message 2028734 - Posted: 21 Jan 2020, 9:43:36 UTC - in response to Message 2028731.  

Maybe that is why I received the biggest number of wu's, ever. Just giving thanks after all my gripes.

Miklos
ID: 2028734 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13765
Credit: 208,696,464
RAC: 304
Australia
Message 2028735 - Posted: 21 Jan 2020, 10:30:12 UTC

Just found a AP WU that had been downloading for over an hour. Disabled & re-enabled network access & it cleared.
Grant
Darwin NT
ID: 2028735 · Report as offensive
Ville Saari
Avatar

Send message
Joined: 30 Nov 00
Posts: 1158
Credit: 49,177,052
RAC: 82,530
Finland
Message 2028754 - Posted: 21 Jan 2020, 15:05:48 UTC

Tuesday outage is late...
ID: 2028754 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 2028755 - Posted: 21 Jan 2020, 15:21:37 UTC - in response to Message 2028754.  
Last modified: 21 Jan 2020, 15:22:15 UTC

yeah not a good sign to start late on what we all know will be a long outage.

but hey, at least this give us more time to fill those caches lol
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 2028755 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2028757 - Posted: 21 Jan 2020, 15:40:44 UTC

Not refilling caches here. Just getting project has 0 tasks to send.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2028757 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 2028760 - Posted: 21 Jan 2020, 16:04:35 UTC - in response to Message 2028757.  

yup i noticed that in the last 30mins or so.

maybe a sign its coming down soon.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 2028760 · Report as offensive
Sleepy
Volunteer tester
Avatar

Send message
Joined: 21 May 99
Posts: 219
Credit: 98,947,784
RAC: 28,360
Italy
Message 2028763 - Posted: 23 Jan 2020, 16:30:00 UTC

And we are back!!
ID: 2028763 · Report as offensive
Boiler Paul

Send message
Joined: 4 May 00
Posts: 232
Credit: 4,965,771
RAC: 64
United States
Message 2028767 - Posted: 23 Jan 2020, 16:40:29 UTC

now we wait to see how long it will take to report and receive new tasks
ID: 2028767 · Report as offensive
Ville Saari
Avatar

Send message
Joined: 30 Nov 00
Posts: 1158
Credit: 49,177,052
RAC: 82,530
Finland
Message 2028768 - Posted: 23 Jan 2020, 16:46:24 UTC - in response to Message 2028767.  

now we wait to see how long it will take to report and receive new tasks
My hosts can report but they get just 'no tasks available' in return. It will take a while to report everything as I have set them to report only 50 tasks at a time to try to avoid causing huge load spikes to the servers.
ID: 2028768 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2028774 - Posted: 23 Jan 2020, 17:00:40 UTC - in response to Message 2028768.  

Haven't been able to get a connection to the server to report yet.
Scheduler request failed: Timeout was reached
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2028774 · Report as offensive
Ville Saari
Avatar

Send message
Joined: 30 Nov 00
Posts: 1158
Credit: 49,177,052
RAC: 82,530
Finland
Message 2028777 - Posted: 23 Jan 2020, 17:07:40 UTC
Last modified: 23 Jan 2020, 17:12:43 UTC

I was able to report a few batches but not any more. Apparently more hosts have now woken up from their long backoffs and the scheduler can't cope with them all any more.

Edit: got a new error message now: 'Server error: feeder not running'
ID: 2028777 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 2028778 - Posted: 23 Jan 2020, 17:08:52 UTC - in response to Message 2028774.  

Haven't been able to get a connection to the server to report yet.
Scheduler request failed: Timeout was reached


. .Yep me too

Stephen

:(
ID: 2028778 · Report as offensive
Previous · 1 . . . 42 · 43 · 44 · 45 · 46 · 47 · 48 . . . 94 · Next

Message boards : Number crunching : The Server Issues / Outages Thread - Panic Mode On! (118)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.