Panic Mode On (110) Server Problems?

Message boards : Number crunching : Panic Mode On (110) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 13 · 14 · 15 · 16 · 17 · 18 · 19 . . . 37 · Next

AuthorMessage
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1918703 - Posted: 14 Feb 2018, 2:41:24 UTC - in response to Message 1918701.  
Last modified: 14 Feb 2018, 2:42:22 UTC

Yes, hope it is just Eric getting all his ducks in a row before starting the server initialization sequence.

BINGO!!! Just as I typed. All GREEN!
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1918703 · Report as offensive
Profile Stargate (SA)
Volunteer tester
Avatar

Send message
Joined: 4 Mar 10
Posts: 1854
Credit: 2,258,721
RAC: 0
Australia
Message 1918706 - Posted: 14 Feb 2018, 2:44:32 UTC

Might be all green but I'm getting nout
ID: 1918706 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1918707 - Posted: 14 Feb 2018, 2:52:55 UTC - in response to Message 1918706.  

Well at least there is something there to talk to. Internal server error for me but that means I just have to fight through all the others trying to connect. Only have one machine that is almost out of work. Just need to report the 3000 or so tasks.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1918707 · Report as offensive
Profile Freewill Project Donor
Avatar

Send message
Joined: 19 May 99
Posts: 766
Credit: 354,398,348
RAC: 11,693
United States
Message 1918708 - Posted: 14 Feb 2018, 2:56:59 UTC

Saddle up, lock and load!
ID: 1918708 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1918710 - Posted: 14 Feb 2018, 3:00:35 UTC

All crunched WU reported, still no new WU DL.
ID: 1918710 · Report as offensive
Profile Chris904395093209d Project Donor
Volunteer tester

Send message
Joined: 1 Jan 01
Posts: 112
Credit: 29,923,129
RAC: 6
United States
Message 1918711 - Posted: 14 Feb 2018, 3:02:11 UTC

WOW you really have to be quick to get any work that was ready to go before the maint. window. I manually updated my faster machines first, then went to my slower machines and those machines I went to last can't get any work. Not sure if the download servers are over loaded or if the work that was ready has already been sucked up. At any rate, as Keith said, all green!
~Chris

ID: 1918711 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1918712 - Posted: 14 Feb 2018, 3:06:55 UTC

The SSP only updates every ten minutes. Between the time the SSP said we had +300K tasks in the RTS buffer and the time the servers went all green, those tasks got sucked up immediately. Now just have to fight everyone else asking for work as the splitters ramp up to handle demand. It will be at least 6-8 hours before the RTS buffer starts filling again other than immediately dumping at every request.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1918712 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13855
Credit: 208,696,464
RAC: 304
Australia
Message 1918713 - Posted: 14 Feb 2018, 3:12:23 UTC - in response to Message 1918712.  

It will be at least 6-8 hours before the RTS buffer starts filling again other than immediately dumping at every request.

Maybe.
The WU-awaiting-deletion backlog continues to grow, and that impacts splitter output.
If it clears soon, then things should recover fairly quickly. If not, who knows how long it will take (as long as we don't run out of disk space).
Grant
Darwin NT
ID: 1918713 · Report as offensive
Profile Chris904395093209d Project Donor
Volunteer tester

Send message
Joined: 1 Jan 01
Posts: 112
Credit: 29,923,129
RAC: 6
United States
Message 1918715 - Posted: 14 Feb 2018, 3:15:47 UTC

Almost 379,000 results received in the last hour, I guess that would dry up the well a bit when those machines then ask for work
~Chris

ID: 1918715 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1918716 - Posted: 14 Feb 2018, 3:19:55 UTC - in response to Message 1918713.  
Last modified: 14 Feb 2018, 3:21:13 UTC

It will be at least 6-8 hours before the RTS buffer starts filling again other than immediately dumping at every request.

Maybe.
The WU-awaiting-deletion backlog continues to grow, and that impacts splitter output.
If it clears soon, then things should recover fairly quickly. If not, who knows how long it will take (as long as we don't run out of disk space).

I agree, that backlog of results purging is continuing to be worrisome. Hasn't reduced since last week's outage. I believe as you do that I/O contention impacts the splitters.

[Edit] And just as I posted, the creation rate cratered at Haveland. Think that means they are doing the wu and results purge.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1918716 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13855
Credit: 208,696,464
RAC: 304
Australia
Message 1918718 - Posted: 14 Feb 2018, 3:32:34 UTC - in response to Message 1918716.  
Last modified: 14 Feb 2018, 3:34:53 UTC

[Edit] And just as I posted, the creation rate cratered at Haveland.

Often does that after an outage (very bursty- lots, none, lots, none..), generally takes them a while to really get going.
I'm more concerned with the lack of work from the Scheduler at the moment. 300k+ WUs Ready-to-send, but i'm getting nothing but "Project has no tasks available" after 46 on the initial request.
Even after the last extended outage most requests resulted in at least some work.
Grant
Darwin NT
ID: 1918718 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1918720 - Posted: 14 Feb 2018, 3:39:13 UTC - in response to Message 1918718.  

Yes, it nosedive, but it is picking back up again. I just put the one cruncher that was out of work onto a GPUGrid Long task and right after that it picked up 14 Seti tasks. So, work is trickling out as usual after the outage. That machine has the new hardware and was just changed over to two tasks per card, so it was the one that worked through my bunker the fastest. Still in good shape on the others.

Will have to get back to my compilation project to increase the allowed tasks.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1918720 · Report as offensive
Profile Chris904395093209d Project Donor
Volunteer tester

Send message
Joined: 1 Jan 01
Posts: 112
Credit: 29,923,129
RAC: 6
United States
Message 1918724 - Posted: 14 Feb 2018, 3:58:25 UTC

20 tasks stuck downloading. The last time I saw this, I had to edit my host file.
~Chris

ID: 1918724 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1918725 - Posted: 14 Feb 2018, 4:01:27 UTC - in response to Message 1918724.  

Yes, I have mucho stalled downloads too. Haven't set logging options yet, but don't think the issue is DNS. Think it is just a lot of machines asking for work.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1918725 · Report as offensive
Lionel

Send message
Joined: 25 Mar 00
Posts: 680
Credit: 563,640,304
RAC: 597
Australia
Message 1918729 - Posted: 14 Feb 2018, 4:27:12 UTC - in response to Message 1918725.  

Seems to be the thing at the moment.
ID: 1918729 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13855
Credit: 208,696,464
RAC: 304
Australia
Message 1918730 - Posted: 14 Feb 2018, 4:27:21 UTC - in response to Message 1918724.  

20 tasks stuck downloading. The last time I saw this, I had to edit my host file.

Tried both servers- it's not DNS. Problems getting work from either server.
And i can't see it being load related as we've had similar (and heavier) loads in the past with no stalled downloads- just slower than usual downloads.

Problems with the servers again.
Grant
Darwin NT
ID: 1918730 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 36832
Credit: 261,360,520
RAC: 489
Australia
Message 1918731 - Posted: 14 Feb 2018, 4:27:27 UTC

No it's not a DNS problem as I just gave my host files a flick about and all variations failed.

Cheers.
ID: 1918731 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1918732 - Posted: 14 Feb 2018, 4:33:51 UTC

It's going to be an extra-long outage until they can get the download servers straightened out.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1918732 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13855
Credit: 208,696,464
RAC: 304
Australia
Message 1918733 - Posted: 14 Feb 2018, 4:35:33 UTC - in response to Message 1918732.  

It's going to be an extra-long outage until they can get the download servers straightened out.

Yep.
Downloads are dead in the water.
Grant
Darwin NT
ID: 1918733 · Report as offensive
Profile Stargate (SA)
Volunteer tester
Avatar

Send message
Joined: 4 Mar 10
Posts: 1854
Credit: 2,258,721
RAC: 0
Australia
Message 1918735 - Posted: 14 Feb 2018, 4:44:28 UTC

This makes for a long day now :(
ID: 1918735 · Report as offensive
Previous · 1 . . . 13 · 14 · 15 · 16 · 17 · 18 · 19 . . . 37 · Next

Message boards : Number crunching : Panic Mode On (110) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.