Panic Mode On (112) Server Problems?

Message boards : Number crunching : Panic Mode On (112) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 4 · 5 · 6 · 7 · 8 · 9 · 10 . . . 33 · Next

AuthorMessage
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1933347 - Posted: 3 May 2018, 11:30:03 UTC - in response to Message 1933330.  

Just found all machines with stalled downloads and major backoffs. Retried all the pending downloads and they came down in a flood. Don't know what happened with the servers. Prognosis is morning sickness I guess.


. . Maybe someone was sorting out the last firewall issue at the data centre, uploads seem normal here now.

Stephen

:)
ID: 1933347 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13770
Credit: 208,696,464
RAC: 304
Australia
Message 1933500 - Posted: 4 May 2018, 5:02:02 UTC

Other than the AP WU assimilators not doing much (their backlog is growing at a pretty good pace), the servers are looking pretty good at present.
Grant
Darwin NT
ID: 1933500 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13770
Credit: 208,696,464
RAC: 304
Australia
Message 1933674 - Posted: 4 May 2018, 21:44:00 UTC

Looks like they've got themselves a major problem figuring out just what went wrong with the new system.
Grant
Darwin NT
ID: 1933674 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13770
Credit: 208,696,464
RAC: 304
Australia
Message 1933744 - Posted: 5 May 2018, 5:40:19 UTC

For those wondering what the splitter_throttle_sah is meant to do, take a look at the Haveland graphs.
splitter_throttle_sah is presently running, there is no Arecibo work being split at present- GBT only. And the Ready-to-send buffer is sitting around 600k like it never has before; maybe this is what it's supposed to do? Can't really tell due to the resolution of the graph, but it looks like it's sitting between 580k & 600k (usually it varies by as much as 85k).
As the Arecibo VLARs (and near VLARs) empty from the Ready-to-send buffer & people's caches it will be interesting to see if this level can be maintained as the load picks up- Received-last-hour is still less than 95k.
Grant
Darwin NT
ID: 1933744 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13770
Credit: 208,696,464
RAC: 304
Australia
Message 1933946 - Posted: 6 May 2018, 7:54:56 UTC

Graphs have flatlined again- Server status data isn't updating.
Grant
Darwin NT
ID: 1933946 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51469
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1933960 - Posted: 6 May 2018, 11:37:48 UTC - in response to Message 1933959.  

Graphs have flatlined again- Server status data isn't updating.

Yeah, it's 5 hours behind now...

Knock on wood, but so far it doesn't seem to have affected the workflow.
Meow!
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1933960 · Report as offensive
Profile Cactus Bob
Avatar

Send message
Joined: 19 May 99
Posts: 209
Credit: 10,924,287
RAC: 29
Canada
Message 1933991 - Posted: 6 May 2018, 15:24:20 UTC

Looking at the SSP ----- 6 May 2018, 15:10:05 UTC

All the stats on the Page are :as of: 9 Hours

Not sure I have saw that before except during shutdowns

Bob
Sometimes I wonder, what happened to all the people I gave directions to?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
SETI@home classic workunits 4,321
SETI@home classic CPU time 22,169 hours
ID: 1933991 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1934074 - Posted: 7 May 2018, 0:00:31 UTC

. . Ruh roh!!

. . Servers not sending new work to my GPU (only on the Windows rig so far) but is sending new CPU work.

. . I will monitor ....

Stephen

? ?
ID: 1934074 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13770
Credit: 208,696,464
RAC: 304
Australia
Message 1934075 - Posted: 7 May 2018, 0:04:30 UTC - in response to Message 1934074.  

. . Servers not sending new work to my GPU (only on the Windows rig so far) but is sending new CPU work.

No issues here.
Reporting x WUs, getting x WUs.
Grant
Darwin NT
ID: 1934075 · Report as offensive
Profile JaundicedEye
Avatar

Send message
Joined: 14 Mar 12
Posts: 5375
Credit: 30,870,693
RAC: 1
United States
Message 1934077 - Posted: 7 May 2018, 0:13:29 UTC

Everything full throttle here, caches full and no hung up or downloads.

"Sour Grapes make a bitter Whine." <(0)>
ID: 1934077 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1934082 - Posted: 7 May 2018, 0:39:26 UTC - in response to Message 1934075.  

. . Servers not sending new work to my GPU (only on the Windows rig so far) but is sending new CPU work.

No issues here.
Reporting x WUs, getting x WUs.


. . Probably just this machine then ...

Stephen

? ?
ID: 1934082 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13770
Credit: 208,696,464
RAC: 304
Australia
Message 1934087 - Posted: 7 May 2018, 0:52:48 UTC - in response to Message 1934082.  
Last modified: 7 May 2018, 0:54:50 UTC

. . Probably just this machine then ...

Your Windows system has 1 GPU, and is doing CPU crunching, so that would mean a total of 200WUs.
According to your task list you've got 206.
So since you've got 6 more than the limit, and you're getting CPU work, I guess it won't let you have any more GPU work till those GPU Ghosts time out, more than 6 GPU WUs have been reported, or you re-claim them.

EDIT-
WTF? That system now shows 313 tasks in progress. 113 more than it should have.
Are you playing with the app_info.xml again?
Grant
Darwin NT
ID: 1934087 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1934120 - Posted: 7 May 2018, 4:59:15 UTC

Oh oh, No work available now ...
I got a pile of resends come in, then nothing now.
My guess is the splitters have been online for about 5 hours now and the server cache is empty.
It's hard to tell with the SSP frozen.
ID: 1934120 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13770
Credit: 208,696,464
RAC: 304
Australia
Message 1934122 - Posted: 7 May 2018, 5:09:08 UTC - in response to Message 1934120.  
Last modified: 7 May 2018, 5:10:32 UTC

Oh oh, No work available now ...

Yeah, just looked at my Event log and it started about 40 minutes ago.
Grant
Darwin NT
ID: 1934122 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1934123 - Posted: 7 May 2018, 5:10:59 UTC - in response to Message 1934122.  

Thanks for confirming this.
Mail sent to Master Eric.
ID: 1934123 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13770
Credit: 208,696,464
RAC: 304
Australia
Message 1934124 - Posted: 7 May 2018, 5:28:18 UTC - in response to Message 1934123.  

Thanks for confirming this.
Mail sent to Master Eric.

Something else for his morning todo list.
Grant
Darwin NT
ID: 1934124 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1934130 - Posted: 7 May 2018, 6:28:46 UTC - in response to Message 1934124.  

Made sure I was all stocked up on Einstein. Down to about a 100 gpu task left now. Won't last more than an hour more and that's it till morning I assume when the staff gets in and hopefully sorts out the splitters. Need to to get the SSP page current too.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1934130 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1934153 - Posted: 7 May 2018, 9:56:41 UTC
Last modified: 7 May 2018, 9:57:06 UTC

Last 6 CPU WU are running now, GPU allready empty, after that will shut down the host and save some electric power.
ID: 1934153 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51469
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1934173 - Posted: 7 May 2018, 12:51:52 UTC

Hope Eric or Jeff get up early today.
Meowsigh.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1934173 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1934178 - Posted: 7 May 2018, 13:07:17 UTC - in response to Message 1934173.  

Hope Eric or Jeff get up early today.

+ 1
In the mean time, all work done but still appearing Em progresso (51) in my host task list.
Sure they are ghosts, did anyone remember the post where the ghost DL sequence is explained?
ID: 1934178 · Report as offensive
Previous · 1 . . . 4 · 5 · 6 · 7 · 8 · 9 · 10 . . . 33 · Next

Message boards : Number crunching : Panic Mode On (112) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.