Panic Mode On (106) Server Problems?

Message boards : Number crunching : Panic Mode On (106) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 16 · 17 · 18 · 19 · 20 · 21 · 22 . . . 29 · Next

AuthorMessage
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1874586 - Posted: 23 Jun 2017, 5:44:58 UTC - in response to Message 1874585.  
Last modified: 23 Jun 2017, 5:47:47 UTC

Well, One machine got 79 while another got 5, the machine that was the lowest struck out.
Now it's back to nothing on all three except;
Fri Jun 23 01:41:09 2017 | SETI@home | Scheduler request completed: got 0 new tasks
Fri Jun 23 01:41:09 2017 | SETI@home | [sched_op] Server version 707
Fri Jun 23 01:41:09 2017 | SETI@home | No tasks sent
Fri Jun 23 01:41:09 2017 | SETI@home | Project requested delay of 303 seconds

That with just Seti@home v8 selected.
ID: 1874586 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13797
Credit: 208,696,464
RAC: 304
Australia
Message 1874587 - Posted: 23 Jun 2017, 5:50:57 UTC - in response to Message 1874586.  
Last modified: 23 Jun 2017, 5:51:37 UTC

That with just Seti@home v8 selected.

Up until this last week or so, I had been getting work with very few drops in my cache level after installing & enabling AP. Prior to that, this weeks cache running down to almost empty behavior had been the norm since the end of Dec last year.

I enabled the Work_fetch_debug in my Event Log, and got some work on the next request. Then none on the following one. But is certainly requesting the work. The Scheduler just doesn't want to hand it out.
Grant
Darwin NT
ID: 1874587 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1874589 - Posted: 23 Jun 2017, 6:08:37 UTC - in response to Message 1874587.  
Last modified: 23 Jun 2017, 6:27:02 UTC

The same two machines got some more and are now full; Fri Jun 23 01:56:50 2017 | SETI@home | Scheduler request completed: got 68 new tasks
The 3rd machine still hasn't gotten a single GPU task. I upped the cache to 1 day and the server sent 4 more Arecibo VLARs to a machine that really needs GPU tasks.
It's requesting nVidia tasks, the server just won't send any.

It just sent the machine 2 GPU tasks, it needs about 150.

Still nothing for this One machine, it takes about 7 to 8 hours to finish 200 GPU tasks, https://setiathome.berkeley.edu/results.php?hostid=6906726
Fri 23 Jun 2017 02:22:47 AM EDT | SETI@home | Reporting 1 completed tasks
Fri 23 Jun 2017 02:22:47 AM EDT | SETI@home | Requesting new tasks for NVIDIA
Fri 23 Jun 2017 02:22:47 AM EDT | SETI@home | [sched_op] CPU work request: 0.00 seconds; 0.00 devices
Fri 23 Jun 2017 02:22:47 AM EDT | SETI@home | [sched_op] NVIDIA work request: 162307.71 seconds; 0.00 devices
Fri 23 Jun 2017 02:22:49 AM EDT | SETI@home | Scheduler request completed: got 0 new tasks
Fri 23 Jun 2017 02:22:49 AM EDT | SETI@home | [sched_op] Server version 707
Fri 23 Jun 2017 02:22:49 AM EDT | SETI@home | No tasks sent
Fri 23 Jun 2017 02:22:49 AM EDT | SETI@home | Project requested delay of 303 seconds
ID: 1874589 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13797
Credit: 208,696,464
RAC: 304
Australia
Message 1874592 - Posted: 23 Jun 2017, 6:17:51 UTC

Looking at the graphs, it's showing that quite a few people are being affected at the moment.
In progress has been dropping off for over 24hrs, with the odd little (almost, but not quite) recovery spike. There was a steady decline for about 4 hours, and it's (mostly) leveled off over the last hour or so.

If they want more people to crunch more work, they need to make sure the servers give it out when the clients ask for more- particularly when they haven't reached either their cache settings, or the server side limits.
We can't process that which we can't get.
Grant
Darwin NT
ID: 1874592 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1874593 - Posted: 23 Jun 2017, 6:25:57 UTC - in response to Message 1874573.  

I am beginning to see the appeal that Zalster has for mostly, exclusively running Einstein.

Yes, but it usually lacks the excitement we have here with the amateure electrical hour. I say just let'm run and get whatever your resource shares are set to.


. . One Linux machine is getting almost normal supplies of new tasks so that is OK, don't ask me why. The second Linux machine is getting sporadic downloads but ATM has enough work to continue so that can keep going too. Both those machines are GPU only crunchers. The two windows machines have full CPU caches and completely empty GPU caches, and the servers will NOT send them any new work. There is no point in leaving them running under those conditions so they are getting turned off. What a waste of resources ... :(

Stephen

:(
ID: 1874593 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1874594 - Posted: 23 Jun 2017, 6:31:53 UTC - in response to Message 1874592.  

Looking at the graphs, it's showing that quite a few people are being affected at the moment.
In progress has been dropping off for over 24hrs, with the odd little (almost, but not quite) recovery spike. There was a steady decline for about 4 hours, and it's (mostly) leveled off over the last hour or so.

If they want more people to crunch more work, they need to make sure the servers give it out when the clients ask for more- particularly when they haven't reached either their cache settings, or the server side limits.
We can't process that which we can't get.


. . EXACTLY!!

. . Now just waiting for Wiggo to let us know he is still getting normal amounts of work on his 3 rigs ... :)

Stephen

:(
ID: 1874594 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 35525
Credit: 261,360,520
RAC: 489
Australia
Message 1874595 - Posted: 23 Jun 2017, 6:40:59 UTC

Yep this time there is a problem with the servers sending out work as I'm suffering this time.

Cheers.
ID: 1874595 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1874599 - Posted: 23 Jun 2017, 6:51:30 UTC

I'm down to half the GPU tasks I should be getting. I've had nothing but an occasional 1-2 GPU tasks downloaded after every 3rd or 4th request. The other requests get the:

Keith-Windows7

2574 SETI@home 6/22/2017 23:42:07 Requesting new tasks for CPU and NVIDIA GPU
2575 SETI@home 6/22/2017 23:42:10 Scheduler request completed: got 0 new tasks
2576 SETI@home 6/22/2017 23:42:10 No tasks sent
2577 SETI@home 6/22/2017 23:42:10 This computer has reached a limit on tasks in progress

message. I've tried all my tricks, nothing is working. Work_fetch_debug and sched_op_debug consistently show a Nvidia shortfall of tens of thousands of seconds. The scheduler just isn't recognizing the requests.

I've changed my project resources a little more towards Einstein. I'll continue to crunch whatever SETI GPU tasks I can get, but I am done with trying to coax the servers into sending me what I should have. It isn't worth the effort anymore.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1874599 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1874600 - Posted: 23 Jun 2017, 6:52:40 UTC - in response to Message 1874595.  

Yep this time there is a problem with the servers sending out work as I'm suffering this time.

Cheers.

Wow! Is that the first for you?? I thought client 6.10.60 was immune.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1874600 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 35525
Credit: 261,360,520
RAC: 489
Australia
Message 1874605 - Posted: 23 Jun 2017, 7:06:06 UTC - in response to Message 1874600.  
Last modified: 23 Jun 2017, 7:07:20 UTC

Yep this time there is a problem with the servers sending out work as I'm suffering this time.

Cheers.

Wow! Is that the first for you?? I thought client 6.10.60 was immune.

I think what's happening now is more than the usual other problem that you guys have been dealing with this time. ;-)

[edit] This for me usually says that something in serverland is about to fall over.

Cheers.
ID: 1874605 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13797
Credit: 208,696,464
RAC: 304
Australia
Message 1874607 - Posted: 23 Jun 2017, 7:39:15 UTC - in response to Message 1874592.  
Last modified: 23 Jun 2017, 7:39:30 UTC

Looking at the graphs, it's showing that quite a few people are being affected at the moment.
In progress has been dropping off for over 24hrs, with the odd little (almost, but not quite) recovery spike. There was a steady decline for about 4 hours, and it's (mostly) leveled off over the last hour or so.

It leveled off for a while there, but now In progress is back to getting lower & lower.
More & more people are getting less & less work.
Grant
Darwin NT
ID: 1874607 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1874618 - Posted: 23 Jun 2017, 10:59:42 UTC - in response to Message 1874595.  

Yep this time there is a problem with the servers sending out work as I'm suffering this time.

Cheers.


. . Hi Wiggo,

. . Dare I say ... at last! :)

Stephen

:)
ID: 1874618 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 35525
Credit: 261,360,520
RAC: 489
Australia
Message 1874636 - Posted: 23 Jun 2017, 12:55:15 UTC - in response to Message 1874618.  

Yep this time there is a problem with the servers sending out work as I'm suffering this time.

Cheers.


. . Hi Wiggo,

. . Dare I say ... at last! :)

Stephen

:)

Well it's the Borg (Athlon II X4 w/ dual 660's and winter time heater) that's having the big problems, but it also has just cleared over 270 ghost tasks over the last 48hrs which may have been part of its problem, though what I see is the splitters just idling away while I've been home (I have been out a few times for a few hours each time today) with no ramp up now and again as what usually happens.

I've thrown an overthrow on bed just in case for the night anyway. ;-)

Cheers.
ID: 1874636 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1874669 - Posted: 23 Jun 2017, 15:46:23 UTC - in response to Message 1874636.  


I've thrown an overthrow on bed just in case for the night anyway. ;-)

Cheers.

Ahh, fond memories. I wish that were my case. I've been dealing with +43° C. for the past two weeks. A/C blasting night and day .... still miserable.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1874669 · Report as offensive
Cruncher-American Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor

Send message
Joined: 25 Mar 02
Posts: 1513
Credit: 370,893,186
RAC: 340
United States
Message 1874726 - Posted: 23 Jun 2017, 21:42:48 UTC - in response to Message 1874669.  


I've thrown an overthrow on bed just in case for the night anyway. ;-)

Cheers.

Ahh, fond memories. I wish that were my case. I've been dealing with +43° C. for the past two weeks. A/C blasting night and day .... still miserable.


Do you live in Phoenix, perchance? My son, who is in the AF at Luke, says it is in the 110-120F range most every day...
ID: 1874726 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13797
Credit: 208,696,464
RAC: 304
Australia
Message 1874739 - Posted: 23 Jun 2017, 22:11:22 UTC

Still a huge number of Arecibo VLARs about, and still barely a sign of any GBT work (some coming through, but only every now and then when actually able to get some work).
Grant
Darwin NT
ID: 1874739 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1874755 - Posted: 23 Jun 2017, 23:17:46 UTC - in response to Message 1874669.  


I've thrown an overthrow on bed just in case for the night anyway. ;-)

Cheers.

Ahh, fond memories. I wish that were my case. I've been dealing with +43° C. for the past two weeks. A/C blasting night and day .... still miserable.


. . That's a scorcher alright. Not often we get prolonged periods at those sorts of temps here. Usually just in the peak of summer, as in last summer which was a real blisterer.

Stephen

:(
ID: 1874755 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1874758 - Posted: 23 Jun 2017, 23:32:25 UTC - in response to Message 1874726.  


I've thrown an overthrow on bed just in case for the night anyway. ;-)

Cheers.

Ahh, fond memories. I wish that were my case. I've been dealing with +43° C. for the past two weeks. A/C blasting night and day .... still miserable.


Do you live in Phoenix, perchance? My son, who is in the AF at Luke, says it is in the 110-120F range most every day...

No, Northern Sacramento Valley in California. Been solidly past 105° F for the past week. Hit 111° F this Wednesday. Hasn't gone below 80° F at night. Too far north to get any of the Delta breeze from Sacramento and plenty of the adiabatic winds flowing south from Redding.

They have to cancel flights out of Sky Harbor airport in Phoenix every summer because it is too hot for planes to take off. No lift.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1874758 · Report as offensive
Cruncher-American Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor

Send message
Joined: 25 Mar 02
Posts: 1513
Credit: 370,893,186
RAC: 340
United States
Message 1874786 - Posted: 24 Jun 2017, 1:51:33 UTC - in response to Message 1874758.  


They have to cancel flights out of Sky Harbor airport in Phoenix every summer because it is too hot for planes to take off. No lift.


He told me that, and I found it hard to believe. Thanks for the confirmation!
ID: 1874786 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1874835 - Posted: 24 Jun 2017, 10:56:27 UTC

<coughing_up_dust> There goes my last NVidia task for my 1080s </coughing_up_dust>

I'm officially dry for the first time in a long time.
More beta tasks to do though :D
ID: 1874835 · Report as offensive
Previous · 1 . . . 16 · 17 · 18 · 19 · 20 · 21 · 22 . . . 29 · Next

Message boards : Number crunching : Panic Mode On (106) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.