Panic Mode On (107) Server Problems?

Message boards : Number crunching : Panic Mode On (107) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 26 · 27 · 28 · 29

AuthorMessage
Profile Kissagogo27 Special Project $75 donor
Avatar

Send message
Joined: 6 Nov 99
Posts: 711
Credit: 8,032,827
RAC: 62
France
Message 1898057 - Posted: 29 Oct 2017, 12:49:09 UTC

it's always hard to believe that you can't have any work unit like AP but the server explain you that it have only Intel and Nvidia in stock .

how does it reserve some AP for some type of GPU, and don't transmute some from one type to each other ? .

29-Oct-2017 13:24:04 [SETI@home] Scheduler request completed: got 0 new tasks
29-Oct-2017 13:24:04 [SETI@home] No tasks sent
29-Oct-2017 13:24:04 [SETI@home] No tasks are available for AstroPulse v7
29-Oct-2017 13:24:04 [SETI@home] Tasks for NVIDIA GPU are available, but your preferences are set to not accept them
29-Oct-2017 13:24:04 [SETI@home] Tasks for Intel GPU are available, but your preferences are set to not accept them


is it a real state of the AP? or just a generik answer ?
ID: 1898057 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13155
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1898082 - Posted: 29 Oct 2017, 15:51:06 UTC - in response to Message 1898057.  
Last modified: 29 Oct 2017, 15:55:31 UTC

No that is just the weird wording the server uses to say it has NO work of any kind for you. My two fastest crunchers are way down in work this morning. Lots of no work is available or receive 1 task per request when I retire 10s of tasks between request intervals. Time to kick the servers again. All this problem with the servers started back in last December with a "fix" for ATI cards that didn't work.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1898082 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14504
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1898145 - Posted: 29 Oct 2017, 22:06:55 UTC
Last modified: 29 Oct 2017, 22:13:29 UTC

OK, see if you prefer this sequence.

29/10/2017 21:49:56 | SETI@home | Requesting new tasks for NVIDIA GPU
29/10/2017 21:49:58 | SETI@home | Scheduler request completed: got 0 new tasks
29/10/2017 21:49:58 | SETI@home | Project has no tasks available
29/10/2017 21:49:58 | SETI@home | Project requested delay of 303 seconds
...
29/10/2017 21:55:07 | SETI@home | Scheduler request completed: got 60 new tasks
My theory is that the BOINC server logic has a hard maximum limit om the number of tasks (all types, all platforms) that can be served per unit time. Each project will have a different limit depending on the precise project server configuration.

The limit is made up of the number of tasks held in the feeder cache at any given time, multiplied by the frequency of re-caching the feeder by database query. Once they're gone, they're gone.

You appear to be saying that SETI is teetering on the brink of this hard limit, and sometimes crosses it (Wednesday mornings come to mind). The only solution that comes to mind is to request work at a time of day when the rest of the world isn't, as in the example above.

Edit: since work requests are denominated in seconds, obviously the hard limit will be reached more quickly when there are VHAR shorties in the queue. As there are now, as a second computer is demonstrating.
ID: 1898145 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13155
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1898154 - Posted: 29 Oct 2017, 23:15:47 UTC - in response to Message 1898145.  

OK, see if you prefer this sequence.

29/10/2017 21:49:56 | SETI@home | Requesting new tasks for NVIDIA GPU
29/10/2017 21:49:58 | SETI@home | Scheduler request completed: got 0 new tasks
29/10/2017 21:49:58 | SETI@home | Project has no tasks available
29/10/2017 21:49:58 | SETI@home | Project requested delay of 303 seconds
...
29/10/2017 21:55:07 | SETI@home | Scheduler request completed: got 60 new tasks
My theory is that the BOINC server logic has a hard maximum limit om the number of tasks (all types, all platforms) that can be served per unit time. Each project will have a different limit depending on the precise project server configuration.

The limit is made up of the number of tasks held in the feeder cache at any given time, multiplied by the frequency of re-caching the feeder by database query. Once they're gone, they're gone.

You appear to be saying that SETI is teetering on the brink of this hard limit, and sometimes crosses it (Wednesday mornings come to mind). The only solution that comes to mind is to request work at a time of day when the rest of the world isn't, as in the example above.

I guess that would work if you only have your computer run SETI for one hour in the day. My computer runs non-stop and needs a full 300 gpu tasks every 2 1/2 hours. So it would be hard to schedule the computer to only ask for work whenever the rest of the world isn't asking.

My observation was that before the December attempted fix for ATI cards, we never ran into this issue of the servers not sending work on request when there is work available. The only observably effective solution is to make the servers think your machine is just now appearing to the server request queue. That makes the servers fill your deficit on the first request every time as long as the feeder buffer has tasks of the approved type for your gpu. Which lately, I might point out has been the case, with a very evenly balanced mix of Arecibo and GBT work.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1898154 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1898158 - Posted: 29 Oct 2017, 23:24:08 UTC - in response to Message 1898154.  
Last modified: 29 Oct 2017, 23:38:38 UTC


My observation was that before the December attempted fix for ATI cards, we never ran into this issue of the servers not sending work on request when there is work available. The only observably effective solution is to make the servers think your machine is just now appearing to the server request queue. That makes the servers fill your deficit on the first request every time as long as the feeder buffer has tasks of the approved type for your gpu. Which lately, I might point out has been the case, with a very evenly balanced mix of Arecibo and GBT work.


. . For what it is worth the problem seems far worse this morning. No amount of kicking of the pants nor tickling of the tonsils can elicit any new work from the servers ...

<edit> . . for what is worth my valid tasks have risen by over 50%, causing me to suspect there is some kind of log jam within SETI HQ.

Stephen

:(
ID: 1898158 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13155
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1898170 - Posted: 30 Oct 2017, 0:03:47 UTC - in response to Message 1898158.  

What I have observed today is that every machine has been operating at 75% cache allotment. And for some reason that I cannot explain, I have seen multiple occurrences of GPU backoff intervals. Those backoffs are generated by the servers. I can't see why that is occurring as I seem to eventually get work after several requests. The RTS buffer has over 600K tasks to send and I am not seeing any overwhelming evidence of over abundant Arecibo VLARs in the download mix.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1898170 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1898187 - Posted: 30 Oct 2017, 0:40:16 UTC - in response to Message 1898170.  

What I have observed today is that every machine has been operating at 75% cache allotment. And for some reason that I cannot explain, I have seen multiple occurrences of GPU backoff intervals. Those backoffs are generated by the servers. I can't see why that is occurring as I seem to eventually get work after several requests. The RTS buffer has over 600K tasks to send and I am not seeing any overwhelming evidence of over abundant Arecibo VLARs in the download mix.


. . Well whatever the cause right now Bertie is down to 40 tasks and will be out of work in half an hour. If the servers don't get sorted out soon this rig will gone. No point wasting power not knowing if/when there will be more work to process.

Stephen

:(
ID: 1898187 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1898206 - Posted: 30 Oct 2017, 1:46:10 UTC
Last modified: 30 Oct 2017, 2:01:21 UTC

All my machines are running LOW, can't get any new work. The creation rate is down to near ZERO, so I suppose SETI has stopped sending any new work. I wonder if the Bunkering trick will net any new work.

Well that worked the first time, the second time I got the old;
Sun Oct 29 21:55:53 2017 | SETI@home | [sched_op] NVIDIA GPU work request: 519923.13 seconds; 0.00 devices
Sun Oct 29 21:55:55 2017 | SETI@home | Scheduler request completed: got 0 new tasks
Sun Oct 29 21:55:55 2017 | SETI@home | Project has no tasks available

Results ready to send = = = 615,935

Since Nothing appears to convince the Server to send tasks...it may be time to Panic.
ID: 1898206 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1898207 - Posted: 30 Oct 2017, 2:10:48 UTC - in response to Message 1898206.  

All my machines are running LOW, can't get any new work. The creation rate is down to near ZERO, so I suppose SETI has stopped sending any new work. I wonder if the Bunkering trick will net any new work.

Well that worked the first time, the second time I got the old;
Sun Oct 29 21:55:53 2017 | SETI@home | [sched_op] NVIDIA GPU work request: 519923.13 seconds; 0.00 devices
Sun Oct 29 21:55:55 2017 | SETI@home | Scheduler request completed: got 0 new tasks
Sun Oct 29 21:55:55 2017 | SETI@home | Project has no tasks available

Results ready to send = = = 615,935

Since Nothing appears to convince the Server to send tasks...it may be time to Panic.


. . I think someone needs to kick the servers on site ...

. . I am out of work completely so shutting down for the interim ...

Stephen

:(
ID: 1898207 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13155
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1898208 - Posted: 30 Oct 2017, 2:12:19 UTC

Thanks for the comment. I just looked at the splitter output. Doesn't bode well for any work to appear and hold me over for the night. All machines down to 50% now. No work is available across all machines. Haven't tried bunkering yet. Might have to do so before bed. That is if there is any work to bunker.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1898208 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1898209 - Posted: 30 Oct 2017, 2:22:08 UTC - in response to Message 1898208.  

The first attempt netted 86 tasks. Every attempt since has netted nada.

It's Dead.
ID: 1898209 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1898215 - Posted: 30 Oct 2017, 4:04:31 UTC

Well, the Splitters kicked in and ran the RTS up to 700000. But, my machines are still half empty and keep being told,
Project has no tasks available
Hard to believe the Splitters just split another 100000 Arecibo VLARs. The 'Results received in last hour' keep steadily dropping.
ID: 1898215 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13155
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1898217 - Posted: 30 Oct 2017, 4:49:24 UTC

My linux machine has been out of gpu work for over an hour. It has been getting nothing but no work messages.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1898217 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13155
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1898219 - Posted: 30 Oct 2017, 5:08:59 UTC

Tried to kick the servers. Nothing works. Its Dead.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1898219 · Report as offensive
Profile Jeff Buck Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 11 Feb 00
Posts: 1441
Credit: 148,764,870
RAC: 0
United States
Message 1898220 - Posted: 30 Oct 2017, 5:14:56 UTC - in response to Message 1898217.  
Last modified: 30 Oct 2017, 5:15:53 UTC

For the third time recently, it looks like I'm in the same boat as y'all. I never used to have this problem. :^(

I'm not quite out of work yet on any boxes, but they're headed that way, and I'm headed for bed. So, I think I'm going to shut down my two main producers for the night, and load a zero-share backup project for the third, just in case. (That one's my bedroom space heater and the nights are getting chilly.) Doesn't look like Asteroids has any work either, so I guess MilkyWay is my next choice.
ID: 1898220 · Report as offensive
Profile arkayn
Volunteer tester
Avatar

Send message
Joined: 14 May 99
Posts: 4438
Credit: 55,006,323
RAC: 0
United States
Message 1898223 - Posted: 30 Oct 2017, 5:32:35 UTC

New thread started, please move over to that one.

https://setiathome.berkeley.edu/forum_thread.php?id=82102

ID: 1898223 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1898226 - Posted: 30 Oct 2017, 6:09:47 UTC - in response to Message 1898217.  

My linux machine has been out of gpu work for over an hour. It has been getting nothing but no work messages.


. . My main Linux cruncher has been down since noon here, no work at all ...

. . My first Linux cruncher is struggling through with a half filled cache and is getting tidbits once in a while, just enough to keep it online ...

Stephen

??
ID: 1898226 · Report as offensive
Previous · 1 . . . 26 · 27 · 28 · 29

Message boards : Number crunching : Panic Mode On (107) Server Problems?


 
©2022 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.