Panic Mode On (104) Server Problems?

Message boards : Number crunching : Panic Mode On (104) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 10 · 11 · 12 · 13 · 14 · 15 · 16 . . . 42 · Next

AuthorMessage
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13848
Credit: 208,696,464
RAC: 304
Australia
Message 1842275 - Posted: 14 Jan 2017, 22:28:03 UTC

Server Status page shows everything's good, but the cache on my main cruncher is running down again. Most requests for work result in "Project has no tasks available" messages and those that do get work don't get enough to make up the developing deficit.
Grant
Darwin NT
ID: 1842275 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51477
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1842277 - Posted: 14 Jan 2017, 22:44:56 UTC - in response to Message 1842275.  

Server Status page shows everything's good, but the cache on my main cruncher is running down again. Most requests for work result in "Project has no tasks available" messages and those that do get work don't get enough to make up the developing deficit.

Not sure what the problem could be there.
I am just a few WUs short of my 2600 cache limit across my 8 rigs.
There are still a lot of the Aerecibo shorties out in the field, so lots of work requests to fill.
I suspect it's just the luck of the draw to hit the scheduler when it has a full bucket.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1842277 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13848
Credit: 208,696,464
RAC: 304
Australia
Message 1842280 - Posted: 14 Jan 2017, 22:56:57 UTC - in response to Message 1842277.  
Last modified: 14 Jan 2017, 23:03:30 UTC

Hmm.
Results received in the last hour is at 111,780. At the height of the Arecibo only work I think I noticed it hit around 15,250/hr- it could be that all the systems that ran out of work now have some, so their Manager backoffs are no longer stopping them from requesting work, hence increasing the demand on the server even though the amount of work being returned has dropped off.

It could be we're looking at another system limit. The project wants more crunchers to process the work, but if it can't continue to support the current load that's just going to lead to more unhappy crunchers that won't hang around and head for greener pastures (ie more work, and more credit).
Time for another Feeder, or increase the size of the present one (if it's possible)? Or is it just a case of something else on Synergy soaking up CPU cycles & inhibiting it's output?


EDIT- just looked at my Event Log and where Scheduler requests usually get a response in 3-4 seconds, at the moment it can take as long as 10 seconds, and has been that way for at least the last 8 hours (as far back as my Event Log goes).
Grant
Darwin NT
ID: 1842280 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51477
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1842289 - Posted: 14 Jan 2017, 23:26:39 UTC

The other thing that sometimes happens is the work being split creates almost all VLAR tasks, which are suppressed from being sent to nvidia GPUs.
So the ready to send cache can be full, but will not be sent to those cards.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1842289 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 36639
Credit: 261,360,520
RAC: 489
Australia
Message 1842290 - Posted: 14 Jan 2017, 23:32:47 UTC - in response to Message 1842289.  

The other thing that sometimes happens is the work being split creates almost all VLAR tasks, which are suppressed from being sent to nvidia GPUs.
So the ready to send cache can be full, but will not be sent to those cards.

I'll amend that to, "Arecibo VLAR's won't be sent to GPU's", as GBT VLAR's are sent to GPU's. ;-)

Cheers.
ID: 1842290 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13848
Credit: 208,696,464
RAC: 304
Australia
Message 1842291 - Posted: 14 Jan 2017, 23:34:38 UTC - in response to Message 1842289.  

The other thing that sometimes happens is the work being split creates almost all VLAR tasks, which are suppressed from being sent to nvidia GPUs.
So the ready to send cache can be full, but will not be sent to those cards.

Yep.
But it's been almost a day or so since I got an Arecibo VLAR on my CPU.

Since my initial post, my system picked up enough work to fill the cache, but then the next 6 requests for work have resulted in none so it's not as full as it was...



I've done a speed test on my net connection, and it's not as healthy as it should be- uploads are actually faster than downloads at the moment and there's quite a bit of latency & jitter which doesn't help things.
Grant
Darwin NT
ID: 1842291 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51477
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1842293 - Posted: 14 Jan 2017, 23:39:40 UTC - in response to Message 1842290.  

The other thing that sometimes happens is the work being split creates almost all VLAR tasks, which are suppressed from being sent to nvidia GPUs.
So the ready to send cache can be full, but will not be sent to those cards.

I'll amend that to, "Arecibo VLAR's won't be sent to GPU's", as GBT VLAR's are sent to GPU's. ;-)

Cheers.

That would be correct.
Thanks.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1842293 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13848
Credit: 208,696,464
RAC: 304
Australia
Message 1842336 - Posted: 15 Jan 2017, 6:18:45 UTC - in response to Message 1842293.  
Last modified: 15 Jan 2017, 6:34:42 UTC

Really struggling to get work here.
15/01/2017 14:43:25 | SETI@home | Sending scheduler request: To fetch work.
15/01/2017 14:43:25 | SETI@home | Reporting 3 completed tasks
15/01/2017 14:43:25 | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
15/01/2017 14:43:32 | SETI@home | Scheduler request completed: got 6 new tasks

15/01/2017 14:48:38 | SETI@home | Sending scheduler request: To fetch work.
15/01/2017 14:48:38 | SETI@home | Reporting 2 completed tasks
15/01/2017 14:48:38 | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
15/01/2017 14:48:46 | SETI@home | Scheduler request completed: got 0 new tasks
15/01/2017 14:48:46 | SETI@home | Project has no tasks available

15/01/2017 14:53:51 | SETI@home | Sending scheduler request: To fetch work.
15/01/2017 14:53:51 | SETI@home | Reporting 2 completed tasks
15/01/2017 14:53:51 | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
15/01/2017 14:53:59 | SETI@home | Scheduler request completed: got 0 new tasks
15/01/2017 14:53:59 | SETI@home | Project has no tasks available

15/01/2017 14:59:05 | SETI@home | Sending scheduler request: To fetch work.
15/01/2017 14:59:05 | SETI@home | Reporting 2 completed tasks
15/01/2017 14:59:05 | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
15/01/2017 14:59:14 | SETI@home | Scheduler request completed: got 0 new tasks
15/01/2017 14:59:14 | SETI@home | Project has no tasks available

15/01/2017 15:04:20 | SETI@home | Sending scheduler request: To fetch work.
15/01/2017 15:04:20 | SETI@home | Reporting 9 completed tasks
15/01/2017 15:04:20 | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
15/01/2017 15:04:27 | SETI@home | Scheduler request completed: got 0 new tasks
15/01/2017 15:04:27 | SETI@home | Project has no tasks available

15/01/2017 15:09:32 | SETI@home | Sending scheduler request: To fetch work.
15/01/2017 15:09:32 | SETI@home | Reporting 2 completed tasks
15/01/2017 15:09:32 | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
15/01/2017 15:09:40 | SETI@home | Scheduler request completed: got 0 new tasks
15/01/2017 15:09:40 | SETI@home | Project has no tasks available

15/01/2017 15:14:46 | SETI@home | Sending scheduler request: To fetch work.
15/01/2017 15:14:46 | SETI@home | Reporting 3 completed tasks
15/01/2017 15:14:46 | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
15/01/2017 15:14:56 | SETI@home | Scheduler request completed: got 0 new tasks
15/01/2017 15:14:56 | SETI@home | Project has no tasks available

15/01/2017 15:20:01 | SETI@home | Sending scheduler request: To fetch work.
15/01/2017 15:20:01 | SETI@home | Reporting 3 completed tasks
15/01/2017 15:20:01 | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
15/01/2017 15:20:08 | SETI@home | Scheduler request completed: got 0 new tasks
15/01/2017 15:20:08 | SETI@home | Project has no tasks available

15/01/2017 15:25:14 | SETI@home | Sending scheduler request: To fetch work.
15/01/2017 15:25:14 | SETI@home | Reporting 3 completed tasks
15/01/2017 15:25:14 | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
15/01/2017 15:25:22 | SETI@home | Scheduler request completed: got 0 new tasks
15/01/2017 15:25:22 | SETI@home | Project has no tasks available

15/01/2017 15:30:27 | SETI@home | Sending scheduler request: To fetch work.
15/01/2017 15:30:27 | SETI@home | Reporting 1 completed tasks
15/01/2017 15:30:27 | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
15/01/2017 15:30:33 | SETI@home | Scheduler request completed: got 0 new tasks
15/01/2017 15:30:33 | SETI@home | Project has no tasks available

15/01/2017 15:35:38 | SETI@home | Sending scheduler request: To fetch work.
15/01/2017 15:35:38 | SETI@home | Reporting 3 completed tasks
15/01/2017 15:35:38 | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
15/01/2017 15:35:45 | SETI@home | Scheduler request completed: got 0 new tasks
15/01/2017 15:35:45 | SETI@home | Project has no tasks available

15/01/2017 15:40:51 | SETI@home | Sending scheduler request: To fetch work.
15/01/2017 15:40:51 | SETI@home | Reporting 3 completed tasks
15/01/2017 15:40:51 | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
15/01/2017 15:40:59 | SETI@home | Scheduler request completed: got 0 new tasks
15/01/2017 15:40:59 | SETI@home | Project has no tasks available

15/01/2017 15:46:04 | SETI@home | Sending scheduler request: To fetch work.
15/01/2017 15:46:04 | SETI@home | Reporting 3 completed tasks
15/01/2017 15:46:04 | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
15/01/2017 15:46:13 | SETI@home | Scheduler request completed: got 0 new tasks
15/01/2017 15:46:13 | SETI@home | Project has no tasks available

15/01/2017 15:51:19 | SETI@home | Sending scheduler request: To fetch work.
15/01/2017 15:51:19 | SETI@home | Reporting 4 completed tasks
15/01/2017 15:51:19 | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
15/01/2017 15:51:27 | SETI@home | Scheduler request completed: got 0 new tasks
15/01/2017 15:51:27 | SETI@home | Project has no tasks available

15/01/2017 15:56:33 | SETI@home | Sending scheduler request: To fetch work.
15/01/2017 15:56:33 | SETI@home | Reporting 2 completed tasks
15/01/2017 15:56:33 | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
15/01/2017 15:56:41 | SETI@home | Scheduler request completed: got 0 new tasks
15/01/2017 15:56:41 | SETI@home | Project has no tasks available

Finally!
15/01/2017 16:01:51 | SETI@home | Sending scheduler request: To fetch work.
15/01/2017 16:01:51 | SETI@home | Reporting 4 completed tasks
15/01/2017 16:01:51 | SETI@home | Requesting new tasks for CPU and NVIDIA GPU
15/01/2017 16:01:55 | SETI@home | Scheduler request completed: got 51 new tasks

Still not enough to fill the cache, but less likely to run out in the very near future.
Grant
Darwin NT
ID: 1842336 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 36639
Credit: 261,360,520
RAC: 489
Australia
Message 1842343 - Posted: 15 Jan 2017, 7:59:23 UTC

I wonder if it's just bad timing or your BOINC version as I've been having no such issues here Grant.

Cheers.
ID: 1842343 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1842345 - Posted: 15 Jan 2017, 8:12:54 UTC - in response to Message 1842343.  
Last modified: 15 Jan 2017, 8:13:23 UTC

I wonder if it's just bad timing or your BOINC version as I've been having no such issues here Grant.

Cheers.


. . FWIW I am running 7.6.33 and while I was having similar problems earlier right now it seems to be OK.

Stephen

:)

[edit] touch wood
ID: 1842345 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 36639
Credit: 261,360,520
RAC: 489
Australia
Message 1842347 - Posted: 15 Jan 2017, 8:23:35 UTC - in response to Message 1842345.  
Last modified: 15 Jan 2017, 8:26:28 UTC

I wonder if it's just bad timing or your BOINC version as I've been having no such issues here Grant.

Cheers.


. . FWIW I am running 7.6.33 and while I was having similar problems earlier right now it seems to be OK.

Stephen

:)

[edit] touch wood

It could very well be those excessive back off times that later versions than mine have causing your problems.

Between that, replacing the Messages tab with a Notice tab and the flakiness of many later versions I decided to stick with what works best for me (I hate being a guinea pig).

[edit] It'll take at least the first 2 fixed and many good reports on the 3rd to convince me to think about upgrading.

Cheers.
ID: 1842347 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13848
Credit: 208,696,464
RAC: 304
Australia
Message 1842353 - Posted: 15 Jan 2017, 8:49:01 UTC - in response to Message 1842347.  

It could very well be those excessive back off times that later versions than mine have causing your problems.

Not backoff related as I haven't (yet) run out of work.
But for some reason I ask for work, and there is none available. The last hour or so it hasn't taken nearly as many requests to finally get work, but it's certainly not available with every request the way it usually is. Normally I can report 2, get 2. Report 5, get 5. Report 1, get 1.
Grant
Darwin NT
ID: 1842353 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 36639
Credit: 261,360,520
RAC: 489
Australia
Message 1842357 - Posted: 15 Jan 2017, 9:05:24 UTC - in response to Message 1842353.  

It could very well be those excessive back off times that later versions than mine have causing your problems.

Not backoff related as I haven't (yet) run out of work.
But for some reason I ask for work, and there is none available. The last hour or so it hasn't taken nearly as many requests to finally get work, but it's certainly not available with every request the way it usually is. Normally I can report 2, get 2. Report 5, get 5. Report 1, get 1.

Well I can't think of any other reason Grant, as I just havn't suffered depleted caches except for the regular outrages here.

Cheers.
ID: 1842357 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13848
Credit: 208,696,464
RAC: 304
Australia
Message 1842358 - Posted: 15 Jan 2017, 9:10:00 UTC - in response to Message 1842357.  

Well I can't think of any other reason Grant, as I just havn't suffered depleted caches except for the regular outrages here.

Yeah.
It's odd.

My download speed is way off what it should be, and jitter & latency way up, but with those sort of issues if they cause a problem usually result in Scheduler timeouts, not just slightly longer than usual waits for a response, and the response being "No work available."
Grant
Darwin NT
ID: 1842358 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1842359 - Posted: 15 Jan 2017, 9:11:45 UTC - in response to Message 1842347.  

I wonder if it's just bad timing or your BOINC version as I've been having no such issues here Grant.

Cheers.


. . FWIW I am running 7.6.33 and while I was having similar problems earlier right now it seems to be OK.

Stephen

:)

[edit] touch wood

It could very well be those excessive back off times that later versions than mine have causing your problems.

Between that, replacing the Messages tab with a Notice tab and the flakiness of many later versions I decided to stick with what works best for me (I hate being a guinea pig).

[edit] It'll take at least the first 2 fixed and many good reports on the 3rd to convince me to think about upgrading.

Cheers.


. . Again, FWIW I can't say I was aware of any problems with 3.6.22, but 3.6.33 is maybe a little flaky at times, though I can't put my finger on it :)

Stephen

:)
ID: 1842359 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1842369 - Posted: 15 Jan 2017, 10:13:21 UTC - in response to Message 1842353.  

Grant, it's nothing to worry about, the scheduler is
1). Just full of vlar's at that time,
2). I was just there 2 seconds before you and emptied it :)
ID: 1842369 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13848
Credit: 208,696,464
RAC: 304
Australia
Message 1842370 - Posted: 15 Jan 2017, 10:26:26 UTC - in response to Message 1842369.  

Grant, it's nothing to worry about, the scheduler is
1). Just full of vlar's at that time,

I haven't had any Arecibo VLARs today at all.
Even in Arecibo VLAR storms it hasn't taken more than about 6 requests to get work.

2). I was just there 2 seconds before you and emptied it :)

For the last 1hr 30min? I've got 2 WUs in that time.
Before that it was just over an hour before getting any WUs.
Grant
Darwin NT
ID: 1842370 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1842371 - Posted: 15 Jan 2017, 10:34:14 UTC - in response to Message 1842370.  

yea, about and hour ago I was 1/2 empty, then poof, 111 tasks, now all is fine. It's just luck of the draw when I need 3-9 per request.
ID: 1842371 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13848
Credit: 208,696,464
RAC: 304
Australia
Message 1842372 - Posted: 15 Jan 2017, 10:39:30 UTC - in response to Message 1842371.  

yea, about and hour ago I was 1/2 empty, then poof, 111 tasks, now all is fine. It's just luck of the draw when I need 3-9 per request.

At least i'm not the only one having problems getting regular work.
Grant
Darwin NT
ID: 1842372 · Report as offensive
Bruce
Volunteer tester

Send message
Joined: 15 Mar 02
Posts: 123
Credit: 124,955,234
RAC: 11
United States
Message 1842375 - Posted: 15 Jan 2017, 10:55:13 UTC - in response to Message 1842372.  

yea, about and hour ago I was 1/2 empty, then poof, 111 tasks, now all is fine. It's just luck of the draw when I need 3-9 per request.

At least i'm not the only one having problems getting regular work.

Grant you are not the only one having trouble getting work!
I'm down to about 30% of work units in my cache, that is only about a couple of hours worth.
I keep getting the same as you.

0 tasks sent
project has no tasks available.

Even if there is a flood of Arecibo VLARs, where is all of the GBT work? The RTS says that it is full.
I don't get it. According to SSP it doesn't look like there should be a problem.
Bruce
ID: 1842375 · Report as offensive
Previous · 1 . . . 10 · 11 · 12 · 13 · 14 · 15 · 16 . . . 42 · Next

Message boards : Number crunching : Panic Mode On (104) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.