Panic Mode On (89) Server Problems?

Message boards : Number crunching : Panic Mode On (89) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 8 · 9 · 10 · 11 · 12 · 13 · 14 . . . 24 · Next

AuthorMessage
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1566556 - Posted: 3 Sep 2014, 18:04:11 UTC - in response to Message 1566331.  
Last modified: 3 Sep 2014, 18:04:39 UTC

In all of my venues, I have the three GPU types unchecked. Neither of my rigs have usable GPUs anyway.

I didn't change anything on my end, and the only difference I see now is server version changed from 703 to 705. My other host has gotten work since after the change to 705.

I did some testing on one of my machines with two old versions.
6.2.19 - Requested work & was not given any. Just as you are seeing.
6.6.33 - Requested work & was given work.

So you are not crazy, well not completely at least... The server code changes have borked you. Defiantly time to bring it to the attention of the BOINC devs, or switch to a version < 6 years old.

I was a bit bored today so I gave some other versions a try.
6.4.5 - Requested work & was not given any.
6.6.0 - Requested work & was not given any.

Looks like there is a specific build of 6.6.x that is the dividing live between working and not. The 6.6.0 version is listed a a development build, but I wanted to try it anyway. To see if all 6.6.x build worked.

IIRC wasn't GPU processing support added in a 6.6.x build? Given they were messing around with the GPU scheduling bits of the server code perhaps the first version to support GPUs will get work when requested.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1566556 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1566559 - Posted: 3 Sep 2014, 18:15:37 UTC - in response to Message 1566549.  

Now for a change of pace.. it looks like the candlelight vigil for the 30 and 31 tapes might come to an end soon. I wonder if that is deliberate. At least one or two of those tapes will get split up, maybe not all of them, but.. there is hope for them after all!

I've been watching that all morning....wondering what will start splitting next or if AP will jump in again before the splitters get there.

When I looked this morning we were at ~130 channels to do. Now 6 hours later we are at 116. If that is a consistent rate we have about 2 more days before those are done.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1566559 · Report as offensive
Claggy
Volunteer tester

Send message
Joined: 5 Jul 99
Posts: 4654
Credit: 47,537,079
RAC: 4
United Kingdom
Message 1566560 - Posted: 3 Sep 2014, 18:15:56 UTC - in response to Message 1566474.  

David has done a checkout to hopefully fix this:

http://boinc.berkeley.edu/gitweb/?p=boinc-v2.git;a=commit;h=60a35821517fb686087aa56bd6dc86134d313a85

scheduler: if client reports zero elapsed time, set ET to CPU time

Just need a new scheduler now.

Claggy
ID: 1566560 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1566564 - Posted: 3 Sep 2014, 18:30:08 UTC - in response to Message 1566556.  

In all of my venues, I have the three GPU types unchecked. Neither of my rigs have usable GPUs anyway.

I didn't change anything on my end, and the only difference I see now is server version changed from 703 to 705. My other host has gotten work since after the change to 705.

I did some testing on one of my machines with two old versions.
6.2.19 - Requested work & was not given any. Just as you are seeing.
6.6.33 - Requested work & was given work.

So you are not crazy, well not completely at least... The server code changes have borked you. Defiantly time to bring it to the attention of the BOINC devs, or switch to a version < 6 years old.

I was a bit bored today so I gave some other versions a try.
6.4.5 - Requested work & was not given any.
6.6.0 - Requested work & was not given any.

Looks like there is a specific build of 6.6.x that is the dividing live between working and not. The 6.6.0 version is listed a a development build, but I wanted to try it anyway. To see if all 6.6.x build worked.

IIRC wasn't GPU processing support added in a 6.6.x build? Given they were messing around with the GPU scheduling bits of the server code perhaps the first version to support GPUs will get work when requested.

The (very ancient) CUDA FAQs say v6.4.4 was the minimum requirement (I'd remembered v6.4.5), but we went through some rapid changes as it settled down.

The powers-that-be are sniffing around, and reckon they've fixed part of the problem, but not that bit yet.
ID: 1566564 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1566663 - Posted: 3 Sep 2014, 21:56:07 UTC - in response to Message 1566559.  

Now for a change of pace.. it looks like the candlelight vigil for the 30 and 31 tapes might come to an end soon. I wonder if that is deliberate. At least one or two of those tapes will get split up, maybe not all of them, but.. there is hope for them after all!

I've been watching that all morning....wondering what will start splitting next or if AP will jump in again before the splitters get there.

When I looked this morning we were at ~130 channels to do. Now 6 hours later we are at 116. If that is a consistent rate we have about 2 more days before those are done.

And it looks like 29no13aa has been resumed. There's hope.. let's see how many of those 30 and 31 tapes can get started before more tapes get loaded.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1566663 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1566666 - Posted: 3 Sep 2014, 21:59:44 UTC - in response to Message 1566663.  

Now for a change of pace.. it looks like the candlelight vigil for the 30 and 31 tapes might come to an end soon. I wonder if that is deliberate. At least one or two of those tapes will get split up, maybe not all of them, but.. there is hope for them after all!

I've been watching that all morning....wondering what will start splitting next or if AP will jump in again before the splitters get there.

When I looked this morning we were at ~130 channels to do. Now 6 hours later we are at 116. If that is a consistent rate we have about 2 more days before those are done.

And it looks like 29no13aa has been resumed. There's hope.. let's see how many of those 30 and 31 tapes can get started before more tapes get loaded.

There is another spike on the inbound data transfer. So they might be loading up a second round of data sets. I think last time they did that we had a ton of them to go through.
Hopefully we can manage to burn off some of the high numbered sets before the new data gets loaded in for splitting.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1566666 · Report as offensive
Profile William
Volunteer tester
Avatar

Send message
Joined: 14 Feb 13
Posts: 2037
Credit: 17,689,662
RAC: 0
Message 1566681 - Posted: 3 Sep 2014, 22:19:21 UTC - in response to Message 1566560.  

David has done a checkout to hopefully fix this:

http://boinc.berkeley.edu/gitweb/?p=boinc-v2.git;a=commit;h=60a35821517fb686087aa56bd6dc86134d313a85

scheduler: if client reports zero elapsed time, set ET to CPU time

Just need a new scheduler now.

Claggy

Well that bit of code certainly improved ;D

If that does the trick, fine.
Personally I doubt it.

it's like watching the ripples on the water and trying to make out what shape the object was that was dropped in... you'd have to find the impact location and dive for it.
Unfortunately I have no time whatsoever to spemd on this atm. If such a thing was possible I'd be in negative time.
A person who won't read has no advantage over one who can't read. (Mark Twain)
ID: 1566681 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1566697 - Posted: 3 Sep 2014, 22:56:19 UTC - in response to Message 1566681.  

David has done a checkout to hopefully fix this:

http://boinc.berkeley.edu/gitweb/?p=boinc-v2.git;a=commit;h=60a35821517fb686087aa56bd6dc86134d313a85

scheduler: if client reports zero elapsed time, set ET to CPU time

Just need a new scheduler now.

Claggy

Well that bit of code certainly improved ;D

If that does the trick, fine.
Personally I doubt it.

it's like watching the ripples on the water and trying to make out what shape the object was that was dropped in... you'd have to find the impact location and dive for it.
Unfortunately I have no time whatsoever to spemd on this atm. If such a thing was possible I'd be in negative time.

We might have a solution:

There was a bug in the new server code
that prevented it from sending jobs to pre-6.7 clients.
I checked in a fix (4 lines in sched/sched_types.h).

Jeff, can you deploy this in beta and public?

Thanks -- David

Could those affected keep an eye on this, and let us know what transpires - pro or anti? Thanks.
ID: 1566697 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1566739 - Posted: 4 Sep 2014, 0:35:39 UTC
Last modified: 4 Sep 2014, 0:37:07 UTC

And AP work is being split once again.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1566739 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1566755 - Posted: 4 Sep 2014, 1:15:24 UTC - in response to Message 1566739.  

We might have a solution:
There was a bug in the new server code that prevented it from sending jobs to pre-6.7 clients. I checked in a fix (4 lines in sched/sched_types.h). Jeff, can you deploy this in beta and public?
Thanks -- David
Could those affected keep an eye on this, and let us know what transpires - pro or anti? Thanks.

Will do. So far there's no change, but I don't know when it will be deployed.




And AP work is being split once again.

More importantly.. two low-numbered tapes got loaded and 29no13aa that WAS being split some more got interrupted and the splitters appear to have jumped to the new tapes. So far, just two tapes got loaded.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1566755 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1566759 - Posted: 4 Sep 2014, 1:25:51 UTC - in response to Message 1566755.  

We might have a solution:
There was a bug in the new server code that prevented it from sending jobs to pre-6.7 clients. I checked in a fix (4 lines in sched/sched_types.h). Jeff, can you deploy this in beta and public?
Thanks -- David
Could those affected keep an eye on this, and let us know what transpires - pro or anti? Thanks.

Will do. So far there's no change, but I don't know when it will be deployed.




And AP work is being split once again.

More importantly.. two low-numbered tapes got loaded and 29no13aa that WAS being split some more got interrupted and the splitters appear to have jumped to the new tapes. So far, just two tapes got loaded.

Yeah at the rate it is going 29no13aa will be done sometime in November,

So far 1 AP channel ended in error. Which is much better than the last set of data. They might be loading in less at a time now to see how the AP splitters handle it, or it could be a different reason all together.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1566759 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1566761 - Posted: 4 Sep 2014, 1:27:57 UTC
Last modified: 4 Sep 2014, 1:30:06 UTC

Has anyone noticed the server is sending out CPU tasks even if your GPUs don't have any work? All 3 of my Hosts are receiving CPU work even though they are not needed. I have a Mac whose 3 GPUs are going idle while the server sends it CPU work. If I change the preferences to GPU only, the server doesn't send any work :-(
ID: 1566761 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1566765 - Posted: 4 Sep 2014, 1:45:07 UTC - in response to Message 1566761.  
Last modified: 4 Sep 2014, 1:46:45 UTC

Has anyone noticed the server is sending out CPU tasks even if your GPUs don't have any work? All 3 of my Hosts are receiving CPU work even though they are not needed. I have a Mac whose 3 GPUs are going idle while the server sends it CPU work. If I change the preferences to GPU only, the server doesn't send any work :-(

On my HTPC I am getting.
SETI@home Tasks for AMD/ATI GPU are available, but your preferences are set to not accept them
SETI@home This computer has reached a limit on tasks in progress

Despite the fact it is in the only venue that allows ATI work. It has 7 GPU & 89 CPU tasks.
I figure they might still be playing around with the schedule stuff.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1566765 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11361
Credit: 29,581,041
RAC: 66
United States
Message 1566766 - Posted: 4 Sep 2014, 1:45:48 UTC - in response to Message 1566739.  

And AP work is being split once again.

And not much.
ID: 1566766 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1566767 - Posted: 4 Sep 2014, 2:01:51 UTC - in response to Message 1566765.  
Last modified: 4 Sep 2014, 2:11:07 UTC

Has anyone noticed the server is sending out CPU tasks even if your GPUs don't have any work? All 3 of my Hosts are receiving CPU work even though they are not needed. I have a Mac whose 3 GPUs are going idle while the server sends it CPU work. If I change the preferences to GPU only, the server doesn't send any work :-(

On my HTPC I am getting.
SETI@home Tasks for AMD/ATI GPU are available, but your preferences are set to not accept them
SETI@home This computer has reached a limit on tasks in progress

Despite the fact it is in the only venue that allows ATI work. It has 7 GPU & 89 CPU tasks.
I figure they might still be playing around with the schedule stuff.

All CPU tasks;
http://setiathome.berkeley.edu/results.php?hostid=7258715
http://setiathome.berkeley.edu/results.php?hostid=6796475&offset=0&show_names=0&state=0&appid=12
http://setiathome.berkeley.edu/results.php?hostid=6796479

I had to Force the Mac to resend CPU task as GPU tasks or else it would be GPU Idle. Now that it has downloaded a few more CPU tasks, it has Stopped downloading Any Tasks. Soon the GPUs will be Idle again.
ID: 1566767 · Report as offensive
Profile Jeff Buck Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 11 Feb 00
Posts: 1441
Credit: 148,764,870
RAC: 0
United States
Message 1566768 - Posted: 4 Sep 2014, 2:03:45 UTC - in response to Message 1566761.  
Last modified: 4 Sep 2014, 2:06:49 UTC

Has anyone noticed the server is sending out CPU tasks even if your GPUs don't have any work? All 3 of my Hosts are receiving CPU work even though they are not needed. I have a Mac whose 3 GPUs are going idle while the server sends it CPU work. If I change the preferences to GPU only, the server doesn't send any work :-(

What I'm seeing over the last several days is that sometimes a work fetch for both CPU and GPU will only return one or more VLARs for the CPU, but nothing for the GPU. However, usually the next request will fill up the GPU queue, too. It seems as if when the scheduler only finds VLARs at the top of its RTS buffer, it sometimes doesn't dig very deep to find non-VLARs for the GPU if it's pressed for time. I haven't had any hosts stop getting GPU tasks altogether.

Edit: Ah, I see you're talking about AP-only hosts. Never mind! AP tasks are rare in my world.
ID: 1566768 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1566775 - Posted: 4 Sep 2014, 2:19:00 UTC - in response to Message 1566768.  
Last modified: 4 Sep 2014, 2:21:36 UTC

This is what my Mac is getting right now,

Wed Sep  3 21:54:52 2014 | SETI@home | Sending scheduler request: To fetch work.
Wed Sep  3 21:54:52 2014 | SETI@home | Reporting 1 completed tasks
Wed Sep  3 21:54:52 2014 | SETI@home | Requesting new tasks for CPU and ATI
Wed Sep  3 21:54:53 2014 | SETI@home | Scheduler request completed: got 0 new tasks
Wed Sep  3 21:54:53 2014 | SETI@home | No tasks sent
Wed Sep  3 21:54:53 2014 | SETI@home | No tasks are available for AstroPulse v6
Wed Sep  3 21:59:59 2014 | SETI@home | Sending scheduler request: To fetch work.
Wed Sep  3 21:59:59 2014 | SETI@home | Requesting new tasks for CPU and ATI
Wed Sep  3 22:00:00 2014 | SETI@home | Scheduler request completed: got 0 new tasks
Wed Sep  3 22:00:00 2014 | SETI@home | No tasks sent
Wed Sep  3 22:00:00 2014 | SETI@home | No tasks are available for AstroPulse v6
Wed Sep  3 22:07:06 2014 | SETI@home Beta Test | Sending scheduler request: To fetch work.
Wed Sep  3 22:07:06 2014 | SETI@home Beta Test | Requesting new tasks for ATI
Wed Sep  3 22:07:08 2014 | SETI@home Beta Test | Scheduler request completed: got 0 new tasks
Wed Sep  3 22:08:19 2014 | SETI@home | Sending scheduler request: To fetch work.
Wed Sep  3 22:08:19 2014 | SETI@home | Requesting new tasks for CPU and ATI
Wed Sep  3 22:08:20 2014 | SETI@home | Scheduler request completed: got 0 new tasks
Wed Sep  3 22:08:20 2014 | SETI@home | No tasks sent
Wed Sep  3 22:08:20 2014 | SETI@home | No tasks are available for AstroPulse v6
Wed Sep  3 22:13:49 2014 | SETI@home | update requested by user
Wed Sep  3 22:13:51 2014 | SETI@home | Sending scheduler request: Requested by user.
Wed Sep  3 22:13:51 2014 | SETI@home | Requesting new tasks for CPU and ATI
Wed Sep  3 22:13:52 2014 | SETI@home | Scheduler request completed: got 0 new tasks
Wed Sep  3 22:13:52 2014 | SETI@home | No tasks sent
Wed Sep  3 22:13:52 2014 | SETI@home | No tasks are available for AstroPulse v6


It has 4 GPU tasks left. I'll bet if I force those new CPU tasks to GPU tasks it will start downloading CPU tasks again...even though it needs GPU tasks instead.
ID: 1566775 · Report as offensive
Profile Jeff Buck Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 11 Feb 00
Posts: 1441
Credit: 148,764,870
RAC: 0
United States
Message 1566779 - Posted: 4 Sep 2014, 2:32:36 UTC - in response to Message 1566775.  

Well, so far I've only gotten 1 AP task in the 3+ hours since they started loading new tapes, and it went to a GPU on my xw9400.
ID: 1566779 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1566781 - Posted: 4 Sep 2014, 2:35:54 UTC - in response to Message 1566768.  
Last modified: 4 Sep 2014, 2:37:31 UTC

Has anyone noticed the server is sending out CPU tasks even if your GPUs don't have any work? All 3 of my Hosts are receiving CPU work even though they are not needed. I have a Mac whose 3 GPUs are going idle while the server sends it CPU work. If I change the preferences to GPU only, the server doesn't send any work :-(

What I'm seeing over the last several days is that sometimes a work fetch for both CPU and GPU will only return one or more VLARs for the CPU, but nothing for the GPU. However, usually the next request will fill up the GPU queue, too. It seems as if when the scheduler only finds VLARs at the top of its RTS buffer, it sometimes doesn't dig very deep to find non-VLARs for the GPU if it's pressed for time. I haven't had any hosts stop getting GPU tasks altogether.

Edit: Ah, I see you're talking about AP-only hosts. Never mind! AP tasks are rare in my world.

If you had a Mac ALL you would run on the GPUs are APs since there isn't an ATI MB App. It's down to 2 GPU tasks while it has downloaded 9 more CPU tasks. The GPUs are about to go Idle again while it keeps getting CPU work.
ID: 1566781 · Report as offensive
Profile Jeff Buck Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 11 Feb 00
Posts: 1441
Credit: 148,764,870
RAC: 0
United States
Message 1566785 - Posted: 4 Sep 2014, 2:58:05 UTC - in response to Message 1566781.  


If you had a Mac ALL you would run on the GPUs are APs since there isn't an ATI MB App. It's down to 2 GPU tasks while it has downloaded 9 more CPU tasks. The GPUs are about to go Idle again while it keeps getting CPU work.

Yeah, I don't have a MAC and only one of my machines has an ATI card (an old 5450).

I just noticed 2 more APs downloaded, this time to my T7400, and both went to the GPU queue, so whatever problem you're experiencing, it isn't hitting here. Perhaps another MAC / ATI user will have to chime in. Certainly the AP RTS buffer is still running on empty, and the new tapes are throwing errors just like the last batch, so that isn't helping.
ID: 1566785 · Report as offensive
Previous · 1 . . . 8 · 9 · 10 · 11 · 12 · 13 · 14 . . . 24 · Next

Message boards : Number crunching : Panic Mode On (89) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.