Long GPU units again?

Message boards : Number crunching : Long GPU units again?
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · Next

AuthorMessage
Brkovip
Avatar

Send message
Joined: 18 May 99
Posts: 274
Credit: 144,414,367
RAC: 0
United States
Message 1078162 - Posted: 17 Feb 2011, 0:51:58 UTC

I am getting some GPU units that are taking over 3 hrs. Did we make up some new type of units that are really taking that long on a GTX 480? Is anyone else noticing this too?
ID: 1078162 · Report as offensive
Brkovip
Avatar

Send message
Joined: 18 May 99
Posts: 274
Credit: 144,414,367
RAC: 0
United States
Message 1078172 - Posted: 17 Feb 2011, 1:22:19 UTC - in response to Message 1078162.  

I am getting some GPU units that are taking over 3 hrs. Did we make up some new type of units that are really taking that long on a GTX 480? Is anyone else noticing this too?


Interesting, I shut off SLI and the same units go back to normal speeds. This using the 266.58 drivers and 6.10.58.
ID: 1078172 · Report as offensive
Profile aaronh
Volunteer tester
Avatar

Send message
Joined: 27 Oct 99
Posts: 169
Credit: 1,442,686
RAC: 0
United States
Message 1078174 - Posted: 17 Feb 2011, 1:29:37 UTC - in response to Message 1078162.  

Are you talking about ones like this?

For some reason, it's showing as "Anonymous platform (NVIDIA GPU)" but it's running the CPU application. Have you modified your app_info.xml recently?

(you updated while I was typing this) Ah, yes, I believe SLI has been known to cause issues. Still, I'd check that your GPU WU's are really using the GPU
ID: 1078174 · Report as offensive
Profile perryjay
Volunteer tester
Avatar

Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 20,676,751
RAC: 0
United States
Message 1078175 - Posted: 17 Feb 2011, 1:31:50 UTC - in response to Message 1078162.  
Last modified: 17 Feb 2011, 1:36:31 UTC

Well, glad you found your answer. I have run into a few that are taking close to an hour for me on my 450 so I do believe we do have some kind of new WUs out there but I've never seen any taking three hours. I have also seen others complaining about long work units, I will remember this post and be sure to ask them about SLI next time I see one.


Aaron, that's called CPU fallback and it is usually caused by not enough memory on the GPU or so it says. Maybe SLI is causing the GPUs to show a false memory reading.


PROUD MEMBER OF Team Starfire World BOINC
ID: 1078175 · Report as offensive
Brkovip
Avatar

Send message
Joined: 18 May 99
Posts: 274
Credit: 144,414,367
RAC: 0
United States
Message 1078178 - Posted: 17 Feb 2011, 1:35:25 UTC - in response to Message 1078174.  

Are you talking about ones like this?

For some reason, it's showing as "Anonymous platform (NVIDIA GPU)" but it's running the CPU application. Have you modified your app_info.xml recently?

(you updated while I was typing this) Ah, yes, I believe SLI has been known to cause issues. Still, I'd check that your GPU WU's are really using the GPU


Yeah, they are using the GPU, it is easy to tell with EVGA Precision and I have run the Rescheduler to make sure they are right.
ID: 1078178 · Report as offensive
Profile aaronh
Volunteer tester
Avatar

Send message
Joined: 27 Oct 99
Posts: 169
Credit: 1,442,686
RAC: 0
United States
Message 1078182 - Posted: 17 Feb 2011, 1:41:24 UTC - in response to Message 1078175.  

Aaron, that's called CPU fallback and it is usually caused by not enough memory on the GPU or so it says. Maybe SLI is causing the GPUs to show a false memory reading.


I'm pretty sure CPU fallback doesn't cause it to run a completely different application. (The task I linked ran with "Version info: SSE4.1 (Intel, Core 2-optimized v8-nographics) V5.13 by Alex Kan" while his other GPU tasks ran with "Multibeam x32f Preview, Cuda 3.0")

However, he has mentioned using the rescheduler, and that can change what application a WU runs with.
ID: 1078182 · Report as offensive
Profile perryjay
Volunteer tester
Avatar

Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 20,676,751
RAC: 0
United States
Message 1078183 - Posted: 17 Feb 2011, 1:48:32 UTC - in response to Message 1078182.  
Last modified: 17 Feb 2011, 1:52:51 UTC

Well, usually there is a mention in the STderr about falling back to CPU but in the case you are linking to is the optimized CPU app. It looks like that WU was actually a CPU task.

After looking closer, I see it does show as a GPU originally, maybe he has his reschedule set to automatically send VHARs to the CPU or moved it himself.


PROUD MEMBER OF Team Starfire World BOINC
ID: 1078183 · Report as offensive
Brkovip
Avatar

Send message
Joined: 18 May 99
Posts: 274
Credit: 144,414,367
RAC: 0
United States
Message 1078196 - Posted: 17 Feb 2011, 3:39:39 UTC - in response to Message 1078183.  

Well, usually there is a mention in the STderr about falling back to CPU but in the case you are linking to is the optimized CPU app. It looks like that WU was actually a CPU task.

After looking closer, I see it does show as a GPU originally, maybe he has his reschedule set to automatically send VHARs to the CPU or moved it himself.


You are correct, I send VLAR's and VHAR's to the CPU. The rescheduler runs every 24 hrs normally but when I see tasks taking a very long time I run it to verify something hasn't slipped through.
ID: 1078196 · Report as offensive
Profile ReiAyanami
Avatar

Send message
Joined: 6 Dec 05
Posts: 116
Credit: 222,900,202
RAC: 174
Japan
Message 1078200 - Posted: 17 Feb 2011, 4:00:37 UTC

I noticed the same.
I'm not using any extra software.
Just SETI@home Enhanced 6.03 windows_intelx86 and SETI@home Enhanced 6.10 windows_intelx86 (cuda_fermi). EVGA precision tells me that GPU's are used (actually GPU usage is higher than usual: ~75% compapred to ~50%). Usually GPU calculations take anywhere between 3 to 14 min, but now 80 to 100 min on GTX-475 (2 units SLI).
ID: 1078200 · Report as offensive
Profile aaronh
Volunteer tester
Avatar

Send message
Joined: 27 Oct 99
Posts: 169
Credit: 1,442,686
RAC: 0
United States
Message 1078292 - Posted: 17 Feb 2011, 14:28:25 UTC

ID: 1078292 · Report as offensive
Profile Miep
Volunteer moderator
Avatar

Send message
Joined: 23 Jul 99
Posts: 2412
Credit: 351,996
RAC: 0
Message 1078339 - Posted: 17 Feb 2011, 17:03:04 UTC - in response to Message 1078175.  

Aaron, that's called CPU fallback and it is usually caused by not enough memory on the GPU or so it says.


FYI CPU fallback is disabled in x32f. The poor WUs just error out - like this one

CPU fallback only works in stock and you would see stderr output on the lines of:

setiathome_CUDA: CUDA runtime ERROR in device memory allocation (Step 1 of 3). Falling back to HOST CPU processing...


and really long runtimes because boinc doesn't know a CPU has been taken over.

I definitely see a couple VLARs that got sent to your GPU somehow.


oops. not good. anybody else seeing .vlar marked tasks on their GPU?
Carola
-------
I'm multilingual - I can misunderstand people in several languages!
ID: 1078339 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1078354 - Posted: 17 Feb 2011, 17:30:56 UTC - in response to Message 1078339.  

oops. not good. anybody else seeing .vlar marked tasks on their GPU?

Not here. A couple (correctly) sent to the host I allow CPU crunching on, but no sign of any in the GPU queues.

But I agree, it would be a bad sign. We should watch, just in case.
ID: 1078354 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1078358 - Posted: 17 Feb 2011, 17:36:56 UTC - in response to Message 1078354.  

oops. not good. anybody else seeing .vlar marked tasks on their GPU?

Not here. A couple (correctly) sent to the host I allow CPU crunching on, but no sign of any in the GPU queues.

But I agree, it would be a bad sign. We should watch, just in case.

Just checked the 2 rigs I have running right now, and no VLAR in the GPU caches there so far. The other 6 rigs are shut down for the day because my electric rates are higher during the day. I can check them later tonight when the come back online.

I seem to recall there were some situations where resend VLAR work would sometimes slip through the filter that keeps the scheduler from sending VLAR to GPU work requests. I don't know if that is still happening or not, but it could be.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1078358 · Report as offensive
Profile James Sotherden
Avatar

Send message
Joined: 16 May 99
Posts: 10436
Credit: 110,373,059
RAC: 54
United States
Message 1078367 - Posted: 17 Feb 2011, 17:53:45 UTC

I see some vlars in my CPU cache but none so far in GPU. But i think Mark is right There might still be some old unmarked vlars from a time out or uncruncehed work units still floating around from months ago.
[/quote]

Old James
ID: 1078367 · Report as offensive
Profile Miep
Volunteer moderator
Avatar

Send message
Joined: 23 Jul 99
Posts: 2412
Credit: 351,996
RAC: 0
Message 1078379 - Posted: 17 Feb 2011, 18:30:05 UTC

The ones Aaron linked are not unmarked and they are not resends.
Carola
-------
I'm multilingual - I can misunderstand people in several languages!
ID: 1078379 · Report as offensive
Profile arkayn
Volunteer tester
Avatar

Send message
Joined: 14 May 99
Posts: 4438
Credit: 55,006,323
RAC: 0
United States
Message 1078388 - Posted: 17 Feb 2011, 18:40:26 UTC

I know I get vlar tasks for the ATI card, but they do better on ATI cards.

ID: 1078388 · Report as offensive
dan
Avatar

Send message
Joined: 18 Oct 02
Posts: 392
Credit: 25,046,383
RAC: 0
United States
Message 1078392 - Posted: 17 Feb 2011, 18:47:52 UTC - in response to Message 1078354.  

oops. not good. anybody else seeing .vlar marked tasks on their GPU?

Not here. A couple (correctly) sent to the host I allow CPU crunching on, but no sign of any in the GPU queues.

But I agree, it would be a bad sign. We should watch, just in case.


I had a couple vlars show up on cuda. Caught them and changed them over to cpu.
Dan
ID: 1078392 · Report as offensive
Profile Viciente
Volunteer tester
Avatar

Send message
Joined: 21 Jan 11
Posts: 228
Credit: 326,384
RAC: 0
Austria
Message 1078394 - Posted: 17 Feb 2011, 18:49:13 UTC - in response to Message 1078339.  

oops. not good. anybody else seeing .vlar marked tasks on their GPU?

everything seems ok here; s@h enhanced 6.03 ~ 1:50:00, astropulse 5.05 ~ 84:00:00, s&h 6.10 cuda ~ 00:30:00 (2x gtx460 sli) .. rgds.
ID: 1078394 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1078402 - Posted: 17 Feb 2011, 18:56:11 UTC - in response to Message 1078398.  

I just measured all of my workunits. None of them are longer than any other. They are all exactly 5.25 centimeters, or 2.1 inches.

ROFLMAO...
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1078402 · Report as offensive
dan
Avatar

Send message
Joined: 18 Oct 02
Posts: 392
Credit: 25,046,383
RAC: 0
United States
Message 1078422 - Posted: 17 Feb 2011, 19:47:13 UTC - in response to Message 1078402.  

I just measured all of my workunits. None of them are longer than any other. They are all exactly 5.25 centimeters, or 2.1 inches.

ROFLMAO...


Nice one Sten.
ID: 1078422 · Report as offensive
1 · 2 · 3 · Next

Message boards : Number crunching : Long GPU units again?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.