Cuda tasks not using GPU


log in

Advanced search

Message boards : Number crunching : Cuda tasks not using GPU

Author Message
Geoff Schultz
Send message
Joined: 2 Jun 99
Posts: 6
Credit: 3,101,417
RAC: 5,611
United States
Message 1308532 - Posted: 21 Nov 2012, 22:37:36 UTC
Last modified: 21 Nov 2012, 22:39:44 UTC

I have a Dell M6300 laptop (http://setiathome.berkeley.edu/show_host_detail.php?hostid=6840544) with an Core 2 Extreme and an Nvidia Quadro 1600M graphics processor. I just installed Windows 8 on it and installed the latest cuda drivers for it. I then downloaded new SETI tasks and I see that one of the tasks which is running is supposed to be utilizing the GPU, but it's utilizing ~50% of the main processor and according to GPU Shark (which is the GPU utilization task that I downloaded), the GPU is running a 0% utilization.

I've included a screen shot showing the BOINC Manager process status and the Task Manager showing both SETI processes each using about 50% of the processor. What's going on here?

-- Geoff


____________

ClaggyProject donor
Volunteer tester
Send message
Joined: 5 Jul 99
Posts: 4098
Credit: 33,064,487
RAC: 8,823
United Kingdom
Message 1308534 - Posted: 21 Nov 2012, 22:42:26 UTC - in response to Message 1308532.
Last modified: 21 Nov 2012, 22:54:03 UTC

You're run out of GPU memory:

http://setiathome.berkeley.edu/result.php?resultid=2721570959

Cuda error 'cudaMalloc((void**) &dev_t_funct_cache' in file 'c:/sw/gpgpu/seti/seti_boinc/client/cuda/cudaAcc_pulsefind.cu' in line 662 : out of memory.
setiathome_CUDA: CUDA runtime ERROR in device memory allocation (Step 3 of 3). Falling back to HOST CPU processing...

Try turning Aero off to free some GPU memory up.

Claggy

Geoff Schultz
Send message
Joined: 2 Jun 99
Posts: 6
Credit: 3,101,417
RAC: 5,611
United States
Message 1308553 - Posted: 22 Nov 2012, 0:00:09 UTC - in response to Message 1308534.

Thanks for the info. I switched themes, basically shutting off Aero & got a lot of memory back. However, now I get the following. Is this just a fools errand to try to use this GPU?

-- Geoff

<message>
Incorrect function. (0x1) - exit code 1 (0x1)
</message>
<stderr_txt>
setiathome_CUDA: Found 1 CUDA device(s):
Device 1 : Quadro FX 1600M
totalGlobalMem = 268435456
sharedMemPerBlock = 16384
regsPerBlock = 8192
warpSize = 32
memPitch = 2147483647
maxThreadsPerBlock = 512
clockRate = 1250000
totalConstMem = 65536
major = 1
minor = 1
textureAlignment = 256
deviceOverlap = 1
multiProcessorCount = 4
setiathome_CUDA: CUDA Device 1 specified, checking...
Device 1: Quadro FX 1600M is okay
SETI@home using CUDA accelerated device Quadro FX 1600M
setiathome_enhanced 6.09 Visual Studio/Microsoft C++
libboinc: 6.3.22

Work Unit Info:
...............
WU true angle range is : 2.715301
Optimal function choices:
-----------------------------------------------------
name
-----------------------------------------------------
v_BaseLineSmooth (no other)
v_GetPowerSpectrum 0.00019 0.00000
v_ChirpData 0.01584 0.00000
v_Transpose4 0.00315 0.00000
FPU opt folding 0.00166 0.00000
CUFFT error in file 'c:/sw/gpgpu/seti/seti_boinc/client/cuda/cudaAcc_fft.cu' in line 62.
____________

Josef W. SegurProject donor
Volunteer developer
Volunteer tester
Send message
Joined: 30 Oct 99
Posts: 4247
Credit: 1,048,560
RAC: 273
United States
Message 1308587 - Posted: 22 Nov 2012, 2:26:11 UTC - in response to Message 1308553.

Thanks for the info. I switched themes, basically shutting off Aero & got a lot of memory back. However, now I get the following. Is this just a fools errand to try to use this GPU?

-- Geoff
...
CUFFT error in file 'c:/sw/gpgpu/seti/seti_boinc/client/cuda/cudaAcc_fft.cu' in line 62.

If you can be patient, the servers ought to eventually send some GPU tasks to be done with the 6.08 CUDA application, which is much more suitable for a GPU with 256MB VRAM.

When the 6.09 CUDA23 application was first available, BOINC's method of deciding whether it was suitable for a given GPU would have prohibited sending that work to your GPU because it has less than 384MB of VRAM. To make a change then required editing a source file and recompiling the server code. More recently BOINC has a revised method which is controlled by a separate XML file. That seems to be buggy, but is so obviously a better approach that the project has adopted it. When more pressing issues are solved, they'll get it debugged, I'm sure.

The increased memory usage of 6.09 is IIRC mainly because of the revised cufft.dll and that also provides a significant boost in productivity on GPUs which can use it. It might be possible on your GPU with even more changes to free up VRAM, perhaps dropping to 16 bit color, etc. That may be more than you want to do. Perhaps the sensible thing to do is to simply abort any CUDA23 work the servers assign, so a resend can go promptly to another host.

Even with 6.08, the GPU ought to provide a significant improvement in productivity over doing only CPU work. But if the project has to go to only WUs with 4 times as much data as indicated in the News, GPUs with only 256MB will probably no longer be usable. That's months away, in the meantime your GPU might do a considerable amount of work.
Joe

Profile dancer42
Volunteer tester
Send message
Joined: 2 Jun 02
Posts: 436
Credit: 1,153,958
RAC: 2,521
United States
Message 1308766 - Posted: 22 Nov 2012, 13:56:28 UTC - in response to Message 1308532.

download tthrottle and see what it says, also temp monitoring and throttling can save you a cpu or gpu perchase in the near future seti can run them hot.also not all client use opencl or cuda and with no installer available from Lunatics site right now installing optimized clients can be a pain hang in there some astropulse
will become available as the scheduling server catches up if you are still having problems after that post again.
____________

Profile dancer42
Volunteer tester
Send message
Joined: 2 Jun 02
Posts: 436
Credit: 1,153,958
RAC: 2,521
United States
Message 1308768 - Posted: 22 Nov 2012, 14:03:09 UTC - in response to Message 1308553.

fft is fast Fourier transform might try the math library s at
http://developer.amd.com/tools/heterogeneous-computing/amd-accelerated-parallel-processing-math-libraries/
do not know if it will work for cuda but worth a try.
____________

Geoff Schultz
Send message
Joined: 2 Jun 99
Posts: 6
Credit: 3,101,417
RAC: 5,611
United States
Message 1308791 - Posted: 22 Nov 2012, 15:07:29 UTC - in response to Message 1308768.

This is actually an old machine of mine that I'm giving to my 91yo dad, so I'm not going to worry about this. I just installed the GPU drivers for desktop and new laptop, so we'll see how those go. Thanks for the help! Hopefully I won't need it with my systems.

-- Geoff
____________

Profile dancer42
Volunteer tester
Send message
Joined: 2 Jun 02
Posts: 436
Credit: 1,153,958
RAC: 2,521
United States
Message 1308867 - Posted: 22 Nov 2012, 18:17:42 UTC - in response to Message 1308791.

the cpu likely does not support the more advanced modes necessary to run new optimized clients but an older one might work.
____________

Geoff Schultz
Send message
Joined: 2 Jun 99
Posts: 6
Credit: 3,101,417
RAC: 5,611
United States
Message 1319375 - Posted: 23 Dec 2012, 18:49:18 UTC
Last modified: 23 Dec 2012, 18:49:50 UTC

I just got a nvidia GeForce GTX 650 with 2 GB of memory & installed it on a core i7 machine running Windows 8. All of my tasks are now aborting immediately with the following errors. The card has plenty of free memory and is running cool. Any ideas?

http://setiathome.berkeley.edu/result.php?resultid=2766312292

<core_client_version>7.0.28</core_client_version>
<![CDATA[
<message>
Incorrect function. (0x1) - exit code 1 (0x1)
</message>
<stderr_txt>
setiathome_CUDA: Found 1 CUDA device(s):
Device 1 : GeForce GTX 650
totalGlobalMem = -2147483648
sharedMemPerBlock = 49152
regsPerBlock = 65536
warpSize = 32
memPitch = 2147483647
maxThreadsPerBlock = 1024
clockRate = 1058500
totalConstMem = 65536
major = 3
minor = 0
textureAlignment = 512
deviceOverlap = 1
multiProcessorCount = 2
setiathome_CUDA: CUDA Device 1 specified, checking...
Device 1: GeForce GTX 650 is okay
SETI@home using CUDA accelerated device GeForce GTX 650
setiathome_enhanced 6.09 Visual Studio/Microsoft C++
libboinc: 6.3.22

Work Unit Info:
...............
WU true angle range is : 0.415212
Optimal function choices:
-----------------------------------------------------
name
-----------------------------------------------------
v_BaseLineSmooth (no other)
v_GetPowerSpectrum 0.00018 0.00000
v_ChirpData 0.01113 0.00000
v_Transpose4 0.00284 0.00000
FPU opt folding 0.00402 0.00000
CUFFT error in file 'd:/Projects/SETI/seti_boinc/client/cuda/cudaAcc_fft.cu' in line 62.

</stderr_txt>
]]>

____________

Profile Wiggo
Avatar
Send message
Joined: 24 Jan 00
Posts: 6952
Credit: 94,650,822
RAC: 75,542
Australia
Message 1319382 - Posted: 23 Dec 2012, 19:09:06 UTC - in response to Message 1319375.
Last modified: 23 Dec 2012, 19:10:32 UTC

Check the 2nd part ( 2) Kepler cards unsupported ) of this post out, http://setiathome.berkeley.edu/forum_thread.php?id=69735#1296126, as it maybe your problem.

Cheers.
____________

Geoff Schultz
Send message
Joined: 2 Jun 99
Posts: 6
Credit: 3,101,417
RAC: 5,611
United States
Message 1319415 - Posted: 23 Dec 2012, 20:30:26 UTC - in response to Message 1319382.

Thanks for the quick reply. As per the suggestions, I set the environment variable CUDA_GRID_SIZE_COMPAT to 1 and restarted BOINC and it's no longer aborting.

Another question: It appears that only 1 CUDA task can run as once and I see that the GPU is only typically running at about 60-70%. I also see that the estimated run time on the CUDA task is ~17 minutes and the actual run time is ~24 minutes, so if the GPU were running at 100%, the actual run time would be right around 17 minutes. Is setting this environmental variable causing some bottleneck?

-- Geoff
____________

Profile Paul D Harris
Volunteer tester
Send message
Joined: 1 Dec 99
Posts: 1123
Credit: 33,598,472
RAC: 0
United States
Message 1319431 - Posted: 23 Dec 2012, 21:27:25 UTC
Last modified: 23 Dec 2012, 21:28:28 UTC

edit app_info
<count>1</count
to
<count>.5</count>
for all of the count entries in the app_info.xml file.
This will do 2 wu instead of 1 and gpu useage will go up.
____________

Profile Gatekeeper
Avatar
Send message
Joined: 14 Jul 04
Posts: 887
Credit: 176,479,616
RAC: 0
United States
Message 1319433 - Posted: 23 Dec 2012, 21:34:38 UTC - in response to Message 1319431.

edit app_info
<count>1</count
to
<count>.5</count>
for all of the count entries in the app_info.xml file.
This will do 2 wu instead of 1 and gpu useage will go up.


He's using stock app...no app_info file.
____________

Richard HaselgroveProject donor
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8502
Credit: 49,975,145
RAC: 50,930
United Kingdom
Message 1319471 - Posted: 24 Dec 2012, 0:15:08 UTC - in response to Message 1319433.

edit app_info
<count>1</count
to
<count>.5</count>
for all of the count entries in the app_info.xml file.
This will do 2 wu instead of 1 and gpu useage will go up.

He's using stock app...no app_info file.

So use app_config.xml instead:

BOINC 7.0.40-42 and new app_config.xml
(requires use of alpha-test version of BOINC)

Geoff Schultz
Send message
Joined: 2 Jun 99
Posts: 6
Credit: 3,101,417
RAC: 5,611
United States
Message 1319480 - Posted: 24 Dec 2012, 1:00:31 UTC - in response to Message 1319471.

So where do I find this alpha version?

-- Geoff
____________

ClaggyProject donor
Volunteer tester
Send message
Joined: 5 Jul 99
Posts: 4098
Credit: 33,064,487
RAC: 8,823
United Kingdom
Message 1319481 - Posted: 24 Dec 2012, 1:08:50 UTC - in response to Message 1319480.
Last modified: 24 Dec 2012, 1:15:07 UTC

So where do I find this alpha version?

-- Geoff

The easiest way is to get the links from the BOINC 7 Change Log and news thread.

Edit: the following is for Anonymous Platform Users only:
(But beware, because of a change in Boinc 7.0.32, where the internal flops value for the GPU was increased by times 10,
all existing GPU work will be on the verge of going Maximum Time Exceeded, so finish any GPU work first prior to upgrading, new GPU work will be O.K}

Claggy

Message boards : Number crunching : Cuda tasks not using GPU

Copyright © 2014 University of California