How to optimize GPU configuration?

Message boards : Number crunching : How to optimize GPU configuration?
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · Next

AuthorMessage
awdorrin

Send message
Joined: 27 Sep 99
Posts: 71
Credit: 106,424,089
RAC: 261
United States
Message 1832188 - Posted: 24 Nov 2016, 15:27:36 UTC

I have been running three AMD GPUs in an Intel 4790 system for a long time without major issues (Windows 10). Tuesday I replaced my slowest AMD GPU (7850) with a nVidia GTX 1070 and started having problems with performance that appear to be GPU related.

If I snooze the GPU, I can use the computer to web surf, read email, etc without issue, but if the GPU processing is enabled mouse and keyboard input is delayed (or screen rendering is delayed, not entirely sure which) but I can't even type this forum post easily, with the GPU processing enabled.

Not sure if this is a result of the mixed AMD and nVidia environment, or a result of adding the 1070 card.

Since the CPU is a hyper threaded quad core, i've always run 6 cpu tasks, and the GPUs are all configured to run two GPU tasks each. the 1070 is running CUDA, while AMD is OpenCL. In task manager I see the CPU and AMD tasks, but I don't see corresponding nVidia tasks.

* MB8_win_x64_AVX_VS2001_r3330.exe
* MB8_win_x86_SEE2_OpenCL_ATi_HD5_r3500.exe (32 bit)
* setiathome_v8 (32 bit) [are these the nvidia support tasks?]

When I installed the GTX 1070, I installed the nVidia drivers. I have seen a couple posts indicating that you might need to install the nVidia drivers first, then install AMD drivers (?) but I have not yet tried that.

Since BOINC seems to be interfacing the to the GPUs fine, I think that maybe this is a configuration issue and not a driver issue. But I'm really not sure what to do next. I am assuming there may be some XML or command line option tweaks I could try, but I really have no idea where to start with that.

I don't see a way to embed an XML file to a post, so I will add my XML files in a following post.

Hoping someone will have some suggestions for me.

Thanks!
ID: 1832188 · Report as offensive
awdorrin

Send message
Joined: 27 Sep 99
Posts: 71
Credit: 106,424,089
RAC: 261
United States
Message 1832189 - Posted: 24 Nov 2016, 15:28:58 UTC
Last modified: 24 Nov 2016, 15:44:58 UTC

Bah, my app_info.xml file is too big to post. 75k

So here are links to my Google Drive:

app_config.xml: https://drive.google.com/open?id=0BzaSHPNnUvY7MnZiQzRVdE1FVzQ

app_info.xml: https://drive.google.com/open?id=0BzaSHPNnUvY7Vy1naTlXNENRS0E
ID: 1832189 · Report as offensive
Profile tullio
Volunteer tester

Send message
Joined: 9 Apr 04
Posts: 8797
Credit: 2,930,782
RAC: 1
Italy
Message 1832193 - Posted: 24 Nov 2016, 16:27:14 UTC

I just switched my Windows 10 PC from GTX 750 OC to GTX 1050 TI. The installation went without a hitch and I installed the latest Geforce driver, 275.95. I am waiting for SETI@home and Einstein@home GPU tasks to see any difference. I plan to install the GTX 750 on my old Linux box, when I feel to overcome the usual Linux problems when installing nVidia Cards.
Tullio
ID: 1832193 · Report as offensive
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 26 May 99
Posts: 9954
Credit: 103,452,613
RAC: 328
United Kingdom
Message 1832200 - Posted: 24 Nov 2016, 16:44:16 UTC
Last modified: 24 Nov 2016, 16:47:52 UTC

and I installed the latest Geforce driver, 275.95


I hope that was meant to be 375.95 :-)

And if it is this machine

7747511

It hasn't contacted SETI@Home for nearly 24 hours.
ID: 1832200 · Report as offensive
awdorrin

Send message
Joined: 27 Sep 99
Posts: 71
Credit: 106,424,089
RAC: 261
United States
Message 1832201 - Posted: 24 Nov 2016, 16:54:14 UTC

I am assuming he does mean 375.95 :)

My system has 375.95 installed along with the Radeon 16.6 drivers.

It seems like the problem comes and goes, and is probably related to CPU utilization when everything is running.
Currently my system is pulling between 90 and 93% CPU (6 CPU, 4 AMD, 2 NVidia)
and I see no input (mouse/keyboard) issues at the moment.

Seems like once CPU utilization passes a certain threshold, the problem returns.

Would love to figure out how to limit CPU utilization to 3 of the four cores.
And then max the GPUs at their optimal task level (not sure if its 3 or 4 simultaneous tasks.)

Would also love if I could keep the AMD GPUs crunching when I game, but it does not look like there is any way to do that with the exclusive apps settings.
I see someone was trying to do that with some batch files, but was running into issues getting things to turn back on.

-Al
ID: 1832201 · Report as offensive
Profile tullio
Volunteer tester

Send message
Joined: 9 Apr 04
Posts: 8797
Credit: 2,930,782
RAC: 1
Italy
Message 1832202 - Posted: 24 Nov 2016, 16:59:25 UTC - in response to Message 1832200.  
Last modified: 24 Nov 2016, 17:04:57 UTC

You're right, but it finished its workload. Now I have asked for more but I did not get it. I received a DVD from Palit with the driver for the new board, but once installed and restarted it opened Windows 10 without a hitch. Now it is running LHC@home, which includes the old vLHC@home, CMS@home and other CERN projects which use VirtualBox. I am waiting for GPU tasks.
Tullio
It has been working also SETI Beta, by the way.
ID: 1832202 · Report as offensive
Profile Mike Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 01
Posts: 34256
Credit: 79,922,639
RAC: 80
Germany
Message 1832215 - Posted: 24 Nov 2016, 17:57:24 UTC

You should use the Beta6 installer and install OpenCL for both AMD and NVidia card.
Most probably you are getting screen lags running Guppie tasks with the cuda app.


With each crime and every kindness we birth our future.
ID: 1832215 · Report as offensive
awdorrin

Send message
Joined: 27 Sep 99
Posts: 71
Credit: 106,424,089
RAC: 261
United States
Message 1832369 - Posted: 25 Nov 2016, 14:51:43 UTC - in response to Message 1832215.  

I just downloaded the Beta6 installer. Waiting for my current tasks to process through, then I'll update and see what happens.

Thanks for the suggestion.
ID: 1832369 · Report as offensive
Profile tullio
Volunteer tester

Send message
Joined: 9 Apr 04
Posts: 8797
Credit: 2,930,782
RAC: 1
Italy
Message 1832371 - Posted: 25 Nov 2016, 15:04:15 UTC
Last modified: 25 Nov 2016, 15:05:07 UTC

I had to downgrade my NVIDIA driver on the GTX 1050 Ti otherwise BOINC would not recognize it. Now it is running SETI@Home beta and making some fan noise, but GPU temperature is around 50 C or less and the fan runs at 34%. A good board by Palit, also with a Thunder Master desktop tool in case I'd like to optimize it. But I am keeping standard parameters, it is fast enough compared to my former GTX 750 OC.
Tullio
ID: 1832371 · Report as offensive
baron_iv
Volunteer tester
Avatar

Send message
Joined: 4 Nov 02
Posts: 109
Credit: 104,905,241
RAC: 0
United States
Message 1832452 - Posted: 25 Nov 2016, 22:33:52 UTC

I'm running an AMD 290 and GTX 780 in my system and it's working just fine. No lags or rendering issues, so it's not running different brand GPUs in the same computer that's causing the problem. However, some people have reported problems with the latest Nvidia driver, I read it on guru3d's site, so that could be your issue.

I do only run 4 CPU tasks, with 2 GPU tasks, since this is my main computer. It's got a 6700K @ 4.6ghz, but it ran just as well with the 2600K @ 4.6ghz, so it's unlikely that your CPU is the problem.
ID: 1832452 · Report as offensive
awdorrin

Send message
Joined: 27 Sep 99
Posts: 71
Credit: 106,424,089
RAC: 261
United States
Message 1832689 - Posted: 26 Nov 2016, 16:58:27 UTC

So I updated using the Lunatics Beta 6 installer.
Currently Task Manager shows me running the following applications:
* M8_win_x86_SSW3_OpenCL_NV_SoG_r3557.exe (32-bit) (Two tasks)
* M8_win_x86_SSE2_OpenCL_ATi_HD5_r3556.exe (32-bit) (Four tasks)
* M8_win_x64_AVX_VS2010_r3330.exe (6 tasks)

My CPU is pegged at 100%, but the mouse and keyboard appears to be acting fine.
No sluggish/stuttering as I am typing this email, so if I can use the computer for that, not too worried about the 100% CPU utilization.

Still have my Usage limit set at 'use at most 80% of the CPUs', so it does seem like the GPU tasks are not paying any attention to that limit.

The unexpected thing, to me, is that the 'OpenCL_NV_SoG' tasks are taking about 12-12.6% CPU, which is a lot higher than the 'OpenCL_ATi_HD5' tasks, which vary from 0.4 to 5.0%.

Will have to keep an eye on it, and see if I can push to 3 tasks per GPU soon.
ID: 1832689 · Report as offensive
Harri Liljeroos
Avatar

Send message
Joined: 29 May 99
Posts: 4070
Credit: 85,281,665
RAC: 126
Finland
Message 1832733 - Posted: 26 Nov 2016, 23:40:28 UTC - in response to Message 1832689.  

The CPU % limit applies to CPU tasks only. It has no effect on GPU tasks.

By default the NV SOG application needs a full CPU core per each task. I that is a problem to you use the -use_sleep command line parameter, this will lower the CPU usage for that application. Read the "ReadMe_MultiBeam_OpenCL_NV_SoG.txt" in docs folder under projects/setiathome.berkeley.edu. There is decribed a lot more about the command line parameters that can be used. Also you can read the Raistmer's sticky thread http://setiathome.berkeley.edu/forum_thread.php?id=80381 about 8.19 Windows GPU application support thread.
ID: 1832733 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1832794 - Posted: 27 Nov 2016, 3:39:51 UTC - in response to Message 1832188.  

I have been running three AMD GPUs in an Intel 4790 system for a long time without major issues (Windows 10). Tuesday I replaced my slowest AMD GPU (7850) with a nVidia GTX 1070 and started having problems with performance that appear to be GPU related.

If I snooze the GPU, I can use the computer to web surf, read email, etc without issue, but if the GPU processing is enabled mouse and keyboard input is delayed (or screen rendering is delayed, not entirely sure which) but I can't even type this forum post easily, with the GPU processing enabled.

Not sure if this is a result of the mixed AMD and nVidia environment, or a result of adding the 1070 card.

Since the CPU is a hyper threaded quad core, i've always run 6 cpu tasks, and the GPUs are all configured to run two GPU tasks each. the 1070 is running CUDA, while AMD is OpenCL. In task manager I see the CPU and AMD tasks, but I don't see corresponding nVidia tasks.

* MB8_win_x64_AVX_VS2001_r3330.exe
* MB8_win_x86_SEE2_OpenCL_ATi_HD5_r3500.exe (32 bit)
* setiathome_v8 (32 bit) [are these the nvidia support tasks?]

When I installed the GTX 1070, I installed the nVidia drivers. I have seen a couple posts indicating that you might need to install the nVidia drivers first, then install AMD drivers (?) but I have not yet tried that.

Since BOINC seems to be interfacing the to the GPUs fine, I think that maybe this is a configuration issue and not a driver issue. But I'm really not sure what to do next. I am assuming there may be some XML or command line option tweaks I could try, but I really have no idea where to start with that.

I don't see a way to embed an XML file to a post, so I will add my XML files in a following post.

Hoping someone will have some suggestions for me.

Thanks!


. . HI,

. . For a card like the GTX1070 I would be running SoG (openCL) rather than CUDA, you might find that a more successful mix.

Stephen

.
ID: 1832794 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1832796 - Posted: 27 Nov 2016, 3:48:53 UTC - in response to Message 1832369.  

I just downloaded the Beta6 installer. Waiting for my current tasks to process through, then I'll update and see what happens.

Thanks for the suggestion.



. . Hi,

. . The beauty of it is you do not need to do that. When you install Beta6 it will run whatever is left in your cache with the app you select. You can even run the install with BOINC still running, but I would suggest stopping it first to guarantee a smoother installation.

Stephen

.
ID: 1832796 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1832797 - Posted: 27 Nov 2016, 3:52:17 UTC - in response to Message 1832452.  

I'm running an AMD 290 and GTX 780 in my system and it's working just fine. No lags or rendering issues, so it's not running different brand GPUs in the same computer that's causing the problem. However, some people have reported problems with the latest Nvidia driver, I read it on guru3d's site, so that could be your issue.

I do only run 4 CPU tasks, with 2 GPU tasks, since this is my main computer. It's got a 6700K @ 4.6ghz, but it ran just as well with the 2600K @ 4.6ghz, so it's unlikely that your CPU is the problem.


. . Hi Baron,

. . Apparently the latter Nvidia drivers include a "telemetry" function (reports GPU usage back to nvidia?) which can upset BOINC client and some other things.

Stephen

.
ID: 1832797 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1832800 - Posted: 27 Nov 2016, 3:59:53 UTC - in response to Message 1832689.  
Last modified: 27 Nov 2016, 4:03:07 UTC

So I updated using the Lunatics Beta 6 installer.
Currently Task Manager shows me running the following applications:
* M8_win_x86_SSW3_OpenCL_NV_SoG_r3557.exe (32-bit) (Two tasks)
* M8_win_x86_SSE2_OpenCL_ATi_HD5_r3556.exe (32-bit) (Four tasks)
* M8_win_x64_AVX_VS2010_r3330.exe (6 tasks)

My CPU is pegged at 100%, but the mouse and keyboard appears to be acting fine.
No sluggish/stuttering as I am typing this email, so if I can use the computer for that, not too worried about the 100% CPU utilization.

Still have my Usage limit set at 'use at most 80% of the CPUs', so it does seem like the GPU tasks are not paying any attention to that limit.

The unexpected thing, to me, is that the 'OpenCL_NV_SoG' tasks are taking about 12-12.6% CPU, which is a lot higher than the 'OpenCL_ATi_HD5' tasks, which vary from 0.4 to 5.0%.

Will have to keep an eye on it, and see if I can push to 3 tasks per GPU soon.


. . HI,

. . Since your CPU is an eight core unit then you should set it to use 75% of CPUs, or some increment of 12.5%. Since it is only running 6 CPU tasks it is clearly responding to the setting, but if you want it to use 7 cores for crunching then set it to 87.5%.

Stephen

.PS: . . if you want to allow more resources to support your GPUs set the other setting to less than 100%, that is use 75% of CPUs and 90% of CPU time.

.
ID: 1832800 · Report as offensive
awdorrin

Send message
Joined: 27 Sep 99
Posts: 71
Credit: 106,424,089
RAC: 261
United States
Message 1832873 - Posted: 27 Nov 2016, 16:48:20 UTC

So the issue with the mouse and keyboard has come back, I guess it is an intermittent condition.

I have tried dropping the CPU usage to 50% (4 tasks) plus the two nvidia SOG tasks, and 4 ATI tasks, (91-98% CPU utilization) but still experience the mouse/keyboard input problems.

I tried using the CPU limitation percentage, but that just results in spiky CPU utilization. Seems like all it does is stop processing for the ratio, so for 90% it will process for 9 seconds, then pause for 1 second, repeat. Which does not improve the user experience.

Seems like the nVidia GPU tasks just use a lot more CPU than the ATI tasks used, and the nVidia driver is, perhaps, not prioritize screen rendering resulting in the stalling of mouse movement and display of keyboard input. (Theorizing here...)

Currently seeing CPU utilization of:
11.2-11.8% for the four CPU tasks
13.2% for the two nVidia tasks.
5.1-7.3% for the four AMD tasks (sometimes these drop to 0.5%)

Harri, I never noticed the 'docs' folder that you pointed out. I'll have to review those ReadMe files to see if I can tune anything with those command line switches, maybe that will help.
ID: 1832873 · Report as offensive
Profile Mike Special Project $75 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 01
Posts: 34256
Credit: 79,922,639
RAC: 80
Germany
Message 1832874 - Posted: 27 Nov 2016, 16:55:11 UTC
Last modified: 27 Nov 2016, 16:55:48 UTC

Some of your GPU tasks indicates you are running low on CPU ressources.
All of your GPU`s are faster than mine.
Check my results.
Running 5 GPU tasks try running only 3 CPU tasks and see if this cures your problem.
Also i strongly suggest implementing comand line options.


With each crime and every kindness we birth our future.
ID: 1832874 · Report as offensive
awdorrin

Send message
Joined: 27 Sep 99
Posts: 71
Credit: 106,424,089
RAC: 261
United States
Message 1833250 - Posted: 30 Nov 2016, 0:43:40 UTC - in response to Message 1832874.  
Last modified: 30 Nov 2016, 0:50:40 UTC

I did try dropping the CPU tasks down to three. The NVidia GPU tasks are still taking a full core each.
Task Manager is showing me:
85% CPU with a break down of:
MB8_win_x86_SSE3_OpenCL_NV_SoG_r35576.exe   13.3%
MB8_win_x86_SSE3_OpenCL_NV_SoG_r35576.exe   13.2%
MB8_win_x64_AVX_VS2010_r3330.exe   12.8%
MB8_win_x64_AVX_VS2010_r3330.exe   12.8%
MB8_win_x64_AVX_VS2010_r3330.exe   12.5%
M8_win_x86_SSE2_OpenCL_ATi_HD5_r3557.exe 6.0%
M8_win_x86_SSE2_OpenCL_ATi_HD5_r3557.exe 5.5%
M8_win_x86_SSE2_OpenCL_ATi_HD5_r3557.exe 1.9%
M8_win_x86_SSE2_OpenCL_ATi_HD5_r3557.exe 1.5%


And the mouse and keyboard input is still very jerky.
After having the AMD OpenCL tasks use so little CPU, I have been surprised to see the NVidia OpenCL require so much CPU. Curious why that is.

Even dropping down to two CPU tasks (69-71% CPU)didn't seem to resolve the mouse and keyboard input completely (although it got a little better) - so I'm thinking it really must be the NVIDIA GPU tasks causing the issue. Although I sure would have thought that I'd see a difference with between the two monitors I have. One is connected to the NVIDIA while one is attached to the AMD. I always believed that mouse cursor movement was hardware driven.
I haven't had much time to play with the command line options yet (been too busy with work, unfortunately.)
And am still trying to understand what they mean and which would apply.
ID: 1833250 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1833271 - Posted: 30 Nov 2016, 3:35:16 UTC - in response to Message 1833250.  
Last modified: 30 Nov 2016, 3:38:34 UTC

Try my 750Ti command line, it is aggressive for mine, but very little lag - the 1070 should handle it fine ...

-sbs 256 -spike_fft_thresh 4096 -tune 1 64 1 4 -oclfft_tune_gr 256 -oclfft_tune_lr 16 -oclfft_tune_wg 256 -oclfft_tune_ls 512 -oclfft_tune_bn 64 -oclfft_tune_cw 64 -hp


PS, Was looking at your runtimes, about 2.2 times faster than be, impressive.

EDIT, I doubt it is CPU load that is causing it, I have tried running GPU and CPU tasks well beyond the 100% threshold and still my computer works fine for things like IE which is at normal priority.
ID: 1833271 · Report as offensive
1 · 2 · 3 · Next

Message boards : Number crunching : How to optimize GPU configuration?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.