Setting up Linux to crunch CUDA90 and above for Windows users

Message boards : Number crunching : Setting up Linux to crunch CUDA90 and above for Windows users
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 86 · 87 · 88 · 89 · 90 · 91 · 92 . . . 162 · Next

AuthorMessage
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1972713 - Posted: 31 Dec 2018, 0:41:08 UTC - in response to Message 1972706.  
Last modified: 31 Dec 2018, 1:16:48 UTC

You can first use the Nvidia X Server Settings app to change the Performance mode from Max Performance to Adaptive or Auto. The setting is on the PowerMizer page.

Next you can set the maximum power level for each card with nvidia-smi. This would set the max power limit to 66W for each of 3 cards for example:

nvidia-smi -i 0 -pl 66
nvidia-smi -i 1 -pl 66
nvidia-smi -i 2 -pl 66


Not sure which cards you are talking about and the minimum power limits vary among the cards. So I would first do this to find the minimum power limit you can set for each card. If you try and set too low, it will just error a message that the command is invalid.

nvidia-smi --query --display=POWER


. . Thanks Keith,

. . I have already dropped the PowerMizer setting to auto but I had a feeling you would be the guy to know other options. I will try that.

[edit]
. . You can always rely on Linux to provide the fly for the ointment. Not enough permissions to reset the power limit. :(

Stephen

thanks
ID: 1972713 · Report as offensive     Reply Quote
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1972771 - Posted: 31 Dec 2018, 5:37:11 UTC - in response to Message 1972706.  

Next you can set the maximum power level for each card with nvidia-smi. This would set the max power limit to 66W for each of 3 cards for example:

nvidia-smi -i 0 -pl 66
nvidia-smi -i 1 -pl 66
nvidia-smi -i 2 -pl 66


Not sure which cards you are talking about and the minimum power limits vary among the cards.

nvidia-smi --query --display=POWER


. . OK, found 'sudo bash' in my notes so have now set lower power limits and this has dropped temps so I could drop the fans speed on one GPU.

. . The 1 machine is running 2 x 970s, I have left the fans on those at 100% but dropped the max power on the hot card from 180W to 124W. The 2nd card is still running power max @ 180 but was running cooler anyway. The cool card is now 63 deg C and the hot card is now 67 deg C compared to 77 deg C before, that's a win. And the run times have only been increased by a few seconds so I am chuffed. Machine 2 has the 1050ti, this was running about 75 deg C but I have set the power limit to 57 instead of the default 75 and it is now running at about 66 deg C even after dropping the fan speed back to 80%. I was keen to do that as the previous 1050ti had the fans seize up from running at high speeds for prolonged periods (it is a low profile card so very small fans).

. . But thanks for your help, I am delighted with the results. Oh and the run times on the 1050ti have only increased by a similarly small margin as on the 970.

Stephen

:)
ID: 1972771 · Report as offensive     Reply Quote
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1972777 - Posted: 31 Dec 2018, 8:12:24 UTC - in response to Message 1972771.  

. . OK, found 'sudo bash' in my notes
For any command that you get a error message about rights/privileges ... just put sudo in front of it ... problem solved :)
ID: 1972777 · Report as offensive     Reply Quote
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1972795 - Posted: 31 Dec 2018, 11:55:07 UTC - in response to Message 1972777.  

. . OK, found 'sudo bash' in my notes
For any command that you get a error message about rights/privileges ... just put sudo in front of it ... problem solved :)


. . Thanks Brent. I've been through all of that before when I first set it up, but haven't much needed to bother with it while things were working OK. So naturally I have forgotten most of it.

Stephen

:)
ID: 1972795 · Report as offensive     Reply Quote
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 1973703 - Posted: 5 Jan 2019, 23:34:40 UTC - in response to Message 1967102.  

I vaguely remember something that Raistmer said about the synchronization issue. But don't know enough about how Petri implemented his search algorithm. I looked over on the CUDA developers forums and see that others are developing CUDA apps in Windows. So it must be doable in some fashion. I do see a lot of posts about issues running CUDA or installing CUDA with Visual Studio which I believe is one of the major Windows development tools.

Well, to make it more clear - I didn't work directly with CUDA for few years already so hardly could tell smth about its modern implementation in Windows.
What I could tell is the way nVidia implemented OpenCL Runtime for their devices.
If I recall correctly, memory transfers that expected to be async actually were blocking sync ones.
Perhaps there are some comments in code where I discovered that. But it's not new, NV has own language and hardly will put too much efforts in anything other support.

BTW, speaking about Petri's app what most current sources of it we have?
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 1973703 · Report as offensive     Reply Quote
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1973706 - Posted: 5 Jan 2019, 23:48:32 UTC - in response to Message 1973703.  

BTW, speaking about Petri's app what most current sources of it we have?


. . Since you have asked I suspect Petri would be able to answer that best. There are several versions for different levels of CUDA.

Stephen

:)
ID: 1973706 · Report as offensive     Reply Quote
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1973746 - Posted: 6 Jan 2019, 4:48:14 UTC - in response to Message 1973703.  

I vaguely remember something that Raistmer said about the synchronization issue. But don't know enough about how Petri implemented his search algorithm. I looked over on the CUDA developers forums and see that others are developing CUDA apps in Windows. So it must be doable in some fashion. I do see a lot of posts about issues running CUDA or installing CUDA with Visual Studio which I believe is one of the major Windows development tools.

Well, to make it more clear - I didn't work directly with CUDA for few years already so hardly could tell smth about its modern implementation in Windows.
What I could tell is the way nVidia implemented OpenCL Runtime for their devices.
If I recall correctly, memory transfers that expected to be async actually were blocking sync ones.
Perhaps there are some comments in code where I discovered that. But it's not new, NV has own language and hardly will put too much efforts in anything other support.

BTW, speaking about Petri's app what most current sources of it we have?

Nothing that we plebians have access to. Only Petri has sources or if Richard is correct, the source code must be published somewhere because of GPL. He said that Petri must have the source code over in the Lunatics beta site which only developers have access to.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1973746 · Report as offensive     Reply Quote
Sleepy
Volunteer tester
Avatar

Send message
Joined: 21 May 99
Posts: 219
Credit: 98,947,784
RAC: 28,360
Italy
Message 1975391 - Posted: 16 Jan 2019, 11:52:48 UTC

Dear all,
while fiddling with my installation, I have noticed that, at least in my experience, the TBar Boinc version does not handle correctly the option to pause GPU work when the PC is in use.

I can enable and disable the GPU, but not that particular option.

With the stock Boinc all works as expected.

Just in case you may have different experience or to avoid others getting mad trying to have it work. If there is a solution to this or I am wrong, then even better.

Sleepy
ID: 1975391 · Report as offensive     Reply Quote
J. Mileski
Volunteer tester
Avatar

Send message
Joined: 9 Jun 02
Posts: 632
Credit: 172,116,532
RAC: 572
United States
Message 1975395 - Posted: 16 Jan 2019, 12:01:40 UTC - in response to Message 1975391.  

I don't know how others are doing it, but I suspend all the gpu wu except for the ones processing and allow them to complete. Then I can do a manual switch.
It is a known issue in TBar's gpu app that it cannot be paused. But I don't remember if the wu starts over when resumed or errors out the wu.
ID: 1975395 · Report as offensive     Reply Quote
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1975397 - Posted: 16 Jan 2019, 12:09:05 UTC - in response to Message 1975391.  
Last modified: 16 Jan 2019, 12:10:07 UTC

Tbar could explain better but with this build you can't suspend the task in the middle of it's work, so you can't pause the GPU while the host is running or your WU will be crash. That is explained on the help file IIRC.

Remember this builds are highly optimized and was made for heavy cruncher hosts who don't suspend the work.
ID: 1975397 · Report as offensive     Reply Quote
Sleepy
Volunteer tester
Avatar

Send message
Joined: 21 May 99
Posts: 219
Credit: 98,947,784
RAC: 28,360
Italy
Message 1975398 - Posted: 16 Jan 2019, 12:10:01 UTC - in response to Message 1975395.  

I know about the issue of the Seti applications themselves.

But my invalids are normally at 0 and inconclusives are at normal levels, however I behave.

This is more a problem of the client, as I see it.

But your reminder of caution about suspending and restarting GPU tasks with the special Linux application is well placed.

Sleepy
ID: 1975398 · Report as offensive     Reply Quote
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1975407 - Posted: 16 Jan 2019, 13:10:59 UTC - in response to Message 1975398.  

Perhaps you can explain what you believe is incorrect about the client? Since your computers are hidden I don't know even which version you are using. I just tested 7.8.3 and it seems to work perfectly normal.
All the GPU tasks Suspend and when Resumed they all resume, even the running tasks. Since I have the preferences set to not leave a checkpoint, the two running tasks restart from the beginning, just as they should. If they don't restart from the beginning there is a good chance one or both will immediately overflow when resumed giving the wrong result. It's been like that from the beginning years ago, before I compiled the client using the stock code. I see nothing wrong with the client here.
ID: 1975407 · Report as offensive     Reply Quote
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1975436 - Posted: 16 Jan 2019, 16:50:07 UTC

i never had an issue with pausing GPU jobs either. they pause and resume just like any other client for me. resuming usually starts the job over from the beginning.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1975436 · Report as offensive     Reply Quote
Sleepy
Volunteer tester
Avatar

Send message
Joined: 21 May 99
Posts: 219
Credit: 98,947,784
RAC: 28,360
Italy
Message 1975467 - Posted: 16 Jan 2019, 20:21:10 UTC - in response to Message 1975407.  

Dear TBar,
to begin with, a big thank you for the software you have shared.

I am actually using the 7.4.44 Boinc version that I think you compiled, if I remember correctly.

I am running under Kubuntu 18.04 LTS

The BOINC package on the repository works as expected for me as well. When I use the same settings with 7.4.4 then I can switch GPU ON and OFF, but, at least the last time I tried before reverting to stock on one PC, the "idle when the PC is in use" setting was not working. The GPU job continues to run whatever I do.
This is happening in 2 PCs. In the one still running 7.4.4, I do not need the option, but it is not working either. In any case, I configured and run both PCs, therefore I may have done the same mistake in both.

Of course, as advised, I also have the write to disk time for checkpoints set larger than the time necessary to complete a MB task to avoid the overflow problem.

All the best,
Sleepy
ID: 1975467 · Report as offensive     Reply Quote
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1975472 - Posted: 16 Jan 2019, 20:50:03 UTC

I have no problems suspending any or all gpu tasks on either the 7.4.44 or 7.8.3 or 7.15.0 clients. I have never not used the "use gpu while computer in use" option so I have never seen how that functions if unchecked.

From the discussions that both Jord and Richard have posted in, that part of the code is so conditional on so many variables in the totality of BOINC and all projects the client may be attached to with many different deadlines, I got the impression if that feature works, it is mostly a one-off event.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1975472 · Report as offensive     Reply Quote
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1975550 - Posted: 17 Jan 2019, 6:03:03 UTC - in response to Message 1975467.  

...using the 7.4.44 Boinc version....
I am running under Kubuntu 18.04 LTS
So, you're having problems with the Oldest BOINC version and the Newest LTS version. You might try the much newer 7.8.3 version and see how that works. The only purpose for 7.4.44 is to provide added caching ability for the fastest machines to allow them to work through a typical Tuesday outage. If you don't need extra WUs to work through an outage, you should be using the 7.8.3 version. It is much newer and the only change from stock was to leave out an unneeded 'workaround' aimed at an older version of wxWidgets that isn't even used in 7.8.3 and had been causing problems for Years in the newer versions of BOINC. The 7.8.3 version was the First version since 7.4.42 that didn't suffer from the Tasks page jumping to the bottom of the page during most scheduler updates. For many, 7.8.3 was the first usable version of BOINC for Linux in Years. It certainly was for me.
ID: 1975550 · Report as offensive     Reply Quote
Profile petri33
Volunteer tester

Send message
Joined: 6 Jun 02
Posts: 1668
Credit: 623,086,772
RAC: 156
Finland
Message 1975823 - Posted: 18 Jan 2019, 21:51:37 UTC - in response to Message 1973746.  

Hi All,

Whenever I release an executable I release the source code too. I deliver both links to my repository (a personal Google drive with limited access to the files). The GPL mandates me to do so and have the source available for download for three years.

My executables are not always compatible with the total variety of hardware so you may have acquired a link to a more wide-use executable download and that download site should publish the source code too. I like to believe the code is somewhere out there -- in the X folder.

The most recent source code resides on my SSD and you can always ask for a copy of it. It may not work, it may be slower than the previous or have some unintended bugs lurking inside of it since it is not tested for reliability but for any possible speed gain.

... and the latest ideas of how to gain more speed (while maintaining compatibility) reside in my head and I'm quite confident that those ideas can not be downloaded from anywhere.

Petri
To overcome Heisenbergs:
"You can't always get what you want / but if you try sometimes you just might find / you get what you need." -- Rolling Stones
ID: 1975823 · Report as offensive     Reply Quote
Lane42

Send message
Joined: 17 May 99
Posts: 59
Credit: 227,150,556
RAC: 11
United States
Message 1975996 - Posted: 20 Jan 2019, 2:00:11 UTC - in response to Message 1975823.  

and the latest ideas of how to gain more speed (while maintaining compatibility) reside in my head and I'm quite confident that those ideas can not be downloaded from anywhere. (-: (-:
ID: 1975996 · Report as offensive     Reply Quote
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1976945 - Posted: 25 Jan 2019, 16:16:13 UTC - in response to Message 1970166.  

Tnx gentlemen. I see from Nvidia a 415.xx is available as a .run file. I will see what is available from the repositories as well. This will be the first time I have tried to install a special app. (Cuda app) into Boinc. Is there a readme that covers the steps to do that part?


i wouldn't bother with the 415 driver. just install the 396 or 410 driver from the Ubuntu graphics drivers PPA.

since you already have the repository version of BOINC installed, it would be easiest to just copy and paste the seven (7) package files into your existing boinc project directory.

1. shutdown/close boinc manager
2. stop the boinc-client service with the following command in the terminal : "sudo /etc/init.d/boinc-client stop"
3. copy the seven (7) package files into the directory : /var/lib/boinc-client/projects/setiathome.berkeley.edu/
4. start the boinc-client service with the following command in the terminal: "sudo /etc/init.d/boinc-client start"
5. start the boinc manager


I ended up with a repository version of Boinc Manager installed after getting my Lubuntu screwed up/re-installed. But any attempt to access the projects, notices or other folders under boinc-client gives me a "permission denied". When I check the properties it claims that anyone can view it. Etc.

What am I missing or shall I re-install again and try to find the "all-in-one"?

Tom
A proud member of the OFA (Old Farts Association).
ID: 1976945 · Report as offensive     Reply Quote
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1976949 - Posted: 25 Jan 2019, 16:29:42 UTC - in response to Message 1976945.  

you have to change permissions to make the apps executable, not just able to "view".

Why do you keep torturing yourself with Lubuntu? just install the vanilla Ubuntu 18.04 and make your life easier.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1976949 · Report as offensive     Reply Quote
Previous · 1 . . . 86 · 87 · 88 · 89 · 90 · 91 · 92 . . . 162 · Next

Message boards : Number crunching : Setting up Linux to crunch CUDA90 and above for Windows users


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.