Setting up Linux to crunch CUDA90 and above for Windows users

Message boards : Number crunching : Setting up Linux to crunch CUDA90 and above for Windows users
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 150 · 151 · 152 · 153 · 154 · 155 · 156 . . . 162 · Next

AuthorMessage
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 2031993 - Posted: 11 Feb 2020, 23:45:33 UTC - in response to Message 2031988.  

I think he’s saying that he doesn’t see any difference in behavior. Effectively it’s not checkpointing even before the change. So the checkpoint issue is probably a non-issue anymore.


If that is the case we could forget about and ask Tbar correct the AIO doc.
ID: 2031993 · Report as offensive     Reply Quote
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13854
Credit: 208,696,464
RAC: 304
Australia
Message 2032043 - Posted: 12 Feb 2020, 4:22:32 UTC - in response to Message 2031965.  
Last modified: 12 Feb 2020, 4:22:51 UTC

I am running Linux Mint 19.2 with dual Nvidia RTX 2070 super and have had no issues for the past two days. I installed driver 440.59 and it works great. Be warned though, these are not official drivers.
You might want to give the -nobs option a go as well as reserve a CPU core to support each GPU WU if you haven't already done so.
My RTX 2060 Super is doing blc66s in around 1min 2sec, your RTX 2070 Super is taking around 1min 50sec.
Grant
Darwin NT
ID: 2032043 · Report as offensive     Reply Quote
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2032046 - Posted: 12 Feb 2020, 4:56:05 UTC - in response to Message 2031965.  

I am running Linux Mint 19.2 with dual Nvidia RTX 2070 super and have had no issues for the past two days. I installed driver 440.59 and it works great. Be warned though, these are not official drivers.

They are about as "official" as you can get. Just managed by someone other than Nvidia or Ubuntu.
Still the same source code being compiled.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2032046 · Report as offensive     Reply Quote
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 2032066 - Posted: 12 Feb 2020, 11:58:18 UTC - in response to Message 2032046.  

Ubuntu devs run that PPA.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 2032066 · Report as offensive     Reply Quote
Profile Buckeye4LF Project Donor
Avatar

Send message
Joined: 19 Jun 00
Posts: 173
Credit: 54,916,209
RAC: 833
United States
Message 2032069 - Posted: 12 Feb 2020, 12:20:12 UTC - in response to Message 2032043.  

I still need to tweak settings for sure. I just got up and running and when I attached to all my backup projects I downloaded a ton of WUs from them. It will be a week until this machine is only running seti. I installed the AIO and I thought that I saw -noobs set already in the supplied config file. I will have to revisit for sure as I should be under a min WU time

ID: 2032069 · Report as offensive     Reply Quote
Profile Buckeye4LF Project Donor
Avatar

Send message
Joined: 19 Jun 00
Posts: 173
Credit: 54,916,209
RAC: 833
United States
Message 2032070 - Posted: 12 Feb 2020, 12:24:11 UTC - in response to Message 2032043.  
Last modified: 12 Feb 2020, 12:29:15 UTC

I have my count set to .33 on my gpu and i suspect that is the culprit since I am running three wu per gpu. I just switched it to .5 to see if that helps

I trimmed 30 seconds off of each run by going to .5 count. I am not sure which is better a slightly longer run but multiples at a time or balls to the wall and set to 1 for 1 at a time

ID: 2032070 · Report as offensive     Reply Quote
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 2032071 - Posted: 12 Feb 2020, 12:42:42 UTC

Unless you choose to run the mutex option, with the Linux Special Sauce you must set your host to run only 1 (one) WU at a time.
ID: 2032071 · Report as offensive     Reply Quote
Profile Buckeye4LF Project Donor
Avatar

Send message
Joined: 19 Jun 00
Posts: 173
Credit: 54,916,209
RAC: 833
United States
Message 2032072 - Posted: 12 Feb 2020, 12:49:36 UTC - in response to Message 2032071.  

I just changed it to 1 and they are coming in under a minute. I will look into mutex and other tweaks this weekend. Thanks for catching that my times were off as I had not had a chance to cross check them.

ID: 2032072 · Report as offensive     Reply Quote
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 2032074 - Posted: 12 Feb 2020, 13:10:23 UTC - in response to Message 2032072.  

If you have a fast SSD or m2 storage you won't see any improvement using the mutex version. When I tested it, it was actually a little slower than the standard version. I noticed your CPUs are showing a higher CPU-time than Run-time, which is odd, it's usually the other way around. Are you using the default Intel version of the r3711 CPU App or the AMD version? If you look at the BOINC/projects/setiathome.berkeley.edu/docs/README_x41p_V0.98.txt it will tell you about switching CPU Apps, and a few other pointers. I would suggest using the Newer CPU App r4101 which I compiled with Ryzen flags, it's the fastest CPU App I've made yet. It will be in the next All-In-One folder as soon as I finish working on building an updated version of BOINC. So far I'm still getting some strange Upload problems with 7.16.x that I'm not getting with 7.14.2. The New CPU App is in this post, https://setiathome.berkeley.edu/forum_thread.php?id=85086&postid=2029125#2029125
ID: 2032074 · Report as offensive     Reply Quote
Profile Buckeye4LF Project Donor
Avatar

Send message
Joined: 19 Jun 00
Posts: 173
Credit: 54,916,209
RAC: 833
United States
Message 2032077 - Posted: 12 Feb 2020, 13:22:00 UTC - in response to Message 2032074.  

I am using the amd app, i changed the flag in the app_info file. I will defiantly grab the 4101 file. I am not sure what is going on yet..... thanks for your suggestions.

ID: 2032077 · Report as offensive     Reply Quote
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 2032080 - Posted: 12 Feb 2020, 13:53:57 UTC - in response to Message 2032077.  

Hmmm, strange the CPU times are higher than Run times. Let's see how the New CPU App works. The r3711 version was built before anyone was able to compile a working optimized CPU App in Ubuntu 18.04, so, it used the older Bulldozer flags with some cpu calls added and removed. The r4101 was compiled in 18.04 so it could use the standard Ryzen flags. It will probably need at least 18.04 or higher to work, so, it may not work with any lower OS version, just in case someone has any ideas about trying it in 16.04.
ID: 2032080 · Report as offensive     Reply Quote
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14679
Credit: 200,643,578
RAC: 874
United Kingdom
Message 2032098 - Posted: 12 Feb 2020, 17:45:47 UTC - in response to Message 2031988.  

I think he’s saying that he doesn’t see any difference in behavior. Effectively it’s not checkpointing even before the change. So the checkpoint issue is probably a non-issue anymore.
Yes, that's exactly what I meant. It really does come at the very end of the run:

Wed 12 Feb 2020 17:41:18 GMT | SETI@home | [checkpoint] result blc41_2bit_guppi_58838_00416_TIC434234955_0014.21668.409.19.28.252.vlar_1 checkpointed
Wed 12 Feb 2020 17:41:20 GMT | SETI@home | Computation for task blc41_2bit_guppi_58838_00416_TIC434234955_0014.21668.409.19.28.252.vlar_1 finished
Wed 12 Feb 2020 17:41:35 GMT | SETI@home | [checkpoint] result blc41_2bit_guppi_58838_00416_TIC434234955_0014.22110.409.19.28.227.vlar_1 checkpointed
Wed 12 Feb 2020 17:41:37 GMT | SETI@home | Computation for task blc41_2bit_guppi_58838_00416_TIC434234955_0014.22110.409.19.28.227.vlar_1 finished
What chance of BOINC exiting or the power dying in those 2 seconds?
ID: 2032098 · Report as offensive     Reply Quote
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 2032113 - Posted: 12 Feb 2020, 20:02:28 UTC - in response to Message 2032098.  
Last modified: 12 Feb 2020, 20:03:00 UTC

I think he’s saying that he doesn’t see any difference in behavior. Effectively it’s not checkpointing even before the change. So the checkpoint issue is probably a non-issue anymore.
Yes, that's exactly what I meant. It really does come at the very end of the run:

Wed 12 Feb 2020 17:41:18 GMT | SETI@home | [checkpoint] result blc41_2bit_guppi_58838_00416_TIC434234955_0014.21668.409.19.28.252.vlar_1 checkpointed
Wed 12 Feb 2020 17:41:20 GMT | SETI@home | Computation for task blc41_2bit_guppi_58838_00416_TIC434234955_0014.21668.409.19.28.252.vlar_1 finished
Wed 12 Feb 2020 17:41:35 GMT | SETI@home | [checkpoint] result blc41_2bit_guppi_58838_00416_TIC434234955_0014.22110.409.19.28.227.vlar_1 checkpointed
Wed 12 Feb 2020 17:41:37 GMT | SETI@home | Computation for task blc41_2bit_guppi_58838_00416_TIC434234955_0014.22110.409.19.28.227.vlar_1 finished
What chance of BOINC exiting or the power dying in those 2 seconds?

I could remove this checkpoint too.
ID: 2032113 · Report as offensive     Reply Quote
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 2032115 - Posted: 12 Feb 2020, 20:03:37 UTC - in response to Message 2032113.  

i don't think it's worth it.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 2032115 · Report as offensive     Reply Quote
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 2032116 - Posted: 12 Feb 2020, 20:34:59 UTC - in response to Message 2032070.  
Last modified: 12 Feb 2020, 20:40:17 UTC

I have my count set to .33 on my gpu and i suspect that is the culprit since I am running three wu per gpu. I just switched it to .5 to see if that helps

I trimmed 30 seconds off of each run by going to .5 count. I am not sure which is better a slightly longer run but multiples at a time or balls to the wall and set to 1 for 1 at a time


. . It doesn't pay to run multiple instance of special sauce on a GPU, you need to drop it to one task unless you are running the mutex version which preloads one to hold 2 at a time but it only runs the first one.


. . Late to the party again ...

Stephen

:(
ID: 2032116 · Report as offensive     Reply Quote
Profile Buckeye4LF Project Donor
Avatar

Send message
Joined: 19 Jun 00
Posts: 173
Credit: 54,916,209
RAC: 833
United States
Message 2032118 - Posted: 12 Feb 2020, 20:49:06 UTC - in response to Message 2032116.  

No worries, I only check this a few times a day and usually am behind the conversation when I do. I have the count set to 1 and I am much faster now. I look forward to tweaking later this weekend.

What I really need to do is put a custom cooling loop in for my CPU and OC the shit out of it........

ID: 2032118 · Report as offensive     Reply Quote
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2032121 - Posted: 12 Feb 2020, 21:38:25 UTC - in response to Message 2032118.  

No worries, I only check this a few times a day and usually am behind the conversation when I do. I have the count set to 1 and I am much faster now. I look forward to tweaking later this weekend.

What I really need to do is put a custom cooling loop in for my CPU and OC the shit out of it........

PM me for water cooling suggestions. I am rebuilding the slowest rig today into a water cooled version. Trying to get ahead of the summer temps effect on the 3900X.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2032121 · Report as offensive     Reply Quote
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5126
Credit: 276,046,078
RAC: 462
Message 2032134 - Posted: 13 Feb 2020, 0:16:54 UTC - in response to Message 2031969.  

i installed openCL before I updated the driver so you may have to do that as a separate command as I am unsure if it is included.
Can you remember where you got OpenCL from? Less important, I can come back to that later - it's CUDA I was planning to test today.


Its available from the Ubuntu library (OpenCL).

Let me find my notes.

sudo apt-get install ocl-icd-libopencl1
Has been working for me.

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt-get update


Once do that you can install either with a commandline or use the OS update program. It has a tab for "additional" drivers that shows up nicely.

Tom
A proud member of the OFA (Old Farts Association).
ID: 2032134 · Report as offensive     Reply Quote
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14679
Credit: 200,643,578
RAC: 874
United Kingdom
Message 2032213 - Posted: 13 Feb 2020, 12:35:44 UTC - in response to Message 2032134.  

sudo apt-get install ocl-icd-libopencl1
Has been working for me.

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt-get update


Once do that you can install either with a commandline or use the OS update program. It has a tab for "additional" drivers that shows up nicely.
Thanks for the advice everyone. Looks like my existing OpenCL setup has survived the driver update:

Thu 13 Feb 2020 12:18:29 GMT |  | CUDA: NVIDIA GPU 0: GeForce GTX 1660 SUPER (driver version 440.59, CUDA version unknown, compute capability 7.5, 4096MB, 3974MB available, 5153 GFLOPS peak)
Thu 13 Feb 2020 12:18:29 GMT |  | OpenCL: NVIDIA GPU 0: GeForce GTX 1660 SUPER (driver version 440.59, device version OpenCL 1.2 CUDA, 5943MB, 5943MB available, 5153 GFLOPS peak)
and as a bonus I've got my GPUs properly named.

The remaining mystery is that 'CUDA version unknown'. The machine is crunching happily on the CUDA 10.2 app: would I be right in assuming that the huge size of the app (230 MB!) implies that it's a full static build with all the libraries like FFT linked in? Not that I'm ever likely to run another CUDA app on it (unless GPUGrid come back to full production), but it would seem to defeat the general principle of shared libraries if everybody has to download another copy every time.

[Having said that, the whole design of BOINC negates the principle of sharing libraries - as the people failing to run x64 clients on low-power GPUs has demonstrated. That old, unshared libeay32.dll, and the non-updated ca-bundle.crt, give the game away. But that's an off-topic Windows comment.]
ID: 2032213 · Report as offensive     Reply Quote
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5126
Credit: 276,046,078
RAC: 462
Message 2032220 - Posted: 13 Feb 2020, 13:52:16 UTC - in response to Message 2032213.  


The remaining mystery is that 'CUDA version unknown'. The machine is crunching happily on the CUDA 10.2 app: would I be right in assuming that the huge size of the app (230 MB!) implies that it's a full static build with all the libraries like FFT linked in? Not that I'm ever likely to run another CUDA app on it (unless GPUGrid come back to full production), but it would seem to defeat the general principle of shared libraries if everybody has to download another copy every time.

[Having said that, the whole design of BOINC negates the principle of sharing libraries - as the people failing to run x64 clients on low-power GPUs has demonstrated. That old, unshared libeay32.dll, and the non-updated ca-bundle.crt, give the game away. But that's an off-topic Windows comment.]


I think I remember a discussion where someone said the current "special sauce" code is compiled with all the test instrumentation turned on. And someone said if you change the compiler flags (somehow) you can generate a much smaller .exe file?

If this were a commercial operation it is likely we would want to shrink the resource usage footprint to the smallest possible. But ease of installation trumps the lowest resource usage every time in a volunteer-based project.

Tom
A proud member of the OFA (Old Farts Association).
ID: 2032220 · Report as offensive     Reply Quote
Previous · 1 . . . 150 · 151 · 152 · 153 · 154 · 155 · 156 . . . 162 · Next

Message boards : Number crunching : Setting up Linux to crunch CUDA90 and above for Windows users


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.