Setting up Linux to crunch CUDA90 and above for Windows users

Message boards : Number crunching : Setting up Linux to crunch CUDA90 and above for Windows users
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 97 · 98 · 99 · 100 · 101 · 102 · 103 . . . 131 · Next

AuthorMessage
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 9324
Credit: 833,487,779
RAC: 1,959,343
United States
Message 1990983 - Posted: 21 Apr 2019, 16:41:13 UTC

From what I gather from the daily announcements at phoronix.com, that is an inevitability since the major push in development seems to be being able to pass through a gpu with full compute capabilities in a VM. Vulkan group and the Mesa group publish new developments it seems daily, surely at least once a week. No idea when full pass through may happen.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1990983 · Report as offensive     Reply Quote
Profile M_M
Avatar

Send message
Joined: 20 May 04
Posts: 65
Credit: 38,754,440
RAC: 17,001
Serbia
Message 1991669 - Posted: 27 Apr 2019, 7:05:30 UTC

Yesterday I pushed myself to install Ubuntu 18.04 LTS on external usb3 hdd and managed Boinc (TBar All-In-One package) with latest 0.98b1 Cuda10.1 to work with nVidia 418 drivers. :) Total time needed about 2hrs, counting from Ubuntu ISO download to all installed and crunching Seti WUs. It was a nice experience I have to say.... Now I have UEFI dual boot, Win10 and Ubuntu, so also chance to learn Linux/Ubuntu a bit more...

It really wasn't difficult - actually went easier then I initially expected even for last 12-13 years I had no any experience with any Linux distribution, but only Windows - back in mid 2000`s I worked with Sun Solaris 9 for some time at work.

0.98b1 seems like a really big improvement in crunching efficiency/speed on my gtx 1070 - according to WU completion times, I would say at 2.5-3x comparing to parameter optimized SoG on Win10.

I would like to thank Petri33 & TBar on awesome work done on customized app development, and also everyone else on guides/instructions/tips&tricks in this and other Linux threads on how to make it work.
ID: 1991669 · Report as offensive     Reply Quote
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 9324
Credit: 833,487,779
RAC: 1,959,343
United States
Message 1991672 - Posted: 27 Apr 2019, 7:40:52 UTC - in response to Message 1991669.  

Welcome to the party. Try the dips and beer!
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1991672 · Report as offensive     Reply Quote
Filipe

Send message
Joined: 12 Aug 00
Posts: 200
Credit: 9,198,251
RAC: 1,865
Portugal
Message 1991726 - Posted: 27 Apr 2019, 19:32:24 UTC

Is it possible to run the Petri33 special sauce through Boinc virtual box?
ID: 1991726 · Report as offensive     Reply Quote
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 9324
Credit: 833,487,779
RAC: 1,959,343
United States
Message 1991729 - Posted: 27 Apr 2019, 20:11:32 UTC - in response to Message 1991726.  

Is it possible to run the Petri33 special sauce through Boinc virtual box?

I don't think so. Not an expert, far from it, in knowledge of virtual hosting. I thought that no virtual hosting service allows pass-through of gpu cards for compute. Hope that someone more knowledgeable chimes in.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1991729 · Report as offensive     Reply Quote
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 1626
Credit: 626,724,774
RAC: 2,959,004
United States
Message 1991737 - Posted: 27 Apr 2019, 21:19:35 UTC - in response to Message 1991729.  
Last modified: 27 Apr 2019, 21:25:07 UTC

you certainly can pass through GPU to the VM with some hypervisors, doesnt matter if you're doing compute or gaming, once it's passed through, the OS can use it. but there has been some need recently to obfuscate the pass through in some cases. I don't remember the exact details, but I think there is a driver issue when using normal Geforce cards in a VM. Nvidia doesnt want you doing this kind of thing with the cheaper cards. they want to restrict you to the professional Quadro and Tesla cards.

but it will depend on a lot of factors, including your exact hardware setup for motherboard and CPU. and in some cases, even if the CPU and MB hardware support it, you could be locked out if the MB BIOS doesnt have the proper features enabled.

in terms of VirtualBox, looks like the feature is experimental and it's only supported on Linux hosts, which defeats the purpose if you're trying to run a Linux VM from within Windows.

from here: https://www.virtualbox.org/manual/ch09.html

When running on Linux hosts with a kernel version later than 2.6.31, experimental host PCI devices passthrough is available.
Note
The PCI passthrough module is shipped as a Oracle VM VirtualBox extension package, which must be installed separately. See Section 1.6, “Installing Oracle VM VirtualBox and Extension Packs”.


This feature enables a guest to directly use physical PCI devices on the host, even if host does not have drivers for this particular device. Both, regular PCI and some PCI Express cards, are supported. AGP and certain PCI Express cards are not supported at the moment if they rely on Graphics Address Remapping Table (GART) unit programming for texture management as it does rather non-trivial operations with pages remapping interfering with IOMMU. This limitation may be lifted in future releases.

To be fully functional, PCI passthrough support in Oracle VM VirtualBox depends upon an IOMMU hardware unit which is not yet too widely available. If the device uses bus mastering, for example it performs DMA to the OS memory on its own, then an IOMMU is required. Otherwise such DMA transactions may write to the wrong physical memory address as the device DMA engine is programmed using a device-specific protocol to perform memory transactions. The IOMMU functions as translation unit mapping physical memory access requests from the device using knowledge of the guest physical address to host physical addresses translation rules.

Intel's solution for IOMMU is called Intel Virtualization Technology for Directed I/O (VT-d), and AMD's solution is called AMD-Vi. Check your motherboard datasheet for the appropriate technology. Even if your hardware does not have a IOMMU, certain PCI cards may work, such as serial PCI adapters, but the guest will show a warning on boot and the VM execution will terminate if the guest driver will attempt to enable card bus mastering.

It is very common that the BIOS or the host OS disables the IOMMU by default. So before any attempt to use it please make sure that the following apply:

Your motherboard has an IOMMU unit.

Your CPU supports the IOMMU.

The IOMMU is enabled in the BIOS.

The VM must run with VT-x/AMD-V and nested paging enabled.

Your Linux kernel was compiled with IOMMU support, including DMA remapping. See the CONFIG_DMAR kernel compilation option. The PCI stub driver (CONFIG_PCI_STUB) is required as well.

Your Linux kernel recognizes and uses the IOMMU unit. The intel_iommu=on boot option could be needed. Search for DMAR and PCI-DMA in kernel boot log.

Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1991737 · Report as offensive     Reply Quote
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 3384
Credit: 176,825,731
RAC: 829,535
United States
Message 1991887 - Posted: 29 Apr 2019, 13:21:55 UTC - in response to Message 1991726.  

Is it possible to run the Petri33 special sauce through Boinc virtual box?


Depends exactly what you mean by "Boinc virtual box"? If you are talking about downloading an image directly from the project (which is what I think you maybe talking about) then I don't think it is available.

However, I do know that there are bio's settings on several different motherboards that support hardware level pass through for virtual machines. Some of the toggles appear to refer to gpu-based pass through.

If you are going to experiment, I would first get the petri33 special sauce running on native hardware. Then I would replicate that hardware in a virtual machine and copy/move the "all in one" folder to that virtual machine.

Because there has to be some "overhead" when running a virtual machine I don't expect you to have as high a production as you would on native hardware. Because the programming takes advantage of knowing the way the hardware works, there is a possibility that it will not work because the interface(s) on the VM are not exactly the same as native hardware.

My impression is it should be possible because many of the VM instances that are being sold on the cloud are being touted has being able to do Machine Learning (lots of gpu's) and most are running as non-dedicated VM's.

This is all (slightly) informed speculation on my part.
Your results may vary (considerably).

Tom
I will stop procrastinating tomorrow.
\\// Live Long & Prosper (starting tomorrow ;)
ID: 1991887 · Report as offensive     Reply Quote
Filipe

Send message
Joined: 12 Aug 00
Posts: 200
Credit: 9,198,251
RAC: 1,865
Portugal
Message 1993114 - Posted: 8 May 2019, 14:10:50 UTC - in response to Message 1991887.  
Last modified: 8 May 2019, 14:11:33 UTC

Depends exactly what you mean by "Boinc virtual box"?


I was talking about virtual box, like lhc@home use within boinc. Which is available at boinc download page.

But i saw maybe it's better to build a dual boot machine? (With linux on a second drive for seti@home)

Waiting on Shaggie76's update to compare GTX1660ti Vs RTX 2060 at Seti@home
ID: 1993114 · Report as offensive     Reply Quote
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 9324
Credit: 833,487,779
RAC: 1,959,343
United States
Message 1993151 - Posted: 8 May 2019, 18:23:21 UTC - in response to Message 1993114.  
Last modified: 8 May 2019, 18:34:29 UTC

Off-topic but I see in the news that Microsoft is going to include a full complete copy of Linux in the upcoming releases. On the Insider Preview releases they are getting a new version of WSL called WSL2 or Windows Subsystem for Linux. Also they are putting the standard Terminal with cmd.exe on the backburner and moving to "Windows Terminal" which for all intents and purposes is the Terminal from Ubuntu 19.04 and later versions with tabs and other enhancements. Supposed to be coming out in June.

Windows is becoming more Linux-esque all the time.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1993151 · Report as offensive     Reply Quote
Profile Siran d'Vel'nahr
Volunteer tester
Avatar

Send message
Joined: 23 May 99
Posts: 6915
Credit: 19,884,134
RAC: 65,888
United States
Message 1993156 - Posted: 8 May 2019, 18:59:22 UTC - in response to Message 1993151.  

Hi Keith,

Windows is becoming more Linux-esque all the time.

Yes, but will Winblows become less bloated? Are they gonna stick with the NTFS file system or will they adopt the Linux file system (not sure what it is called)? Since Linux is open source and virtually free (I know some Linux are actually sold) will they charge their exorbitant price like they do for Winblows or will they reduce their prices since they really don't make squat diddly on Winblows anyway?

I kinda think that it's funny that Micro$oft should be adopting anything Linux, unless they are starting to see the light about their junky OS. Perhaps some day Winblows will go the way of the dinosaurs. Yeah, right! I ain't holdin' my breath. ;)

Have a great day! :)

Siran
CAPT Siran d'Vel'nahr XO - L L & P _\\//
USS Vre'kasht NCC-33187

Windows 10 OS? "What a piece of junk!" - L. Skywalker

Microsoft wants us to "Imagine life without walls"...
I say, "If there are no walls, who needs Windows?"
ID: 1993156 · Report as offensive     Reply Quote
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 9324
Credit: 833,487,779
RAC: 1,959,343
United States
Message 1993158 - Posted: 8 May 2019, 19:16:03 UTC - in response to Message 1993156.  

they really don't make squat diddly on Winblows anyway?

They only made $10B on Windows last year down again from past years. They make most of their profit from cloud computing..
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1993158 · Report as offensive     Reply Quote
Stephen "Heretic" Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 4572
Credit: 137,376,081
RAC: 245,854
Australia
Message 1993190 - Posted: 9 May 2019, 0:02:00 UTC - in response to Message 1993114.  

Depends exactly what you mean by "Boinc virtual box"?


I was talking about virtual box, like lhc@home use within boinc. Which is available at boinc download page.

But i saw maybe it's better to build a dual boot machine? (With linux on a second drive for seti@home)

Waiting on Shaggie76's update to compare GTX1660ti Vs RTX 2060 at Seti@home


. . If you take a peek at Bernie Vine's machines he is running a GTX1660ti on one of them under windows which would give you an idea of he relative performance with that OS. But a dual boot machine for Linux with Petri's Special Sauce would give the best results.

Stephen

:)
ID: 1993190 · Report as offensive     Reply Quote
Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 26 May 99
Posts: 9826
Credit: 79,555,851
RAC: 275,508
United Kingdom
Message 1993231 - Posted: 9 May 2019, 6:53:47 UTC

If you take a peek at Bernie Vine's machines he is running a GTX1660ti on one of them under windows which would give you an idea of he relative performance with that OS


It is currently running tasks around 6mins 20sec average. Compared with the GTX 970 on an older AMD CPU which is around 10mins 30secs average and the GTX 1060 on a newer AMD Ryzen 3 2200G which is around 9mins 50secs

Both the machines with the 970 and 1060 are dedicated crunchers that mostly do nothing else 24/7, however the GTX 1660ti is in my daily driver and is in use for gaming and watching videos for several hours a day and then Boinc is suspended, however it still has the highest RAC.

I am considering getting another 1660ti to put in the Ryzen box, move the 1060 to the older AMD box and retire the 970 which has done sterling work for several years.
ID: 1993231 · Report as offensive     Reply Quote
Dallasdawg Project Donor

Send message
Joined: 19 Aug 99
Posts: 43
Credit: 83,438,565
RAC: 261,137
United States
Message 1993278 - Posted: 9 May 2019, 15:12:36 UTC - in response to Message 1993156.  

Windows is becoming more Linux-esque all the time.
Yes, but will Winblows become less bloated? Are they gonna stick with the NTFS file system or will they adopt the Linux file system (not sure what it is called)? Since Linux is open source and virtually free (I know some Linux are actually sold) will they charge their exorbitant price like they do for Winblows or will they reduce their prices since they really don't make squat diddly on Winblows anyway?

I kinda think that it's funny that Micro$oft should be adopting anything Linux, unless they are starting to see the light about their junky OS. Perhaps some day Winblows will go the way of the dinosaurs. Yeah, right! I ain't holdin' my breath. ;)


Linux has several file systems available to use, and you can choose which you want at install. In the desktop realm, ext4 is the most popular (and probably the one you are thinking of). In the server space, you see a mix of ext4, zfs, btrfs, and a smattering of others. btrfs is gaining rapidly in popularity and not just in the server space.

Microsoft already has its own linux distro, callled SphereOS (also called Azure Sphere), and is currently targeting the IoT space.

WSL was born from several *nix projects Microsoft has done over the years....like Cygwin and the Microsoft POSIX Subsystem. MS is really not new to Linux or Unix. They have just openly changed their tone for Linux. There is still some skepticism in the Linux community of Microsoft's motives. It is hard to forget Microsoft's internal "embrace, extend, and extinguish" philosophy to enter project categories, build proprietary components, and then strong arm the original developer out of the space. One thing here that is different...outside of the desktop, they are not the dominant player.

There is a lot of talk in the Linux space about Microsoft and its adoption of Linux. They now have a seat on the board of the Linux Foundation and have appeared to cease their open hostility of Linux. There has been further talk whether Microsoft, at some point, might replace their kernel with the Linux kernel. It's not a totally crazy idea from a business perspective. Using the Linux kernel frees up developers from your internal kernel development to do other things. They can even keep Windows "Windows-y" while doing this...maintain look and feel and general operation. The only cost is they must share any code they add to the kernel as part of their agreement to the GNU General Public License (GPL).
ID: 1993278 · Report as offensive     Reply Quote
Iona
Avatar

Send message
Joined: 12 Jul 07
Posts: 786
Credit: 20,195,269
RAC: 18,575
United Kingdom
Message 1993403 - Posted: 11 May 2019, 8:12:02 UTC - in response to Message 1993231.  

If you take a peek at Bernie Vine's machines he is running a GTX1660ti on one of them under windows which would give you an idea of he relative performance with that OS


It is currently running tasks around 6mins 20sec average. Compared with the GTX 970 on an older AMD CPU which is around 10mins 30secs average and the GTX 1060 on a newer AMD Ryzen 3 2200G which is around 9mins 50secs

Both the machines with the 970 and 1060 are dedicated crunchers that mostly do nothing else 24/7, however the GTX 1660ti is in my daily driver and is in use for gaming and watching videos for several hours a day and then Boinc is suspended, however it still has the highest RAC.

I am considering getting another 1660ti to put in the Ryzen box, move the 1060 to the older AMD box and retire the 970 which has done sterling work for several years.



Retire the 970? From my perspective, it is much further away from retirement than the AMD CPU 'driving' it! I regularly suspend GPU tasks for about five hours a day, to do a bit of gaming.....this is my 'daily', running an 'old hat' 3570K and a 970. If you can't 'drive' the GPU sufficiently well enough, it doesn't really matter what GPU is 'strapped' to the CPU. This is 'gamer' sort of stuff and does have some relevance. Lets face it, what would be the point of having a Triumph T100A engine, in a Rolls-Royce Phantom V? Same thing.
Don't take life too seriously, as you'll never come out of it alive!
ID: 1993403 · Report as offensive     Reply Quote
Stephen "Heretic" Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 4572
Credit: 137,376,081
RAC: 245,854
Australia
Message 1993489 - Posted: 12 May 2019, 0:23:47 UTC - in response to Message 1993403.  
Last modified: 12 May 2019, 0:41:42 UTC

Retire the 970? From my perspective, it is much further away from retirement than the AMD CPU 'driving' it! I regularly suspend GPU tasks for about five hours a day, to do a bit of gaming.....this is my 'daily', running an 'old hat' 3570K and a 970. If you can't 'drive' the GPU sufficiently well enough, it doesn't really matter what GPU is 'strapped' to the CPU. This is 'gamer' sort of stuff and does have some relevance. Lets face it, what would be the point of having a Triumph T100A engine, in a Rolls-Royce Phantom V? Same thing.


. . My Gainward GTX970s are a long way from retirement. I am running the penultimate version of Cuda90 on that i5-6600 machine but I think run times for Blc33 tasks of 2.8 mins makes it well worthwhile keeping in production.

. . But even older CPUs can be useful. My GTX1050ti running the same version of Cuda90 on a Core 2 Duo E7600 box is doing those tasks in 4.4 mins. Of course, there is NO CPU crunching being done an that machine, the CPU is purely life support for the 1050ti.

. . Both machines are dedicated crunchers.

Stephen

:)
ID: 1993489 · Report as offensive     Reply Quote
Profile -= Vyper =-
Volunteer tester
Avatar

Send message
Joined: 5 Sep 99
Posts: 1557
Credit: 803,442,621
RAC: 1,094,741
Sweden
Message 1993523 - Posted: 12 May 2019, 11:48:41 UTC - in response to Message 1993489.  

I tend to agree.

The 970 is actually quite strong contender and doesnt go much further than 160W or so when crunching.
With the special sauce it still produces around 42K Rac without much trouble and in terms of bang/buck when looking on the second hand markets you actually can build quite a cruncher if you get held of a Mining board with lots of PCIe's and those cheap 970s.

Take a descent cpu, 8GB ram (Perhaps) , SSD , and two good PSUs, couple that with 10 pcs of old 970s and you have a box that pulls around 1,8Kw producing 400K Rac+ for not that much money (compared what it can cost when purchasing new stuff)

So all in all i agree that the 970s are still quite good power/performance ratio but is certainly not comparable to a 2080Ti that pulls 185W and produces roughly 150K/day but for the cost of one 2080Ti you could buy almost 15 GTX 970s in my country nonetheless.

Thumbs up for 970 & special sauce.

_________________________________________________________________________
Addicted to SETI crunching!
Founder of GPU Users Group
ID: 1993523 · Report as offensive     Reply Quote
Stephen "Heretic" Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 4572
Credit: 137,376,081
RAC: 245,854
Australia
Message 1993595 - Posted: 12 May 2019, 23:31:28 UTC - in response to Message 1993523.  

I tend to agree.
The 970 is actually quite strong contender and doesnt go much further than 160W or so when crunching.
With the special sauce it still produces around 42K Rac without much trouble and in terms of bang/buck when looking on the second hand markets you actually can build quite a cruncher if you get held of a Mining board with lots of PCIe's and those cheap 970s.
Take a descent cpu, 8GB ram (Perhaps) , SSD , and two good PSUs, couple that with 10 pcs of old 970s and you have a box that pulls around 1,8Kw producing 400K Rac+ for not that much money (compared what it can cost when purchasing new stuff)
So all in all i agree that the 970s are still quite good power/performance ratio but is certainly not comparable to a 2080Ti that pulls 185W and produces roughly 150K/day but for the cost of one 2080Ti you could buy almost 15 GTX 970s in my country nonetheless.
Thumbs up for 970 & special sauce.


. . Well according to nvidia-smi my 970s draw less than 150W each at full crunch but I have power meters on each rig and they tell me that the i5-6600 box with them in it draws about 375w from the wall in full flight. With almost solely blc33 tasks as at present the RAC is down but is still holding close to the 40K/card level. So about 82K/24/375 gives a RAC/watt of 9.1 for the box. The 1050ti in the Core 2 Duo draws less than 120w from the wall and has a current RAC close to 26K so 26K/24/120 gives a rating of 9.0 putting them very close to same level of power efficiency, in fact I would call them equal.

. . I hope to eventually get Linux sorted out on the newest rig, a Ryzen 7-1700 with 2 x 1060s, and see how that comes in. I am hoping it will achieve a slightly higher power efficiency than both of them.

Stephen

:)
ID: 1993595 · Report as offensive     Reply Quote
Gene Project Donor

Send message
Joined: 26 Apr 99
Posts: 126
Credit: 37,469,961
RAC: 42,300
United States
Message 1993610 - Posted: 13 May 2019, 3:38:20 UTC

@Stephen
I hope to eventually get Linux sorted out on the newest rig, a Ryzen 7-1700 with 2 x 1060s,


With one 1060 (SC) I'm getting about 48K RAC, although maybe 10% to 15% of that is from 6 CPU cores doing their thing. Configured for 6 cores doing CPU Seti, 1 core to drive the 1060 GPU, and 1 core for Einstein@home. nvidia-smi, at the moment, says 94 watts doing the cuda-91 app. The normal "wall" power draw runs 170 watts (including a trickle charge to a UPS).
Just to complete the comparison: 48K/24/170 -> 11.7 RAC/watt for the box I'm running. Note that I am intentionally avoiding the hyper-threading capabilities. Mostly for CPU temperature considerations.
ID: 1993610 · Report as offensive     Reply Quote
Lionel

Send message
Joined: 25 Mar 00
Posts: 680
Credit: 502,764,802
RAC: 319,387
Australia
Message 1994306 - Posted: 19 May 2019, 7:32:13 UTC

Decided to take the plunge and moved 2 boxes to Linux.
ID: 1994306 · Report as offensive     Reply Quote
Previous · 1 . . . 97 · 98 · 99 · 100 · 101 · 102 · 103 . . . 131 · Next

Message boards : Number crunching : Setting up Linux to crunch CUDA90 and above for Windows users


 
©2019 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.