Multi-GPU hardware setups: My method (feel free to share your own!)

Message boards : Number crunching : Multi-GPU hardware setups: My method (feel free to share your own!)
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 · Next

AuthorMessage
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1985211 - Posted: 15 Mar 2019, 2:35:32 UTC - in response to Message 1982736.  

now with 10 GPUs :)


Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1985211 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1985213 - Posted: 15 Mar 2019, 2:40:11 UTC - in response to Message 1985211.  
Last modified: 15 Mar 2019, 2:42:30 UTC

now with 10 GPUs :)


Hurray!!!

+1

----edit---
Your using 1 slot to help power the pcie bus? So that is actually all the gpus you have room for unless you use a 1 to 4 expander ---edit---

What I want to see is using the 1 slot to 4 gpu expander cards, how many gpus can you hang off it before the bios screams "insufficient pcie resources"..... Who knows, maybe you can run 15 gpus.

It is supposed to be a gpu server, after all :)

Tom
A proud member of the OFA (Old Farts Association).
ID: 1985213 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1985218 - Posted: 15 Mar 2019, 3:26:52 UTC - in response to Message 1985213.  

i've been thinking about doing that just to test it. i have two 4x pcie expansion cards. if i had enough spare gpus it would be 16 i could try to fit. i'm sure it would work. this guy here got 18 GPUs working on this board. granted, older cards, but discrete pcie devices nonetheless. 18 GPUs in a single rig and it works

but i'd never leave it like that, GPUs just sprawled out everywhere. i like them to be secured down and kept tidy. so for actual long term operations, i wont go over 11 GPUs, it's all that'll comfortably fit on the rack.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1985218 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1985225 - Posted: 15 Mar 2019, 4:30:32 UTC - in response to Message 1985218.  


but i'd never leave it like that, GPUs just sprawled out everywhere. i like them to be secured down and kept tidy. so for actual long term operations, i wont go over 11 GPUs, it's all that'll comfortably fit on the rack.


You need a miner rack upgrade :)

I will admit, mine is NOT pretty or organized looking. I have a 7 slot Supermicro dual cpu MB shipped/not arrived. Plus the 11 slot MB that I haven't heard back about.

The ASROCK MB I am using now has 6 pcie slots and 1 non-pcie slot of some type. I am hoping the 7 slot MB will let me get past 9 gpus. But who knows? If I setup a work bench breadboard and see if I can get 10 gpus to be recognized. May have to borrow all my other gpus. Not sure if I have enough riser bases since some seemed to have stopped working as well as possibly 2 gpus now. I think I can get scourge up maybe 14 gpus if I strip out all the other machines. But I don't have enough riser bases for that.

Oh, well.

Tom

.
A proud member of the OFA (Old Farts Association).
ID: 1985225 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1985405 - Posted: 15 Mar 2019, 23:49:28 UTC - in response to Message 1984934.  
Last modified: 15 Mar 2019, 23:51:48 UTC


Supermicro X9DRX+-F


Just found a listing for the above here: https://www.aztekcomputers.com/X9DRX+-F-O-SUPERMICRO-3835847.html

For $162.33! It said it was "Special Order" but I figured, at that price how can I lose? So I ordered one.

If enough of us order, maybe the dealer can get a bulk discount?

Tom


The vendor just confirmed they can't get one anymore.

Tom
ps. Now that I have a suspect gpu offline I am trying 9 gpus again. Wish me luck.... and the ability to run more than 2 weeks without stopping :)
A proud member of the OFA (Old Farts Association).
ID: 1985405 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1985409 - Posted: 16 Mar 2019, 0:10:43 UTC - in response to Message 1985405.  


Supermicro X9DRX+-F


Just found a listing for the above here: https://www.aztekcomputers.com/X9DRX+-F-O-SUPERMICRO-3835847.html

For $162.33! It said it was "Special Order" but I figured, at that price how can I lose? So I ordered one.

If enough of us order, maybe the dealer can get a bulk discount?

Tom


The vendor just confirmed they can't get one anymore.

Tom
ps. Now that I have a suspect gpu offline I am trying 9 gpus again. Wish me luck.... and the ability to run more than 2 weeks without stopping :)


They rise the price to $487.92..... sure at this price they get one.
ID: 1985409 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1985410 - Posted: 16 Mar 2019, 0:14:05 UTC - in response to Message 1985409.  


Supermicro X9DRX+-F


Just found a listing for the above here: https://www.aztekcomputers.com/X9DRX+-F-O-SUPERMICRO-3835847.html

For $162.33! It said it was "Special Order" but I figured, at that price how can I lose? So I ordered one.

If enough of us order, maybe the dealer can get a bulk discount?

Tom


The vendor just confirmed they can't get one anymore.

Tom
ps. Now that I have a suspect gpu offline I am trying 9 gpus again. Wish me luck.... and the ability to run more than 2 weeks without stopping :)


They raised the price to $487.92..... sure at this price they get one.


It is still the cheapest one on the market. Most seem to start at $600-900USD.

Tom
A proud member of the OFA (Old Farts Association).
ID: 1985410 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1985912 - Posted: 19 Mar 2019, 0:47:29 UTC
Last modified: 19 Mar 2019, 1:04:51 UTC

In the top 40 on the Leaderboard there are about 9 systems that are running more than 4 gpus.

I believe that Ian&Steve and Tbar are both running at least two systems each with more than 4 gpu's. Me. And some other folk.

Most are running 4 gpus, many are very high end/expensive ones.

But I just did find an 8 gpu system mired in 74th place. Apparently, he/she is running Windows with the stock SOG's. On GPUs called NVIDIA GRID K520 (4095MB) driver: 353.90 OpenCL: 1.2 gpu. And another 6 gpu box mired in 96th place.

Tom
A proud member of the OFA (Old Farts Association).
ID: 1985912 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1985924 - Posted: 19 Mar 2019, 2:35:27 UTC - in response to Message 1985912.  

That system is actually only 4 physical cards. The K520 is a dual GPU card. So 4 of them equals 8 GPUs. It’s how BOINC handles dual GPU cards.

But I looked through the top 100 or so the other day. A couple people running more than 4 GPUs. I saw that wildcardcorp is back also with 10 GTX1080tis, But he’s running stock tasks it looks like. And looking at his stats, seems like he’s putting most of his power on other projects.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1985924 · Report as offensive
marmot
Avatar

Send message
Joined: 15 May 99
Posts: 144
Credit: 1,220,664
RAC: 0
United States
Message 1986079 - Posted: 20 Mar 2019, 7:00:51 UTC - in response to Message 1985912.  
Last modified: 20 Mar 2019, 7:10:05 UTC

Tom M:
driver: 353.90 OpenCL: 1.2 gpu.


Something I ran into last week. Attempting to run AMD and nVidia GPU's under Windows 10, with newest drivers, doesn't allow OpenCL on both cards. Catalyst 19.3.1 wipes out nVidia OpenCL and Forceware 419.x does the same to AMD cards. Many, many install (DDU cleaned or overwriting install) attempts. Its a testing machine for common baseline testing of WU's for any project I join.

I'll go back to earlier Forceware till I find a version that leaves the AMD OpenCL alone. 388.x probably; is there a good reason they stayed with 353.90 OpenCL?
ID: 1986079 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1986446 - Posted: 22 Mar 2019, 1:23:52 UTC

I have the
Supermicro Server Motherboard MBD-X9DRH-IF
motherboard in.

After going around and around with it I finally got it to recognize one and then more gpu cards in the Pcie slots. I have made it up to 5 gpus today. I will try some more tomorrow. The trick apparently was to "toggle" the pcie slots in the bios. This bios has an entry that is all about pcie's and some other stuff. That is where they hid the above 4G toggle for the large(r) gpu counts.

I am running a "lashup" rig with incorrect cpu coolers while I find out if this can exceed 7 to 10+ gpus. Once it passes 9 gpus, I will feel the urge to buy the correct cpu coolers. :) I just don't want to throw more money at rectangular coolers until I am fairly confident I am going to use this one as my new high end production MB.

Yes, I am only running 3 gtx 1070Ti's on my current big box. All the rest have relocated to close to the MBD-X9DRH-IF MB. :)

Tom
A proud member of the OFA (Old Farts Association).
ID: 1986446 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1986470 - Posted: 22 Mar 2019, 2:45:45 UTC - in response to Message 1985211.  

now with 10 GPUs :)



This system has now passed 1,000,000 RAC :))))
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1986470 · Report as offensive
Profile tazzduke
Volunteer tester

Send message
Joined: 15 Sep 07
Posts: 190
Credit: 28,269,068
RAC: 5
Australia
Message 1986482 - Posted: 22 Mar 2019, 3:39:12 UTC - in response to Message 1986470.  

Awesome effort
ID: 1986482 · Report as offensive
Filipe

Send message
Joined: 12 Aug 00
Posts: 218
Credit: 21,281,677
RAC: 20
Portugal
Message 1986531 - Posted: 22 Mar 2019, 10:41:49 UTC

Impressive!
ID: 1986531 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1986664 - Posted: 23 Mar 2019, 0:16:46 UTC - in response to Message 1986470.  

now with 10 GPUs :)

This system has now passed 1,000,000 RAC :))))


And you are just getting started. Right ;)

Tom
A proud member of the OFA (Old Farts Association).
ID: 1986664 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1986725 - Posted: 23 Mar 2019, 13:13:44 UTC - in response to Message 1986470.  
Last modified: 23 Mar 2019, 13:14:15 UTC

now with 10 GPUs :)
--edit----
This system has now passed 1,000,000 RAC :))))


I was playing with some of your gpu processing timings and RAC #'s they were being assigned.

It looks like the gpus alone can manage in the neighborhood of 1,280,000 RAC. And then add the production of 36 cpus(aka: 90% of 40) and you look like you could easily top out past 1,300,000+

Tom
A proud member of the OFA (Old Farts Association).
ID: 1986725 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1986736 - Posted: 23 Mar 2019, 13:42:30 UTC - in response to Message 1986470.  

now with 10 GPUs :)



This system has now passed 1,000,000 RAC :))))

I believe you are qualified to win the SETI Toaster. Nice work.
ID: 1986736 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1986737 - Posted: 23 Mar 2019, 13:43:38 UTC - in response to Message 1986470.  
Last modified: 23 Mar 2019, 13:44:42 UTC

now with 10 GPUs :)



This system has now passed 1,000,000 RAC :))))

I believe you are qualified to win the SETI Toaster. Nice work.

I imagine this baby pushed around 2 KW form the AC outlet when running at full throttle!
ID: 1986737 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1986739 - Posted: 23 Mar 2019, 13:51:13 UTC - in response to Message 1986736.  

[quote]now with 10 GPUs :)
--edit---
This system has now passed 1,000,000 RAC :))))

I believe you are qualified to win the SETI Toaster. Nice work.


Which kind? The GBT toaster or the "other one"? :)

Tom
A proud member of the OFA (Old Farts Association).
ID: 1986739 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1986753 - Posted: 23 Mar 2019, 15:08:09 UTC - in response to Message 1986739.  

I’ll take any toaster :)
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1986753 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 · Next

Message boards : Number crunching : Multi-GPU hardware setups: My method (feel free to share your own!)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.