Multi-GPU hardware setups: My method (feel free to share your own!)

Message boards : Number crunching : Multi-GPU hardware setups: My method (feel free to share your own!)
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 . . . 8 · Next

AuthorMessage
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1982226 - Posted: 25 Feb 2019, 23:20:43 UTC

I blame Rick. :/> LOL. If he hadn't dropped that defective workstation motherboard into my lap and which I tackled as a challenge to both get it running and learn something about Intel cpus, I probably would have gone the conventional path of just adding gpus to an already built machine.
And kept the system costs down a bit more. C'est la vie.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1982226 · Report as offensive
Profile bloodrain
Volunteer tester
Avatar

Send message
Joined: 8 Dec 08
Posts: 231
Credit: 28,112,547
RAC: 1
Antarctica
Message 1982272 - Posted: 26 Feb 2019, 8:10:58 UTC - in response to Message 1982226.  

lol. once you build you firs pc.... you want to keep building more. be dam with cost!
ID: 1982272 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1982736 - Posted: 1 Mar 2019, 3:09:45 UTC

i've been working on a new build. shooting for a system to hit 1,000,000 RAC :D it's on its way and should hit it if the current credit payout stays the same. currently sitting at 870k and rising.

Supermicro X9DRX+-F
2x Xeon E5-2680v2
32 GB DDR3 Reg ECC
3x RTX 2070
6x GTX 1080ti
EVGA power boost to add some more power to the PCIe slots
1x Corsair 1000w PSU powering the motherboard (and slot power for GPUs)
2x HP 1200w server PSUs for the GPU external power connectors
all put together with and custom mounted to a cheap mining rack.

8 of the GPUs are connected to the motherboard with PCIe3 shielded riser ribbons to get all 8 PCIe lanes from each card (8 lanes per card is a motherboard limit since all the slots are 8x only). 1 1080ti still on a riser due to a clearance issue with the rear CPU heatsink. i need a right angle adapter at the GPU.

nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current,gpu_name --format=csv
pcie.link.gen.current, pcie.link.width.current, name
3, 8, GeForce RTX 2070
3, 8, GeForce RTX 2070
3, 8, GeForce RTX 2070
3, 8, GeForce GTX 1080 Ti
3, 8, GeForce GTX 1080 Ti
3, 8, GeForce GTX 1080 Ti
3, 1, GeForce GTX 1080 Ti
3, 8, GeForce GTX 1080 Ti
3, 8, GeForce GTX 1080 Ti


pics:








Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1982736 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1982741 - Posted: 1 Mar 2019, 4:07:06 UTC

Can't wait to see this monster in action.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1982741 · Report as offensive
Profile ML1
Volunteer moderator
Volunteer tester

Send message
Joined: 25 Nov 01
Posts: 21233
Credit: 7,508,002
RAC: 20
United Kingdom
Message 1982817 - Posted: 1 Mar 2019, 12:53:04 UTC
Last modified: 1 Mar 2019, 12:53:43 UTC

Impressive! Will be watching the updates!!


Just got to mention... With all that spectacular GPU video output power there is yet just the one (very old) VGA cable connected up...!

;-)


Mmmm... Wild idea... Is a GPU 'screensaver' possible whereby the GPU memory usage/patterns are displayed semi-raw for interesting displays of utilization and activity?...

Happy fast crunchin'!
Martin
See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)
ID: 1982817 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22535
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1982826 - Posted: 1 Mar 2019, 13:35:50 UTC

Well if it still works don't mend it.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1982826 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1982834 - Posted: 1 Mar 2019, 13:49:51 UTC - in response to Message 1982817.  

It’s because the system is hooked to a 8-port KVM that only does VGA
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1982834 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5126
Credit: 276,046,078
RAC: 462
Message 1982860 - Posted: 1 Mar 2019, 15:46:25 UTC - in response to Message 1982826.  

Well if it still works don't mend it.


Something I need to "work at" :)

Tom
A proud member of the OFA (Old Farts Association).
ID: 1982860 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5126
Credit: 276,046,078
RAC: 462
Message 1982861 - Posted: 1 Mar 2019, 15:47:57 UTC - in response to Message 1982741.  

Can't wait to see this monster in action.


I think it is this one: https://setiathome.berkeley.edu/show_host_detail.php?hostid=8589036

Tom
A proud member of the OFA (Old Farts Association).
ID: 1982861 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1982863 - Posted: 1 Mar 2019, 15:50:23 UTC - in response to Message 1982861.  

yes, it's already in action.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1982863 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1982909 - Posted: 1 Mar 2019, 17:17:57 UTC - in response to Message 1982863.  

Hadn't looked at that host in a while. Last configuration had 1060's I think. Count is up to nine now. Think it will go higher before resources peter out?
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1982909 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1982915 - Posted: 1 Mar 2019, 17:30:25 UTC - in response to Message 1982909.  

That host is the same Host ID as the server case system in the first post.
initially it had [6] 1080tis + [1] 1060
then I swapped out the 1060 for a 1080.

I preserved that system (just copied the whole BOINC folder) and moved it over to the big system when i built it. wanted to keep the RAC and host credit count so i wouldn't have to start from scratch on the new system. and i moved over the 6 1080tis to the new system so I justified it that way lol.

the server case setup featured in the first post is still in use here: https://setiathome.berkeley.edu/show_host_detail.php?hostid=8608164
it has [1] 1080 + [3] 1060
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1982915 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5126
Credit: 276,046,078
RAC: 462
Message 1984934 - Posted: 13 Mar 2019, 14:24:18 UTC - in response to Message 1982736.  


Supermicro X9DRX+-F


Just found a listing for the above here: https://www.aztekcomputers.com/X9DRX+-F-O-SUPERMICRO-3835847.html

For $162.33! It said it was "Special Order" but I figured, at that price how can I lose? So I ordered one.

If enough of us order, maybe the dealer can get a bulk discount?

Tom
A proud member of the OFA (Old Farts Association).
ID: 1984934 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1984935 - Posted: 13 Mar 2019, 14:44:06 UTC - in response to Message 1984934.  

Have you ever ordered from them before? I hope it’s not a scam.

But if you end up getting it, that’s a great price, even if the board is refurbished. (Assume no warranty)
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1984935 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5126
Credit: 276,046,078
RAC: 462
Message 1984938 - Posted: 13 Mar 2019, 14:52:03 UTC - in response to Message 1984935.  

Have you ever ordered from them before? I hope it’s not a scam.

But if you end up getting it, that’s a great price, even if the board is refurbished. (Assume no warranty)


No I haven't dealt with them before. But the "original price" was "normal" and they have a large storefront online. It does appear to be refurbished or otherwise, used. I think either that was an "automagic" discount that no human was in the loop on. Or the "special order" gives them the wiggle room to refund the money. Or they actually have a supplier at that price. I hope it is not a scam. But the only place I have run across the "low price, too good to be true" so far is on eBay. And those started out strictly overseas and then started showing up with US addresses. So I can hope.

In any case, I will post the results. I may end up with 3 different LGA 2011 motherboards. But if I can get this one, I will deal with the "open air, breadboard" approach that I would need. :)

And keep at least 1 non-installed MB for a backup.

Tom
A proud member of the OFA (Old Farts Association).
ID: 1984938 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1984944 - Posted: 13 Mar 2019, 15:13:47 UTC - in response to Message 1984938.  

my only advice is that if you don't have it in your hands in 30 days, and they are giving you the runaround about it, cancel the order or issue a chargeback from from your CC company. not worth the hassle and you don't want it to stretch too long to where you cant take action to get your money back.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1984944 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22535
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1984950 - Posted: 13 Mar 2019, 15:53:20 UTC - in response to Message 1984934.  

I would avoid those boards on two grounds:
End-of-line = support will decrease just as you need it.
Refurbished = already broken once, back for repair, and now going out for testing again.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1984950 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1984959 - Posted: 13 Mar 2019, 16:45:48 UTC - in response to Message 1984950.  

valid points. though I haven't had really any problems with SM products, and I've bought a bunch of their things used. (admittedly, none classified as refurbished). refurbished can have many meanings. someone could have simply blown the dust off of it and called it refurbished. you never know the extent.

My particular board was new, and even though it's old, still carries the SM 3 year warranty from date of purchase.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1984959 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5126
Credit: 276,046,078
RAC: 462
Message 1984979 - Posted: 13 Mar 2019, 20:23:53 UTC - in response to Message 1984944.  

my only advice is that if you don't have it in your hands in 30 days, and they are giving you the runaround about it, cancel the order or issue a chargeback from from your CC company. not worth the hassle and you don't want it to stretch too long to where you cant take action to get your money back.


Very good points. I know I have submitted my CC info. I will look and see if they have actually charged it. Often a company won't charge till they ship.

As of this moment, they haven't charged my CC. They have also only sent automated acknowledgment(s).

Tom
A proud member of the OFA (Old Farts Association).
ID: 1984979 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5126
Credit: 276,046,078
RAC: 462
Message 1984980 - Posted: 13 Mar 2019, 20:27:32 UTC - in response to Message 1984950.  

I would avoid those boards on two grounds:
End-of-line = support will decrease just as you need it.
Refurbished = already broken once, back for repair, and now going out for testing again.


I have had variable success with used motherboards. Mostly (over 75% of the time) there has not been an issue. For this low a price (compared to $500-$700) I am willing to take the risk.

I am not sure if I have ever had a used bad server MB. But even new server MB's will sometimes burn out "early" after running 9 months of 24/7 Seti. (BTDT).

Tom
A proud member of the OFA (Old Farts Association).
ID: 1984980 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 . . . 8 · Next

Message boards : Number crunching : Multi-GPU hardware setups: My method (feel free to share your own!)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.