Multi-GPU hardware setups: My method (feel free to share your own!)

Message boards : Number crunching : Multi-GPU hardware setups: My method (feel free to share your own!)
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 . . . 8 · Next

AuthorMessage
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5126
Credit: 276,046,078
RAC: 462
Message 1982079 - Posted: 25 Feb 2019, 4:22:43 UTC - in response to Message 1982078.  

Ok, finally adding a picture of my 9 gpu hardware setup. I am perfectly aware that this is not a pretty rig.



Here is a link https://drive.google.com/open?id=1PWJ5gCp2vWIz9kZ-w4FAWs835AI3wRZSRQ

Apparently getting it to display requires something I am not figuring out yet.

Tom




fixed it for you.

it's because of Google.

to find the direct img URL:

http://drive.google.com/uc?export=view&id=PASTE YOU ID HERE

you'll be redirected to the direct image. copy and paste it from the address bar


At least on the Foxfire I am using under Linux, the image didn't display. But if I can get a direct link with the file name ending in a graphics file it will display? Hmmmmmm......

Let me look around.

Tom





Ok, now it is showing up in this browser, but will it show up in someone else's browser. The resulting string looks a lot like the one Ian&SteveC created.

Tom
A proud member of the OFA (Old Farts Association).
ID: 1982079 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1982081 - Posted: 25 Feb 2019, 4:30:51 UTC - in response to Message 1982078.  
Last modified: 25 Feb 2019, 4:34:59 UTC

Let’s see. Can you see this one?



Change “open” to “uc”

https://drive.google.com/uc?id=1PWJ5gCp2vWIz9kZ-w4FAWs835AI3wRZSRQ
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1982081 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1982083 - Posted: 25 Feb 2019, 4:32:37 UTC - in response to Message 1982079.  

Ok, finally adding a picture of my 9 gpu hardware setup. I am perfectly aware that this is not a pretty rig.



Here is a link https://drive.google.com/open?id=1PWJ5gCp2vWIz9kZ-w4FAWs835AI3wRZSRQ

Apparently getting it to display requires something I am not figuring out yet.

Tom




fixed it for you.

it's because of Google.

to find the direct img URL:

http://drive.google.com/uc?export=view&id=PASTE YOU ID HERE

you'll be redirected to the direct image. copy and paste it from the address bar


At least on the Foxfire I am using under Linux, the image didn't display. But if I can get a direct link with the file name ending in a graphics file it will display? Hmmmmmm......

Let me look around.

Tom





Ok, now it is showing up in this browser, but will it show up in someone else's browser. The resulting string looks a lot like the one Ian&SteveC created.

Tom


I think that method is generating a one time image link (nonce in the url) which is why we can only see the images we post but no one else.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1982083 · Report as offensive
Profile Bill G Special Project $75 donor
Avatar

Send message
Joined: 1 Jun 01
Posts: 1282
Credit: 187,688,550
RAC: 182
United States
Message 1982087 - Posted: 25 Feb 2019, 4:55:00 UTC - in response to Message 1982083.  

and this message is not wrapping properly on my machine.

SETI@home classic workunits 4,019
SETI@home classic CPU time 34,348 hours
ID: 1982087 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1982093 - Posted: 25 Feb 2019, 5:50:19 UTC - in response to Message 1982081.  
Last modified: 25 Feb 2019, 5:56:43 UTC

Let’s see. Can you see this one?

Change “open” to “uc”

This is the only post [Message 1982081] in this thread that displays an image in Chromium browser.

Appears that changing “open” to “uc” is the trick to making the image display with the Google Drive shareable link.

Just tried a post with that trick and it displays.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1982093 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 36853
Credit: 261,360,520
RAC: 489
Australia
Message 1982097 - Posted: 25 Feb 2019, 6:10:28 UTC

...This is the only post [Message 1982081] in this thread that displays an image in Chromium browser.

Appears that changing “open” to “uc” is the trick to making the image display with the Google Drive shareable link.

Just tried a post with that trick and it displays.
You probably did what I did Keith and opened that pic up through the url at some stage and it now shows, but opening the same page on my other PC (not logged in) just display busted links. ;-)

But I will repeat what I said before, "Actually when using the image tags your link must end with a .jpg/.bmp/.png/etc for the image to be displayed."

Cheers.
ID: 1982097 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1982104 - Posted: 25 Feb 2019, 7:41:24 UTC - in response to Message 1982097.  

OK, thanks Wiggo for the clarification. I think you are correct. Google Drive does not give any explicit naming for images that have a normal image extension. I think that is the difference. When I hosted images on Flickr, the images had the normal image extensions and when I grabbed a link, the link still had the explicit naming convention of the image. Those worked. I will have to start reusing Flickr for hosted images again I think.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1982104 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1982105 - Posted: 25 Feb 2019, 7:49:42 UTC - in response to Message 1982096.  

Can I suggest and ask please that contributors to this thread post their total system cost and monthly power usage and cost.

Total System cost for 5 crunching computers = $19715

Monthly power usage = 3000 kWh

Monthly cost = $600
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1982105 · Report as offensive
Sirius B Project Donor
Volunteer tester
Avatar

Send message
Joined: 26 Dec 00
Posts: 24913
Credit: 3,081,182
RAC: 7
Ireland
Message 1982115 - Posted: 25 Feb 2019, 10:21:34 UTC - in response to Message 1982113.  

£460 per month equates to £5,520 per annum...£15,000?
ID: 1982115 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5126
Credit: 276,046,078
RAC: 462
Message 1982142 - Posted: 25 Feb 2019, 14:57:19 UTC - in response to Message 1982081.  

Let’s see. Can you see this one?

---snip---

Change “open” to “uc”

-snip---


Yes :)

Tom
A proud member of the OFA (Old Farts Association).
ID: 1982142 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5126
Credit: 276,046,078
RAC: 462
Message 1982146 - Posted: 25 Feb 2019, 15:17:22 UTC - in response to Message 1982096.  

Can I suggest and ask please that contributors to this thread post their total system cost and monthly power usage and cost.


Ah, I would have to do some calculations on the power bill question. I am on a "level pay" plan that allows me to setup my banks bill paying service and forget it. Yes, it has gone down.

I can give you a fairly good ballpark on the costs for my biggest box.

Please understand that the cpus I am running are significantly more expensive than a pair of Intel e5-2670 v1's. The 2670's started hitting the used market when the data centers went through an upgrade cycle. They are now under $100USD per cpu on eBay.

I paid about $450USD for the pair of cpus I have now (e5-2690 v2).
The motherboard is an ASROCK that I paid under $300USD for.

I got my 3 gtx 1070Ti's used for about $300 a piece on eBay.

The gtx 1060 3GB's cost anyway from $250 down to $130 or less on eBay (used).

The case is about $100 new.
The PSU was used at $160 but new on sale it is ~ $300

The riser card kits and 1 to 4 expander boards totaled about $100~

Almost forgot the ram. Since this is DDR3 it is a bunch less expensive than DDR4 (as well as slower).

And since I bought a lot of this system one piece at a time the upgrades were much less noticeable.

Looks like I could re-create this system for about $2500+ or less. If I went with the cheaper e5-2670 v1's and nothing but gtx 1060 3GB's I should be able to do it from around $1,500+.

It would cause me a pause if I forked out this cash at once. But since it was a "one piece at a time" type of project followed by "re-allocating" the hardware it was much less visibly painful.

Tom
A proud member of the OFA (Old Farts Association).
ID: 1982146 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5126
Credit: 276,046,078
RAC: 462
Message 1982149 - Posted: 25 Feb 2019, 15:25:39 UTC - in response to Message 1982113.  
Last modified: 25 Feb 2019, 15:26:47 UTC

In English money that is £15,000 and £460 a month.

What can one say??? You have a very impressive RAC at 1 1/2 million plus, but my god it has cost you a pretty penny to be where you are today, obviously you can afford it.

I just hope you think it is worth it when the project has no effective back end to process the data that you produce. 2+ years and still waiting for Nebula :-(

But nevertheless your set-up is impressive.


When you are running AMD motherboards with DDR4 ram plus high end gpus (when new, especially) it is a bunch more expensive to buy.

I have been consolidating all my gpus onto basically one box. So my major expense is in fact the gpus on one box.

I think most of us have been upgrading one system at a time. Because of that, the cost of the "installment loan payment" makes the cost(s) of the system(s) less noticeable to our cash flow.

I know that Ian&SteveC's last upgrade was mainly a new gpu-server motherboard (very expensive and rare) plus a miners rig platform with an extra sheet of Lucite. Pretty much everything else was parts from an existing system(s).

Tom
A proud member of the OFA (Old Farts Association).
ID: 1982149 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1982157 - Posted: 25 Feb 2019, 16:35:17 UTC
Last modified: 25 Feb 2019, 16:42:34 UTC

besides the fact that the price of a system is irrelevant to this thread, it would only serve the purpose for some people to gloat about how they spent less. rather boring. RAC more or less scales to money invested. but a reasonable (mid-high end consumer GPUs, not enterprise tier or the majorly overpriced Titans) GPU focused system is both more cost and power efficient than a RAC-equivalent CPU based setup. think antony's 500+ systems are using less electricity or cost less than a GPU setup? doubt it.

I don't have time to add up every little expense, and I don't really track my electric use, simply because this is a hobby and I can afford it. that's all that matters. no one in the top list of contributors is being financially reckless.

but rough price list for the main system, which is inching towards being a 1,000,000 RAC system (one host, i'll post pics here later). many parts bought used and at various times. for stuff i cant remember price, i'll just put what they are currently worth in parentheses ()

mining rack/case/frame - $36
Acrylic sheet to custom mount MB - $18
Supermicro X9DRX+-F motherboard - $575
2x Xeon E5-2680v2 CPUs - ($300)
32GB (8x4GB) DDR3 Registered ECC ram - ($65)
OS drive - 2x 250GB (RAID1, you can easily use just one drive) samsung SSDs I had laying around - ($100)
1000w Corsair PSU - $100
2x 1200w HP server PSU - $110
2x breakout boards with cables - $40
3x RTX 2070 - $1500
6x GTX 1080ti - $3000
1x GTX 1060 3GB - ($120)

Total = $5864, or €5167 , or £4487

power use is estimated at ~1750w average
$0.11/kWh in my area = $138.60 (€122.14/£106.10) per month

prices can vary depending on the deals you find and the conditions of your market and how much your electric costs.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1982157 · Report as offensive
Sirius B Project Donor
Volunteer tester
Avatar

Send message
Joined: 26 Dec 00
Posts: 24913
Credit: 3,081,182
RAC: 7
Ireland
Message 1982160 - Posted: 25 Feb 2019, 16:41:59 UTC - in response to Message 1982157.  

besides the fact that the price of a system is irrelevant to this thread, it would only serve the purpose for some people to gloat about how they spent less. rather boring.
RAC more or less scales to money invested.
Best comment seen on these boards in a long time.
ID: 1982160 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1982163 - Posted: 25 Feb 2019, 16:43:28 UTC - in response to Message 1982157.  

Hi Ian

Just curiosity, you need 2 Xeon's to power all the GPU's or you could run with only one? Considering no CPU task are running on the hosts.
ID: 1982163 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5126
Credit: 276,046,078
RAC: 462
Message 1982164 - Posted: 25 Feb 2019, 16:46:06 UTC - in response to Message 1982157.  

besides the fact that the price of a system is irrelevant to this thread, it would only serve the purpose for some people to gloat about how they spent less. rather boring. RAC more or less scales to money invested. but a reasonable (mid-high end consumer GPUs, not enterprise tier or the majorly overpriced Titans) GPU focused system is both more cost and power efficient than a RAC-equivalent CPU based setup. think antony's 500+ systems are using less electricity or cost less than a GPU setup? doubt it.


I didn't think about the "gloating" aspect. So are you "gloaters" out there, you have been WARNED!!!! :/ No gloating allowed on this thread.

Given how fast anything from a gtx 750Ti running Windows on up, processes compared to cpu-only processing I would be hard put to make a case for cpu only processing unless you were playing with "free" cpu/memory/motherboard hardware and solar-powered daytime only crunching. (aka: eng4hire).

Tom
A proud member of the OFA (Old Farts Association).
ID: 1982164 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1982171 - Posted: 25 Feb 2019, 17:08:24 UTC - in response to Message 1982163.  

Hi Ian

Just curiosity, you need 2 Xeon's to power all the GPU's or you could run with only one? Considering no CPU task are running on the hosts.


I'm am running CPU tasks on this host. but only 26 threads. 80% total CPU use between the CPU and GPU jobs

but 2 CPUs are needed on this board to use all the PCIe slots/lanes. about half of the slots wouldn't work if you didnt have one of the CPUs installed. i could use 2x 4-core/8-thread Xeons and be fine with 16 total threads, but i already had these chips so i used them.

i went this route because with -nobs, you need one full CPU thread, and all of the mining boards only support up to a 4-core/8-thread chip. running 7 GPUs pretty much maxes it out when running -nobs.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1982171 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1982172 - Posted: 25 Feb 2019, 17:18:51 UTC

Yes I believe in the project, been here since 2001 continuously. I never intended to build this high a production farm, but I wanted to try my hand with Linux again so I built a computer to learn. Then a free workstation motherboard ended up in my lap from a donation from RueiKe. So that got built out. Then another free workstation motherboard ended up here again as a donation from RuieKe. That too got built out. Yes, I can afford the costs. This crunching farm just morphed into my latest hobby. I haven't sunk any recent monies into my other hobbies of high-end audio or astrophotography. Those had much higher purchase costs. So crunching is a "cheap" hobby compared to those I would say.

I happen to know the exact cost of my hosts because I maintain their construction bill on Rigbuilder on Overclockers.net website. So easy to keep track of the build cost. I participate in the AMD threads there.
https://www.overclock.net/forum/rigbuilder.php?do=rig&action=my-rigs

I know my power usage and cost since I have extensive power generation and consumption monitoring with my PV solar plant. The generation is not near enough to cover the consumption now. So the much higher monthly power bill than when I started.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1982172 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1982186 - Posted: 25 Feb 2019, 19:06:24 UTC

The point people should consider is how much can be Saved by running Multiple GPUs on One System. Cost of the GPUs are irrelevant since they are the same no matter which machine runs them.
Consider My 12 GPU system. All 12 GPUs are running on just One machine instead of 3 or 4 different machines. Big savings....
The One machine was relatively cheap;
CPU = $220
Power Supplies = $170
Ram = $120
Motherboard = $80
GPU Connectors = $100
Case = Ancient Compaq - Free
Total = ~$690

The CPU is running around 50%, using about 50 watts to run the 12 GPU.... Big Savings.
It's currently producing a RAC of around 387k from a machine with just One CPU running at half speed.
ID: 1982186 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5126
Credit: 276,046,078
RAC: 462
Message 1982205 - Posted: 25 Feb 2019, 22:04:33 UTC - in response to Message 1982186.  

The point people should consider is how much can be Saved by running Multiple GPUs on One System. Cost of the GPUs are irrelevant since they are the same no matter which machine runs them.


Right on there. More bang for the buck. That is why I have been playing with more gpus on a single motherboard and/or box. I would not have gone out and bought this all at once. Just incremented my way into it.

Tom
A proud member of the OFA (Old Farts Association).
ID: 1982205 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 · 6 . . . 8 · Next

Message boards : Number crunching : Multi-GPU hardware setups: My method (feel free to share your own!)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.