A Multi-GPU Cruncher for the Less Affluent Among Us

Message boards : Number crunching : A Multi-GPU Cruncher for the Less Affluent Among Us
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Jeff Buck Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 11 Feb 00
Posts: 1441
Credit: 148,764,870
RAC: 0
United States
Message 1812550 - Posted: 25 Aug 2016, 17:42:42 UTC

Being rather frugal and with apparently somewhat shallower pockets than some crunchers, I read with some bemusement the threads detailing the building of crunch-monsters with the latest and greatest top-of-the-line processors, motherboards, GTX 10xx GPUs, and cooling solutions that rival the refrigerator in my kitchen. All with apparently little regard to the dollars/euros/pounds/yen/rupees that go out the door.

My approach tends to be a bit different, recycling/reusing/repurposing pre-owned components, primarily from sources on eBay and elsewhere online. Just about everything I currently crunch with, except for my daily driver, has come to life via that route.

A recent plan to start upgrading my GTX 660s to GTX 960s, now that the folks on the bleeding edge were starting to dump their barely used 9xx cards in favor of the new 10xx series, instead evolved into the assembly of an entirely new multi-GPU cruncher, at a cost that, at least to me, was relatively affordable.

The new machine is host 8064262. It's an HP xw9400 Workstation that arrived at my door with the following specifications:

Dual Quad Core AMD Opteron 2380 2.5GHz Processors
HP OEM Liquid Cooler
16GB ECC RAM
Two 300GB SAS Hard Drives
1050-Watt PSU
NVIDIA Quadro FX3700 Graphics Card

After pulling half the RAM, one of the HDDs and the Quadro FX3700, I gradually added one GPU at a time to get to the cruncher that's now been happily humming along for a couple of weeks.

At what cost, you may ask. Well, even if you didn't, that's the whole point of this post. Here's the tale of the [cash register] tape (all prices are in US$ and include any S/H charges):

HP xw9400 Workstation =============== $170.00
Windows 7 Pro 64-bit ================= 34.94
HP (OEM) GTX 960 Reference GPU =========== 119.00
Gigabyte GTX 960 GPU ================= 122.50
EVGA GTX 960 SSC GPU ================= 122.50
Gigabyte GTX 960 GPU =================== 120.00
PCIe x16 12-inch Riser Cable =============== 10.32
Two PCIe 12-inch 6-pin Extension Cables ======= 13.98
PCIe x16 7-inch Riser Cable (from my parts bin) === N/C
KVM Cable (from my parts bin) =============== N/C
-------------------------------------------------------------
TOTAL Cost ============================= $713.24

That cost could also be slightly offset by the extra items (8GB ECC Memory, 1 300GB SAS HDD, Quadro FX3700) pulled from the machine. In fact the FX3700 has already been sold for a net return of $10.52.

So, with only a couple weeks of occasionally intermittent crunching, that box has already reached a rack of 20K and I'm hoping to see it reach somewhere close to 35K by the time it stabilizes (not running 24/7 but rather about 143 hours per week). Does that compare with any of the "latest and greatest" boxes built by the deep pockets guys? Probably not, but I still have money to spend on other things, like food.

So, for those of you with a few cobwebs in your wallet, you can see that there are actually some fairly heavy-duty crunching possibilities out there that don't require a second mortgage! ;^)
ID: 1812550 · Report as offensive
Al Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Avatar

Send message
Joined: 3 Apr 99
Posts: 1682
Credit: 477,343,364
RAC: 482
United States
Message 1812557 - Posted: 25 Aug 2016, 18:15:21 UTC - in response to Message 1812550.  

Jeff, I guess I'm one of those with the crunching monsters you're referring to. I actually have been swinging both ways in this regard, yes, I have put together a couple fairly over the top machines recently, but I also have on tap a couple that would definitely fall into the recycling/reusing/repurposing pre-owned components side of things as well. For example, I am re-purposing a couple of HP Z800 motherboards, which weren't terribly expensive, under $125, which is less than many new motherboards out there. Also using 4-5 generation old Xeon procs in them, which again is less expensive than many new I series procs available today. Not to mention that I am also most recently buying $80-90 (after rebate) 750Ti cards, as the 10*0 series are still way out of line for the purchasing more than one or 2, at least for me until they have a significant price drop, or I win the lottery. Neither of which I am counting on any time soon.

I guess the way I look at it, is this is one of my few hobbies, that I enjoy spending my time and $ on. I don't go to bars and drink (and then literally p*ss) away my money, I don't do lavish vacations for weeks at at time every year, I've been pretty good at saving for quite a while, and have had decent jobs over the years. I guess to me it's more along the lines of spending your money on whatever brings you enjoyment today, as long as you're being responsible of course. And, preferably if it's legal. ;-)

So, you'll probably see a few more of the older systems coming on line in the not too distant future in my computer listing, because that is where I usually live, I just thought it would be fun to toss together a couple crazy machines and see how they go. It's all in good fun, as it should be for everyone. So, keep on crunchin', hopefully one of is will have that WoW moment we're all searching for, because in the end, that's why I'm here! :-)

ID: 1812557 · Report as offensive
Profile Jeff Buck Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 11 Feb 00
Posts: 1441
Credit: 148,764,870
RAC: 0
United States
Message 1812570 - Posted: 25 Aug 2016, 20:07:56 UTC - in response to Message 1812557.  

Yeah, I have nothing at all against those with the money to spend on the shiniest stuff. I get a bit envious sometimes, too. But since it seems like there's been so much emphasis on the high-end toys lately, especially since the 10xx series was launched, I thought it worthwhile to remind those who are itching to increase their contribution to the project that there can be more economical ways to do it, too.
ID: 1812570 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1812675 - Posted: 26 Aug 2016, 3:49:49 UTC - in response to Message 1812570.  
Last modified: 26 Aug 2016, 3:51:44 UTC

I thought it worthwhile to remind those who are itching to increase their contribution to the project that there can be more economical ways to do it, too.

Lots of GTX 750Tis available at very reasonable prices, and as Shaggie76's charts show they're no slouch when it comes to work done for power consumed. Still keeping pace with the latest & greatest & leading all the older cards.
Grant
Darwin NT
ID: 1812675 · Report as offensive
Profile Jeff Buck Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 11 Feb 00
Posts: 1441
Credit: 148,764,870
RAC: 0
United States
Message 1812685 - Posted: 26 Aug 2016, 4:53:13 UTC - in response to Message 1812675.  

I thought it worthwhile to remind those who are itching to increase their contribution to the project that there can be more economical ways to do it, too.

Lots of GTX 750Tis available at very reasonable prices, and as Shaggie76's charts show they're no slouch when it comes to work done for power consumed. Still keeping pace with the latest & greatest & leading all the older cards.

My other xw9400 (with an 800w PSU) has been running for a couple years now with 2 GTX 750Tis and 2 GTX 660s. The GTX 660s are more productive overall but do use more power.

I went ahead and stuck with the 960s on the new xw9400 because I figured the 1050w PSU gave me more headroom and I went for higher total output instead of output per watt. As it is, the new build actually seems to be drawing slightly less total power than that first xw9400, and my preliminary estimate is for at least 15% more crunching.

I also think that, although the newer Maxwell cards aren't as productive as they could be with the current apps, I've gotten the impression that once Jason and Petri dish out that Special app to the general public, that whole scenario will improve.
ID: 1812685 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1812732 - Posted: 26 Aug 2016, 7:15:44 UTC - in response to Message 1812685.  

I also think that, although the newer Maxwell cards aren't as productive as they could be with the current apps, I've gotten the impression that once Jason and Petri dish out that Special app to the general public, that whole scenario will improve.

Oh yes.
As it is, using SoG with the example command line gives a huge boost to their output.
Grant
Darwin NT
ID: 1812732 · Report as offensive
Profile tazzduke
Volunteer tester

Send message
Joined: 15 Sep 07
Posts: 190
Credit: 28,269,068
RAC: 5
Australia
Message 1812757 - Posted: 26 Aug 2016, 12:53:46 UTC - in response to Message 1812732.  

Greetings Jeff,

That is a nice economical build, on your side of the world, but here in Australia, those 960's would be pulling AUD$ 200 plus.

But I am in the way of reuse/recylce, my 3 crunchers are old LGA775 P45 Boards running a GTX 580, GTX 760 and GTX 770. Yep 3 really power munchers.

They are not mega crunchers, and mostly part time GPU crunch (WOW Event exception - max throttle)

I have over the last 12 months rotated a few GPU's through the machines, and still have 3 sitting in the computer crate (in shed) for when these ones start to show signs of wear and tear lol.

Happy Crunching
Regards
ID: 1812757 · Report as offensive
Profile Jeff Buck Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 11 Feb 00
Posts: 1441
Credit: 148,764,870
RAC: 0
United States
Message 1813073 - Posted: 27 Aug 2016, 23:03:51 UTC

A PM pointed out to me that I omitted the ongoing electricity usage for this rig. Kill-A-Watt is showing me a usage fluctuating from about 510 to 550 watts. That's with all 4 GPUs and 8 CPUs averaging about a 95% load. I don't know how that wattage compares to some of the big rigs with the latest and greatest hardware, but it's the upfront cost that I was focusing on, inasmuch as some people here seem to have very low electric rates, while others trend toward the high end.
ID: 1813073 · Report as offensive
Al Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Avatar

Send message
Joined: 3 Apr 99
Posts: 1682
Credit: 477,343,364
RAC: 482
United States
Message 1813090 - Posted: 28 Aug 2016, 0:07:29 UTC - in response to Message 1813073.  

Jeff, as a (sort of) comparison, my 6 GPU dual Xeon rig, with 5 750Ti's and a 980Ti thrown in for good measure, when I just now checked it, was pulling between 680-800 watts (it fluctuates constantly, though looked like the average while watching it for a minute or 2 was in the 725-755 range) out of the wall. I'm sure if I pulled the 980 out of the mix it'd probably drop by a good 100 watts or so. It will interesting to see the watts/work produced ratio once it's had a chance to stabilize a bit.

ID: 1813090 · Report as offensive
Profile petri33
Volunteer tester

Send message
Joined: 6 Jun 02
Posts: 1668
Credit: 623,086,772
RAC: 156
Finland
Message 1813263 - Posted: 28 Aug 2016, 21:32:58 UTC

My i7 draws 140W, mobo 50W, 4 pcs of 1080's draw a total of 4x140W (overestimated avg). Sum is 750W. The room is much cooler now than with the 780's or 980's.
To overcome Heisenbergs:
"You can't always get what you want / but if you try sometimes you just might find / you get what you need." -- Rolling Stones
ID: 1813263 · Report as offensive
Al Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Avatar

Send message
Joined: 3 Apr 99
Posts: 1682
Credit: 477,343,364
RAC: 482
United States
Message 1813471 - Posted: 29 Aug 2016, 16:25:46 UTC - in response to Message 1813263.  

Lol, Petri, that is pretty much a perfect world scenario, in terms of power and especially output the 1080 seems to be a fairly primo. Now if we could all win the Powerball, well, load em up, boys! lol

ID: 1813471 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1813494 - Posted: 29 Aug 2016, 17:11:05 UTC - in response to Message 1813471.  

@Al, go for the hybrid 1080s if you win, that 1080 reference and/or gamers seem underpowered, at least that is my opinion...
ID: 1813494 · Report as offensive
Al Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Avatar

Send message
Joined: 3 Apr 99
Posts: 1682
Credit: 477,343,364
RAC: 482
United States
Message 1813502 - Posted: 29 Aug 2016, 17:41:38 UTC - in response to Message 1813494.  

Naa, I think I'll bide my time and wait till Dec or so when the Ti versions supposedly might be coming out, and preferably Hydro Cooled versions! Hey, the things you can do when money is no object, eh? ;-)

ID: 1813502 · Report as offensive

Message boards : Number crunching : A Multi-GPU Cruncher for the Less Affluent Among Us


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.