How to stuff 4 x GTX295 in one box

Message boards : Number crunching : How to stuff 4 x GTX295 in one box
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
MarkJ Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 08
Posts: 1139
Credit: 80,854,192
RAC: 5
Australia
Message 934530 - Posted: 19 Sep 2009, 15:57:12 UTC

The guys over at GPUgrid posted this link. I thought i'd repeat it here...

Manifold Runs over 1,920 Processor Cores in New GIS Supercomputing Desktop Record
BOINC blog
ID: 934530 · Report as offensive
Profile Hydropower
Avatar

Send message
Joined: 16 May 99
Posts: 15
Credit: 923,836
RAC: 0
Netherlands
Message 934532 - Posted: 19 Sep 2009, 16:10:00 UTC - in response to Message 934530.  

You might also want to check my previous post there on the pitfalls, especially on powersupplies:

http://www.gpugrid.net/forum_thread.php?id=1091&nowrap=true#10119
ID: 934532 · Report as offensive
zpm
Volunteer tester
Avatar

Send message
Joined: 25 Apr 08
Posts: 284
Credit: 1,659,024
RAC: 0
United States
Message 934537 - Posted: 19 Sep 2009, 16:52:27 UTC - in response to Message 934532.  

um, crunch.
ID: 934537 · Report as offensive
MarkJ Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 08
Posts: 1139
Credit: 80,854,192
RAC: 5
Australia
Message 934542 - Posted: 19 Sep 2009, 17:02:01 UTC - in response to Message 934532.  

You might also want to check my previous post there on the pitfalls, especially on powersupplies:

http://www.gpugrid.net/forum_thread.php?id=1091&nowrap=true#10119


Yes I remember our discussions about them. Did you end up RMAing all 4 power supplies? They even used the same motherboard as you.

Interesting that the guys in this article only had a single 1250w power supply. I don't think it would last too long with the load that 4 x GTX295's would take (according to nvidia 4 x 289w = 1156w).
BOINC blog
ID: 934542 · Report as offensive
Profile Hydropower
Avatar

Send message
Joined: 16 May 99
Posts: 15
Credit: 923,836
RAC: 0
Netherlands
Message 934559 - Posted: 19 Sep 2009, 17:47:54 UTC - in response to Message 934542.  

Well, I RMA'ed one and they investigated it. It seems that the sound was okay and I am still using all four of them. I did RMA one Evga card which gave me trouble and received a replacement card. So far so good.

I cannot believe that one single 1250 W PS will feed them reliably. When crunching for MilkyWay my powermeter shows a fairly constant 1300 W real draw, and that project is not pushing the limits on the NVidia cards yet.
ID: 934559 · Report as offensive
MarkJ Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 17 Feb 08
Posts: 1139
Credit: 80,854,192
RAC: 5
Australia
Message 934749 - Posted: 20 Sep 2009, 5:37:02 UTC - in response to Message 934559.  

Well, I RMA'ed one and they investigated it. It seems that the sound was okay and I am still using all four of them. I did RMA one Evga card which gave me trouble and received a replacement card. So far so good.

I cannot believe that one single 1250 W PS will feed them reliably. When crunching for MilkyWay my powermeter shows a fairly constant 1300 W real draw, and that project is not pushing the limits on the NVidia cards yet.


Assuming a 90% efficency power supply they would have next to nothing left over to drive the motherboard and hard drives, so it clearly wouldn't be reliable.

I thought you were looking at trying to get more cards in one machine as your next project. ASUS P6T workstation perhaps? You could give them a run for their money, or better yet get them to pay you. You'll need to take lots of photos as you put it together.
BOINC blog
ID: 934749 · Report as offensive
Profile Hydropower
Avatar

Send message
Joined: 16 May 99
Posts: 15
Credit: 923,836
RAC: 0
Netherlands
Message 934849 - Posted: 20 Sep 2009, 21:21:14 UTC - in response to Message 934749.  

That is correct. I have been looking into the possibility of running seven double cards in one system. From a heat-perspective I have it figured out, also for the powersupplies and way-of-build.

I need to rethink fincances versus probability of success though. Drivers, bios, tools and applications all need to recognise 14 gpus. this would not look like your average PC anymore and have a substantial weight.

ID: 934849 · Report as offensive
zpm
Volunteer tester
Avatar

Send message
Joined: 25 Apr 08
Posts: 284
Credit: 1,659,024
RAC: 0
United States
Message 934850 - Posted: 20 Sep 2009, 21:46:59 UTC - in response to Message 934849.  
Last modified: 20 Sep 2009, 22:42:09 UTC

it would be nice if some manu. made a pci-e 16x 2.0 connector bridge.... sort of like a

| |
| |
| Wire |
|||||||||||||| pci-e extender...
| |
| |
| |
ID: 934850 · Report as offensive
Profile ML1
Volunteer moderator
Volunteer tester

Send message
Joined: 25 Nov 01
Posts: 20147
Credit: 7,508,002
RAC: 20
United Kingdom
Message 934874 - Posted: 21 Sep 2009, 0:49:49 UTC - in response to Message 934849.  

... seven double cards in one system...

What motherboard have you got that can take that many?

Happy hot crunchin'!

Regards,
Martin


See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)
ID: 934874 · Report as offensive
Profile 52 Aces
Avatar

Send message
Joined: 7 Jan 02
Posts: 497
Credit: 14,261,068
RAC: 67
United States
Message 934892 - Posted: 21 Sep 2009, 1:46:14 UTC - in response to Message 934874.  

When folks are running dual quad core CPUs with 1920 more cores on video, I gotta wonder what is definitionally a single box anymore :-) If it's OS instance, then just virtualize in the other direction.

Regardless, more power to you :-) Speaking of, are multiple (centrally controlled) power supplies allowed for a single box (to fuel all those nVidia cards)? If so, then the runway for more and more PCI-E slots can be extended quite a bit provided there was a market for bigger boards. And you know nVidia & ATI will continue to race one another for more beef per slot.

Who will be first to a single box RAC above 1 million ? Probably only 2 years away.
ID: 934892 · Report as offensive
Profile dnolan
Avatar

Send message
Joined: 30 Aug 01
Posts: 1228
Credit: 47,779,411
RAC: 32
United States
Message 934897 - Posted: 21 Sep 2009, 1:57:14 UTC - in response to Message 934892.  


Who will be first to a single box RAC above 1 million ? Probably only 2 years away.


4 cycles of Moore's law, if you're talking Seti. Vyper was the first to break the 100K barrier (at least I think so), and he's down to the high 80s now. So if we assume 100K has been reached, 200K next, then 400K, then 800K, then over a million. Closer to 6 years for Seti... Assuming work is available.

-Dave
ID: 934897 · Report as offensive
Profile Lint trap

Send message
Joined: 30 May 03
Posts: 871
Credit: 28,092,319
RAC: 0
United States
Message 934913 - Posted: 21 Sep 2009, 3:16:07 UTC
Last modified: 21 Sep 2009, 3:41:48 UTC

Want to upgrade your 295? Get an Asus MARS Dual 285! 4GB, faster clocks, all around MORE than a 295.

See the review here:
http://www.techpowerup.com/reviews/ASUS/GeForce_GTX_285_MARS/

Hurry, they are only making 1000 of them!

Martin

[edited]the charts near the end of the review rate several boards for performance per watt/per dollar, etc.
ID: 934913 · Report as offensive
Profile 52 Aces
Avatar

Send message
Joined: 7 Jan 02
Posts: 497
Credit: 14,261,068
RAC: 67
United States
Message 934920 - Posted: 21 Sep 2009, 3:24:28 UTC - in response to Message 934897.  
Last modified: 21 Sep 2009, 3:29:06 UTC

It doesn't need Moore's law, just throw money at it ;-)

That is presumably how Vyper got there (walk the numbers backwards if you don't believe me -- 100K, 50k, 25K, 12K were not the previous 6 years). Premise of my mail was: as there isn't much of a definition of what constitutes a single box, you can just about extend the RAC runway to one million today... just bus up enough nVidia cards.

Of course, a little help from Moore's law makes the statement a whole lot more practical. I'm sticking by the 2 year estimate, possibly less if either ATI or nVidia decide there is marketing value in getting there first.

[Edit] And Yes, I read someplace there is a limit of 8 cards in S@H, that's presumably enforced by software.
ID: 934920 · Report as offensive
Profile dnolan
Avatar

Send message
Joined: 30 Aug 01
Posts: 1228
Credit: 47,779,411
RAC: 32
United States
Message 934940 - Posted: 21 Sep 2009, 4:46:04 UTC - in response to Message 934920.  

It doesn't need Moore's law, just throw money at it ;-)

That is presumably how Vyper got there (walk the numbers backwards if you don't believe me -- 100K, 50k, 25K, 12K were not the previous 6 years). Premise of my mail was: as there isn't much of a definition of what constitutes a single box, you can just about extend the RAC runway to one million today... just bus up enough nVidia cards.

Of course, a little help from Moore's law makes the statement a whole lot more practical. I'm sticking by the 2 year estimate, possibly less if either ATI or nVidia decide there is marketing value in getting there first.

[Edit] And Yes, I read someplace there is a limit of 8 cards in S@H, that's presumably enforced by software.


While what you say is true (if you had access to a real supercomputer you could probably get a million today - quotas aside), I'm not sure the dollars add up... Vyper is presumably paying a lot to run that system, and while you could try to do much more, until the cash/credit ratio comes down, I don't know if anyone will want to do a million on a single box. I'm guessing that unless there's a real breakthrough in the grid (electric) and costs come way down, no one will want to pay for a million credit machine in the next 2 years. I'm also going to guess that unless someone starts pumping some real cash into the project, it's going to become harder and harder to sustain a big RAC at Seti.

-Dave
ID: 934940 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 934961 - Posted: 21 Sep 2009, 8:51:39 UTC

Well I know a way to possibly get 1M out of a single "host".. cluster. Although BOINC is not built for clustering, it could technically be done, I believe.

At the college I work at, I oversee 10 classrooms with 24 P4 530 3.0's and 2gb of RAM in each box. I was trying to build a Linux cluster out of one of the rooms, but couldn't figure it out, even when following guides with pictures. I couldn't even get the first computer to try to say it was part of the cluster.

But at any rate, that would have been what appeared to be one computer with 48 P4's and 48gb of RAM. At the time I was planning that, the only thing the server issued was AP, and very rarely would you get any MBs unless you had CUDA. During that time, one machine was pulling about 2500 for a RAC. Now 2500x24 does not add up to 1M, but that was just one classroom that was unoccupied that semester. I had two or three more that were sitting unoccupied as well.

That's what we need. Somebody needs to rewrite the BOINC code to make a cluster-friendly build. :p
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 934961 · Report as offensive
Profile maikj
Avatar

Send message
Joined: 17 Jun 06
Posts: 51
Credit: 8,024,345
RAC: 0
Germany
Message 934985 - Posted: 21 Sep 2009, 13:40:20 UTC

Well, i am building a cruncher with 4x GTX295 Hydro Copper right now. It took a lot of research to come up with the right parts. Motherboard is a ASUS P6T7,
The Case is a Thermaltake Armor+ with 10 Expansion Slots. There are 3 separate Cooling Loops with 2Fan Radiators. (2xGTX, 2xGTX, CPU/Mem) The Power Supply is a X-3 1600W with a secondary 600W PSU. The total power consumption comes to about 1350W. So 1.35kW x 24 32.4kWh/day
With $0.11kWh average in the US, you are at $3.54 / day or $106.92 per month.
That doesnt include the cost for AC...

I am hoping to finish this machine soon. I am just waiting for the last water cooling parts.

ID: 934985 · Report as offensive
Profile 52 Aces
Avatar

Send message
Joined: 7 Jan 02
Posts: 497
Credit: 14,261,068
RAC: 67
United States
Message 935072 - Posted: 21 Sep 2009, 19:14:40 UTC - in response to Message 934985.  

i am building a cruncher


Very cool & thx for the part specifics. A few questions:

(1) What speed cpu & will you overclock & how much ram ya going with? I was shocked that 4g can be had for about $99 now (times are a changin). I was also reading Tom's hardware, and they said that with Nitrogen cooling they were able to overclock an i7 to above 5 ghz (I know, the reliability guys just cringed).

(2) Electricity aside, what will all the parts cost final (including drives, etc)

(3) From the day you project enlist it, what do you estimate your peak RAC will be on that box within the first 30 days?

(4) What will you name it? I was gonna name my next system "WU-Man" just because of it's horrible sci-fi cliche value, but thought better of it ;-)
ID: 935072 · Report as offensive
Profile Hydropower
Avatar

Send message
Joined: 16 May 99
Posts: 15
Credit: 923,836
RAC: 0
Netherlands
Message 935099 - Posted: 21 Sep 2009, 21:23:15 UTC - in response to Message 935072.  

I'll give some short answers.

@52aces:
Regardless, more power to you :-) Speaking of, are multiple (centrally controlled) power supplies allowed for a single box (to fuel all those nVidia cards)? If so,

Yes, you use a splitter or automatic slave switch box. I built my own slave box.
You do not need to overclock the i7 to feed the GPUs, doing work on the i7 aside form feeding the GPU is counterproductive, it eats memory bandwidth.

@martin:
What motherboard have you got that can take that many?

I currently have the asrock supercomputer and asrock deluxe motherboards. I was thinking of an Asus P6T7.

@martin's lighthouse:
Want to upgrade your 295? Get an Asus MARS Dual 285! 4GB, faster clocks, all around MORE than a 295

Hmm, have you seen them at newegg yet ? :) And, how WIDE are these cards ? If memory serves me they are slightly wider than 2 slots, so no 4 cards in one box.

@Maikj:
Thermaltake Armor+
good choice. Hydropower is built around one.
With $0.11kWh average in the US
Electricity in the Netherlands costs around EUR 0,31 per KWh. that's around 0,40 US$. go figure..
ID: 935099 · Report as offensive
Profile -= Vyper =-
Volunteer tester
Avatar

Send message
Joined: 5 Sep 99
Posts: 1652
Credit: 1,065,191,981
RAC: 2,537
Sweden
Message 935100 - Posted: 21 Sep 2009, 21:24:50 UTC
Last modified: 21 Sep 2009, 21:26:27 UTC

The sad part is that i didn't break the 100K barrier yet, i was up to 97K then the s@h servers started to have difficulties.

If they would've coped some more days i would've breached the 100K line :) ..

So all in all , worlds highest achieved yet is 97K for the record.

Kind regards Vyper

P.S While you're all at it, please join my team GPU Users , i know this isn't the team recruite forum but there were many potential users here atm :) D.S

_________________________________________________________________________
Addicted to SETI crunching!
Founder of GPU Users Group
ID: 935100 · Report as offensive
OzzFan Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Apr 02
Posts: 15691
Credit: 84,761,841
RAC: 28
United States
Message 935103 - Posted: 21 Sep 2009, 21:33:09 UTC - in response to Message 935100.  
Last modified: 21 Sep 2009, 21:35:14 UTC

I see your RAC as 103,441 as of right now. Doesn't that mean you've officially broke the 100k RAC barrier? [edit] Or is this a single box RAC achievement you're after?
ID: 935103 · Report as offensive
1 · 2 · Next

Message boards : Number crunching : How to stuff 4 x GTX295 in one box


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.