Message boards :
Number crunching :
How to stuff 4 x GTX295 in one box
Message board moderation
Author | Message |
---|---|
MarkJ Send message Joined: 17 Feb 08 Posts: 1139 Credit: 80,854,192 RAC: 5 |
The guys over at GPUgrid posted this link. I thought i'd repeat it here... Manifold Runs over 1,920 Processor Cores in New GIS Supercomputing Desktop Record BOINC blog |
Hydropower Send message Joined: 16 May 99 Posts: 15 Credit: 923,836 RAC: 0 |
You might also want to check my previous post there on the pitfalls, especially on powersupplies: http://www.gpugrid.net/forum_thread.php?id=1091&nowrap=true#10119 |
zpm Send message Joined: 25 Apr 08 Posts: 284 Credit: 1,659,024 RAC: 0 |
um, crunch. |
MarkJ Send message Joined: 17 Feb 08 Posts: 1139 Credit: 80,854,192 RAC: 5 |
You might also want to check my previous post there on the pitfalls, especially on powersupplies: Yes I remember our discussions about them. Did you end up RMAing all 4 power supplies? They even used the same motherboard as you. Interesting that the guys in this article only had a single 1250w power supply. I don't think it would last too long with the load that 4 x GTX295's would take (according to nvidia 4 x 289w = 1156w). BOINC blog |
Hydropower Send message Joined: 16 May 99 Posts: 15 Credit: 923,836 RAC: 0 |
Well, I RMA'ed one and they investigated it. It seems that the sound was okay and I am still using all four of them. I did RMA one Evga card which gave me trouble and received a replacement card. So far so good. I cannot believe that one single 1250 W PS will feed them reliably. When crunching for MilkyWay my powermeter shows a fairly constant 1300 W real draw, and that project is not pushing the limits on the NVidia cards yet. |
MarkJ Send message Joined: 17 Feb 08 Posts: 1139 Credit: 80,854,192 RAC: 5 |
Well, I RMA'ed one and they investigated it. It seems that the sound was okay and I am still using all four of them. I did RMA one Evga card which gave me trouble and received a replacement card. So far so good. Assuming a 90% efficency power supply they would have next to nothing left over to drive the motherboard and hard drives, so it clearly wouldn't be reliable. I thought you were looking at trying to get more cards in one machine as your next project. ASUS P6T workstation perhaps? You could give them a run for their money, or better yet get them to pay you. You'll need to take lots of photos as you put it together. BOINC blog |
Hydropower Send message Joined: 16 May 99 Posts: 15 Credit: 923,836 RAC: 0 |
That is correct. I have been looking into the possibility of running seven double cards in one system. From a heat-perspective I have it figured out, also for the powersupplies and way-of-build. I need to rethink fincances versus probability of success though. Drivers, bios, tools and applications all need to recognise 14 gpus. this would not look like your average PC anymore and have a substantial weight. |
zpm Send message Joined: 25 Apr 08 Posts: 284 Credit: 1,659,024 RAC: 0 |
it would be nice if some manu. made a pci-e 16x 2.0 connector bridge.... sort of like a | | | | | Wire | |||||||||||||| pci-e extender... | | | | | | |
ML1 Send message Joined: 25 Nov 01 Posts: 20326 Credit: 7,508,002 RAC: 20 |
... seven double cards in one system... What motherboard have you got that can take that many? Happy hot crunchin'! Regards, Martin See new freedom: Mageia Linux Take a look for yourself: Linux Format The Future is what We all make IT (GPLv3) |
52 Aces Send message Joined: 7 Jan 02 Posts: 497 Credit: 14,261,068 RAC: 67 |
When folks are running dual quad core CPUs with 1920 more cores on video, I gotta wonder what is definitionally a single box anymore :-) If it's OS instance, then just virtualize in the other direction. Regardless, more power to you :-) Speaking of, are multiple (centrally controlled) power supplies allowed for a single box (to fuel all those nVidia cards)? If so, then the runway for more and more PCI-E slots can be extended quite a bit provided there was a market for bigger boards. And you know nVidia & ATI will continue to race one another for more beef per slot. Who will be first to a single box RAC above 1 million ? Probably only 2 years away. |
dnolan Send message Joined: 30 Aug 01 Posts: 1228 Credit: 47,779,411 RAC: 32 |
4 cycles of Moore's law, if you're talking Seti. Vyper was the first to break the 100K barrier (at least I think so), and he's down to the high 80s now. So if we assume 100K has been reached, 200K next, then 400K, then 800K, then over a million. Closer to 6 years for Seti... Assuming work is available. -Dave |
Lint trap Send message Joined: 30 May 03 Posts: 871 Credit: 28,092,319 RAC: 0 |
Want to upgrade your 295? Get an Asus MARS Dual 285! 4GB, faster clocks, all around MORE than a 295. See the review here: http://www.techpowerup.com/reviews/ASUS/GeForce_GTX_285_MARS/ Hurry, they are only making 1000 of them! Martin [edited]the charts near the end of the review rate several boards for performance per watt/per dollar, etc. |
52 Aces Send message Joined: 7 Jan 02 Posts: 497 Credit: 14,261,068 RAC: 67 |
It doesn't need Moore's law, just throw money at it ;-) That is presumably how Vyper got there (walk the numbers backwards if you don't believe me -- 100K, 50k, 25K, 12K were not the previous 6 years). Premise of my mail was: as there isn't much of a definition of what constitutes a single box, you can just about extend the RAC runway to one million today... just bus up enough nVidia cards. Of course, a little help from Moore's law makes the statement a whole lot more practical. I'm sticking by the 2 year estimate, possibly less if either ATI or nVidia decide there is marketing value in getting there first. [Edit] And Yes, I read someplace there is a limit of 8 cards in S@H, that's presumably enforced by software. |
dnolan Send message Joined: 30 Aug 01 Posts: 1228 Credit: 47,779,411 RAC: 32 |
It doesn't need Moore's law, just throw money at it ;-) While what you say is true (if you had access to a real supercomputer you could probably get a million today - quotas aside), I'm not sure the dollars add up... Vyper is presumably paying a lot to run that system, and while you could try to do much more, until the cash/credit ratio comes down, I don't know if anyone will want to do a million on a single box. I'm guessing that unless there's a real breakthrough in the grid (electric) and costs come way down, no one will want to pay for a million credit machine in the next 2 years. I'm also going to guess that unless someone starts pumping some real cash into the project, it's going to become harder and harder to sustain a big RAC at Seti. -Dave |
Cosmic_Ocean Send message Joined: 23 Dec 00 Posts: 3027 Credit: 13,516,867 RAC: 13 |
Well I know a way to possibly get 1M out of a single "host".. cluster. Although BOINC is not built for clustering, it could technically be done, I believe. At the college I work at, I oversee 10 classrooms with 24 P4 530 3.0's and 2gb of RAM in each box. I was trying to build a Linux cluster out of one of the rooms, but couldn't figure it out, even when following guides with pictures. I couldn't even get the first computer to try to say it was part of the cluster. But at any rate, that would have been what appeared to be one computer with 48 P4's and 48gb of RAM. At the time I was planning that, the only thing the server issued was AP, and very rarely would you get any MBs unless you had CUDA. During that time, one machine was pulling about 2500 for a RAC. Now 2500x24 does not add up to 1M, but that was just one classroom that was unoccupied that semester. I had two or three more that were sitting unoccupied as well. That's what we need. Somebody needs to rewrite the BOINC code to make a cluster-friendly build. :p Linux laptop: record uptime: 1511d 20h 19m (ended due to the power brick giving-up) |
maikj Send message Joined: 17 Jun 06 Posts: 51 Credit: 8,024,345 RAC: 0 |
Well, i am building a cruncher with 4x GTX295 Hydro Copper right now. It took a lot of research to come up with the right parts. Motherboard is a ASUS P6T7, The Case is a Thermaltake Armor+ with 10 Expansion Slots. There are 3 separate Cooling Loops with 2Fan Radiators. (2xGTX, 2xGTX, CPU/Mem) The Power Supply is a X-3 1600W with a secondary 600W PSU. The total power consumption comes to about 1350W. So 1.35kW x 24 32.4kWh/day With $0.11kWh average in the US, you are at $3.54 / day or $106.92 per month. That doesnt include the cost for AC... I am hoping to finish this machine soon. I am just waiting for the last water cooling parts. |
52 Aces Send message Joined: 7 Jan 02 Posts: 497 Credit: 14,261,068 RAC: 67 |
i am building a cruncher Very cool & thx for the part specifics. A few questions: (1) What speed cpu & will you overclock & how much ram ya going with? I was shocked that 4g can be had for about $99 now (times are a changin). I was also reading Tom's hardware, and they said that with Nitrogen cooling they were able to overclock an i7 to above 5 ghz (I know, the reliability guys just cringed). (2) Electricity aside, what will all the parts cost final (including drives, etc) (3) From the day you project enlist it, what do you estimate your peak RAC will be on that box within the first 30 days? (4) What will you name it? I was gonna name my next system "WU-Man" just because of it's horrible sci-fi cliche value, but thought better of it ;-) |
Hydropower Send message Joined: 16 May 99 Posts: 15 Credit: 923,836 RAC: 0 |
I'll give some short answers. @52aces: Regardless, more power to you :-) Speaking of, are multiple (centrally controlled) power supplies allowed for a single box (to fuel all those nVidia cards)? If so, Yes, you use a splitter or automatic slave switch box. I built my own slave box. You do not need to overclock the i7 to feed the GPUs, doing work on the i7 aside form feeding the GPU is counterproductive, it eats memory bandwidth. @martin: What motherboard have you got that can take that many? I currently have the asrock supercomputer and asrock deluxe motherboards. I was thinking of an Asus P6T7. @martin's lighthouse: Want to upgrade your 295? Get an Asus MARS Dual 285! 4GB, faster clocks, all around MORE than a 295 Hmm, have you seen them at newegg yet ? :) And, how WIDE are these cards ? If memory serves me they are slightly wider than 2 slots, so no 4 cards in one box. @Maikj: Thermaltake Armor+good choice. Hydropower is built around one. With $0.11kWh average in the USElectricity in the Netherlands costs around EUR 0,31 per KWh. that's around 0,40 US$. go figure.. |
-= Vyper =- Send message Joined: 5 Sep 99 Posts: 1652 Credit: 1,065,191,981 RAC: 2,537 |
The sad part is that i didn't break the 100K barrier yet, i was up to 97K then the s@h servers started to have difficulties. If they would've coped some more days i would've breached the 100K line :) .. So all in all , worlds highest achieved yet is 97K for the record. Kind regards Vyper P.S While you're all at it, please join my team GPU Users , i know this isn't the team recruite forum but there were many potential users here atm :) D.S _________________________________________________________________________ Addicted to SETI crunching! Founder of GPU Users Group |
OzzFan Send message Joined: 9 Apr 02 Posts: 15691 Credit: 84,761,841 RAC: 28 |
I see your RAC as 103,441 as of right now. Doesn't that mean you've officially broke the 100k RAC barrier? [edit] Or is this a single box RAC achievement you're after? |
©2024 University of California
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.