Question about back-to-back-to-back GPUs

Message boards : Number crunching : Question about back-to-back-to-back GPUs
Message board moderation

To post messages, you must log in.

AuthorMessage
Cruncher-American Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor

Send message
Joined: 25 Mar 02
Posts: 1513
Credit: 370,893,186
RAC: 340
United States
Message 1835099 - Posted: 10 Dec 2016, 1:34:25 UTC

I am currently running a machine with 3 GTX 980s in it ("Big32"). Due to the fact that the MB has only 5 PCIe slots, they are quite close to each other ( slot 1, 3, 5) so there is little, if any, air flow between the cards. Luckily, they are all reference-type cards, with blower fans and large openings in the ends of the cards, so cooling is not too bad. The bottom card runs around 60-65C, and the two cards immediately above run 70-75C.

I have another machine running 2 GTX 1080s in slots 1 and 5, so there is a 2-slot gap between them. So no cooling problem (the bottom card runs 56C or so, and the top card runs around 58C or so). Note that these have the more standard side fans on them, so I would assume they would be more affected by being adjacent, as in the other machine. My question is: if I insert another 1080 with side fans between them in slot 3, how will that affect the temps?

I know these run cooler than the 980s, but I am wondering if the lack of clean air flow will badly affect them. I don't want to buy another card only to find out I can't use it for heat reasons (too expensive!).

In all my machines I am running 3 WUs/GPU, and therefore, for support I need 9 CPU cores for them. Big32 is a dual EP-2670 with 32 HT cores, so no problem there.

So I won't do the extra 1080 until I get an X5650 to run in the 1080 machine (it's an X58 i7-950 - "arf-PC") to provide me with 12 HT cores. But it's on the way!
ID: 1835099 · Report as offensive
Profile Shaggie76
Avatar

Send message
Joined: 9 Oct 09
Posts: 282
Credit: 271,858,118
RAC: 196
Canada
Message 1835105 - Posted: 10 Dec 2016, 1:51:37 UTC

I know the 1070's are a bit less power-hungry but I've got 3 of them sandwiched together in 2 different machines with no problems (not founders edition but still reference blower designs). I do ensure they're fed with cool air from the front of the case, though.
ID: 1835105 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1835108 - Posted: 10 Dec 2016, 1:59:28 UTC - in response to Message 1835099.  
Last modified: 10 Dec 2016, 2:01:39 UTC

I would ask, are these the EVGA cards? Are they ACX design? Do they have backplates on them?

If the answer to all three of these questions is yes, then I would tell you not to put a third card in between the 2 existing cards.

Why?

There is a little description in the pdf that comes with the EVGA ACX card that says... there is the possibility that the fans of an adjoining cards may strike the backplate of the GPU below it if placed next to each other. They recommend the removal of the backplate if that occurs.

What they are really saying is the ACX with backplate is slightly larger than the standard 2 pcie slot design. Not quite the 2.1 of some cards but close enough that the fans of the superior card will hit the backplate and (best case just stop turning, or worse case shatter and come flying out)

I figured this out with my Mega Cruncher that had 4 ACX 1070s in it. I resolved my issue by places spacers between the GPUS that forced them apart ( but be careful as you can bend the piece that goes into the pcie slot)

Even with that, I discovered the temperatures extremely high ( I set a throttle temp of 80C) which 3 of the 4 hit regularly.

I can't remember where I saw it but I believe there is a change in how heat is removed from the GPU with the new ACX design. Before the heat was removed to the front and back of the GPUs. But somewhere I saw that the new design actually pushes the heat down away from the card and onto the GPU below it. ( I could very well be wrong in this part since I can't find the source video that I saw about this again but it made sense to me when I saw the video)

So what to do? I converted my 1070s to hybrids. By doing so I actually reduced the height of the cards so they no longer touch each other. I also shifted where the heat was exhausted to. Temps are now in the upper 30Cs on the 4 cards


Zalster
ID: 1835108 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1835109 - Posted: 10 Dec 2016, 2:12:05 UTC - in response to Message 1835099.  

My question is: if I insert another 1080 with side fans between them in slot 3, how will that affect the temps?

As long as there is physical clearance between them (so they don't foul each other as Zalster pointed out) the cards will still run cooler than the reference cards, however the one in the middle will be somewhat warmer than the other 2, and the one closest to the CPU warmer than the coolest one, but still cooler than the one in the middle (all depending on your fan speed settings of course).

in all my machines I am running 3 WUs/GPU, and therefore, for support I need 9 CPU cores for them. Big32 is a dual EP-2670 with 32 HT cores, so no problem there.

Have you checked to see if that gives the best throughput?
For my system (GTX 750Ti and a GTX 1070), running 2 SoG WUs at a time gave only slightly better throughput than 1 WU with 1 Guppie & 1 Arecibo on the same card dragging out the Arecibo WU a huge amount. Running 1 WU at a time I've found to be good enough with SoG. With your GTX 1080s 2 WU per card might be worth the overhead.
Grant
Darwin NT
ID: 1835109 · Report as offensive
woohoo
Volunteer tester

Send message
Joined: 30 Oct 13
Posts: 972
Credit: 165,671,404
RAC: 5
United States
Message 1835113 - Posted: 10 Dec 2016, 2:43:13 UTC

I tried running quad cards with backplates and no gaps and temperatures would just skyrocket. I tried two risers but that wasn't reliable for me. So now I'm running triple with one slot gap in between each of the cards, 91C/72C/84C
ID: 1835113 · Report as offensive
Cruncher-American Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor

Send message
Joined: 25 Mar 02
Posts: 1513
Credit: 370,893,186
RAC: 340
United States
Message 1835116 - Posted: 10 Dec 2016, 3:16:13 UTC - in response to Message 1835105.  
Last modified: 10 Dec 2016, 3:18:04 UTC

I know the 1070's are a bit less power-hungry but I've got 3 of them sandwiched together in 2 different machines with no problems (not founders edition but still reference blower designs). I do ensure they're fed with cool air from the front of the case, though.


That's how my 980s roll.

I would ask, are these the EVGA cards? Are they ACX design?X


No - ASUS Strix; the middle card would be an MSI

Do they have backplates on them?


Yes

Have you checked to see if that gives the best throughput?


No - it takes about a week to get anything like a settled-down RAC (after a couple of days wait after making a change to let RAC approach a steady state in terms of validated and pending-validation counts).

I tried running quad cards with backplates and no gaps and temperatures would just skyrocket. I tried two risers but that wasn't reliable for me. So now I'm running triple with one slot gap in between each of the cards, 91C/72C/84C


I am afraid that will be my case, too. I will have to get a larger (more PCIex16 slots) MB. But I will try it first. Perhaps the 1080s have better thermal properties; they do dissipate less power, after all.[/quote]

And thanks for your answers, all!
ID: 1835116 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1835119 - Posted: 10 Dec 2016, 3:36:06 UTC - in response to Message 1835116.  

Have you checked to see if that gives the best throughput?


No - it takes about a week to get anything like a settled-down RAC (after a couple of days wait after making a change to let RAC approach a steady state in terms of validated and pending-validation counts).

A week and more.
I've always just manually checked the runtimes for the various angle ranges. A much more immediate (and due to the randomness of credit, accurate) way of seeing what gives the best throughput.
Grant
Darwin NT
ID: 1835119 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1835121 - Posted: 10 Dec 2016, 3:39:18 UTC

My question is,

If you have room to add a third card in between your other two, Why not just move a card and see how it works?
ID: 1835121 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1835127 - Posted: 10 Dec 2016, 4:25:42 UTC - in response to Message 1835116.  


No - ASUS Strix; the middle card would be an MSI


hmmm...I've had questions about those cards. Looking at the head on they appear to be slightly larger than 2 pcie slots. 2.1 I know the specs say 2 but you can see there is overhang to either side of the plate where the video plugs are. If you do put them in, just check to make sure they aren't making contact before turning them on.
ID: 1835127 · Report as offensive
Cruncher-American Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor

Send message
Joined: 25 Mar 02
Posts: 1513
Credit: 370,893,186
RAC: 340
United States
Message 1835142 - Posted: 10 Dec 2016, 7:50:26 UTC - in response to Message 1835121.  

My question is,

If you have room to add a third card in between your other two, Why not just move a card and see how it works?


But I can't. There are 7 slots on the MB; only 1,3 and 5 (counting down from the CPU) are physically PCIe x16. So if I have a card in 1 and 5, the only other place for the 3rd card is 3. If you mean just have (2) cards in 1 and 3 (say), the cards still each have a completely exposed side; no card is "covered" on both sides. THAT'S the case I am worried about. I know in the case 1,3 there will be some temp rise, but the openness on one side will prevent catastrophic overheating. Or so I believe, not having tried it. Perhaps I should, but I should be in bed now. I'll try that tomorrow and see what happens.
ID: 1835142 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1835143 - Posted: 10 Dec 2016, 7:54:10 UTC - in response to Message 1835142.  

If you mean just have (2) cards in 1 and 3 (say), the cards still each have a completely exposed side; no card is "covered" on both sides. THAT'S the case I am worried about.

Yep, but it will allow you to see just how much room there is, and how warm a card will run with another right up against it's intake.
And pretty much all the after market cards are cooled much more efficiently than those with the reference coolers.
Grant
Darwin NT
ID: 1835143 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1835176 - Posted: 10 Dec 2016, 13:27:46 UTC - in response to Message 1835142.  

If your worried it will run wild when your away you can set a limit in Tthottle or better yet Percision X to prevent that.
But it's certainly worth a try to test the waters.

PercisionX works great. after my 1070 "incident" set a 75C limit on it with a low fan speed and it purred right along without throttling my 750 - as Ttrottle will affect both. Now it's quite happy with 2x120mm and 60mm fans on it -- well it works at 68C :D
ID: 1835176 · Report as offensive
Cruncher-American Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor

Send message
Joined: 25 Mar 02
Posts: 1513
Credit: 370,893,186
RAC: 340
United States
Message 1835180 - Posted: 10 Dec 2016, 14:06:13 UTC - in response to Message 1835176.  

Brent - in fact I do use Precision to control fans, so I will look at limiting temps directly with it. I don't use tthrottle; I tried it once and didn't like how it worked (IIRC). It didn't really throttle cores as such, but just turned them off and on very rapidly (or so it seemed in Task Manager).
ID: 1835180 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1835182 - Posted: 10 Dec 2016, 14:14:21 UTC - in response to Message 1835180.  

Just set your temperature as the priority, not the usage. It will go to that temp then ease off on it's power levels.
ID: 1835182 · Report as offensive

Message boards : Number crunching : Question about back-to-back-to-back GPUs


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.