Building a 32 thread xeon system doesn't need to cost a lot

Message boards : Number crunching : Building a 32 thread xeon system doesn't need to cost a lot
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 7 · 8 · 9 · 10 · 11 · 12 · Next

AuthorMessage
Profile OVERTAKE_ME Project Donor
Avatar

Send message
Joined: 20 May 16
Posts: 21
Credit: 458,341
RAC: 0
United Kingdom
Message 1829401 - Posted: 9 Nov 2016, 18:56:39 UTC - in response to Message 1829168.  

Hi Jim,

... I just assumed there would be a size property setup for the tag used in this forum.

If there is, I've never found it, and I've looked :)
l8r


Some forums have this tag attribute; I think they would have to add it to the SETI forum in order for this to work.

Thanks,

Chris.
ID: 1829401 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1829457 - Posted: 9 Nov 2016, 22:48:56 UTC
Last modified: 9 Nov 2016, 22:55:27 UTC

With thermal paste, I never ever do that "make a glob in the center and call it done" approach. I follow the instructions that Arctic Silver gave years ago.. put a small glob in the center, and then use a razor blade or a credit/gift card to spread it out across the whole surface.

Ideally, you want a very thin layer of compound, because the whole point of the compound is just to fill-in the pits in the porous surface of both the CPU's heat spreader and the base of the heatsink.

If the CPUs report they are 90C, the heatsink should be too hot to touch, even all the way up at the highest point. If the CPU is reporting 90c but the heatsinks are warm but not instant-pain when you touch them, heat is not being transferred to them properly.



But if in doubt: Arctic Silver's instructions for Intel processors. I see some methods are vertical line, horizontal line, middle dot, or surface spread.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1829457 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1849
Credit: 268,616,081
RAC: 1,349
United States
Message 1829463 - Posted: 9 Nov 2016, 23:12:40 UTC - in response to Message 1829457.  

With thermal paste, I never ever do that "make a glob in the center and call it done" approach. I follow the instructions that Arctic Silver gave years ago.. put a small glob in the center, and then use a razor blade or a credit/gift card to spread it out across the whole surface.

Ditto.
Also, the other commonly available tool for spreading this is a plastic bread bag closer. Always found it interesting that the instructions vary based on CPU type.
ID: 1829463 · Report as offensive
Profile OVERTAKE_ME Project Donor
Avatar

Send message
Joined: 20 May 16
Posts: 21
Credit: 458,341
RAC: 0
United Kingdom
Message 1829465 - Posted: 9 Nov 2016, 23:12:51 UTC - in response to Message 1829457.  

Hi Cosmic_Ocean,

Thanks for the reply.

I currently using Nocturas Heat sinks and Fans, so I have just bought some Nocturas CPU cleaner, and Nocturas thermal paste.

I have just read what you wrote with great interest; hopefully this next I seat the heat sinks/fans I will do a better job.

I think all avenues are pointing to the heat sink/fans not being done correctly due to non-excessive heat coming from them (when cpu's at 90c).

Thanks for your reply and advice,

Chris
ID: 1829465 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1829474 - Posted: 10 Nov 2016, 0:14:19 UTC - in response to Message 1829465.  
Last modified: 10 Nov 2016, 0:26:12 UTC

Hi Cosmic_Ocean,

Thanks for the reply.

I currently using Nocturas Heat sinks and Fans, so I have just bought some Nocturas CPU cleaner, and Nocturas thermal paste.

I have just read what you wrote with great interest; hopefully this next I seat the heat sinks/fans I will do a better job.

I think all avenues are pointing to the heat sink/fans not being done correctly due to non-excessive heat coming from them (when cpu's at 90c).

Thanks for your reply and advice,

Chris


I would say if you can place your hands on the heatsinks while the CPU cores are indicating 90ºC and keep them there. Then you may not have good thermal transfer to the heatsinks.

I don't really go for anything fancy when I apply thermal compound anymore. There were times when I spent a great deal of time trying to get that perfectly thin layer or 100% coverage. In the end I found unless you are shooting for some high end OC and need the last 0.0001% of thermal transfer it is a bunch of wasted effort.

My basic steps are:
1. Apply a 3-4mm ball of thermal compound to the middle of the CPU heat spreader.
2. Moosh the compound with the heatsink working it back and forth 3-4 times. (I twist clockwise and back.)
3. Pull the heatsink to see if it is covering ~80% of the CPU thermal spreader.
4. If coverage looks OK bolt the heatsink down.
Since the Xeons are a bit larger than a normal CPU I used a bit more thermal compound. Perhaps a 5-6mm ball.

I'm thinking I might grab a 2nd board so the temps you get will be good to know. I might go for a pair of the NH-D9DX i4 3U's instead of the NH-U12DXi4's if they get the job done for less money.

EDIT: Also I should add that my method of "cleaning" the CPUs. After clearing any large amount from the CPU. I tear a paper towel into ~2"x2" squares. Then rub the CPU spreader until the paper towel stops picking up any old thermal compound.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1829474 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13722
Credit: 208,696,464
RAC: 304
Australia
Message 1829539 - Posted: 10 Nov 2016, 4:50:44 UTC - in response to Message 1829457.  
Last modified: 10 Nov 2016, 4:52:56 UTC

With thermal paste, I never ever do that "make a glob in the center and call it done" approach. I follow the instructions that Arctic Silver gave years ago.. put a small glob in the center, and then use a razor blade or a credit/gift card to spread it out across the whole surface.

Ideally, you want a very thin layer of compound, because the whole point of the compound is just to fill-in the pits in the porous surface of both the CPU's heat spreader and the base of the heatsink.

Agreed.
Metal to metal gives the best heat transfer.
However as the surfaces aren't perfectly smooth there is still a large amount of surface area that isn't in contact with the other surface. The idea of heatsink compound is to fill in these air gaps to transfer the heat (air is actually a pretty good insulator when it comes to heat transfer- look at how well it works for seals & penguins when trapped in their fur & feathers).
It's not there to stop the 2 bits of metal from actually making contact with each other.
If it were possible to get a perfect finish on the face of the heatsink & the heat spreader, heatsink compound wouldn't be necessary.
Grant
Darwin NT
ID: 1829539 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22160
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1829549 - Posted: 10 Nov 2016, 6:25:38 UTC

All heatsink compound age, so it is worth considering replacing them after a few years of service.
Also, they become more mobile at high temperatures, so it is worth checking the tightness of the mounting screws, while the system is still hot, after a few hours of use.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1829549 · Report as offensive
Cruncher-American Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor

Send message
Joined: 25 Mar 02
Posts: 1513
Credit: 370,893,186
RAC: 340
United States
Message 1829566 - Posted: 10 Nov 2016, 9:31:11 UTC
Last modified: 10 Nov 2016, 9:42:28 UTC

Ideally, you want a very thin layer of compound, because the whole point of the compound is just to fill-in the pits in the porous surface of both the CPU's heat spreader and the base of the heatsink.


I use Vaseline (straight petroleum jelly, no scent or other additives) for thermal compound, and a very small amount at that (just get the cpu surface barely sticky with it) - I just put a dab on my fingertip and smear it on the cpu. Never had a problem with cpu temps. It fills the microscopic cracks, doesn't evaporate or dry out, doesn't run out. And is easy to clean up with a tissue. And, it is very cheap compared to the thermal compound you buy as thermal compound. I got the idea from this guy:

http://www.dansdata.com/goop.htm

(An oldie, but a goodie. Originally published in 2002. But still relevant.)

He used some other oddball things, and I realized from what he was saying that petroleum jelly should work, and by gum, it does.

Just make sure your heatsink is properly seated on the cpu!
ID: 1829566 · Report as offensive
Profile OVERTAKE_ME Project Donor
Avatar

Send message
Joined: 20 May 16
Posts: 21
Credit: 458,341
RAC: 0
United Kingdom
Message 1829792 - Posted: 11 Nov 2016, 10:09:45 UTC
Last modified: 11 Nov 2016, 10:15:38 UTC

Hi HAL9000,


I'm thinking I might grab a 2nd board so the temps you get will be good to know. I might go for a pair of the NH-D9DX i4 3U's instead of the NH-U12DXi4's if they get the job done for less money.


I will give you an update on temps once I re-seat the Heat sinks/fan.

Currently, there are around 40c on idle, and 90c+ on 100% load!


It would be interesting to hear what temp's you get on those D9DX's..

Thanks for all the information and advice you have given.


Hi Grant (SSSF), rob smith, and Cruncher-American,

Thanks for the information about attaching heat sinks / fans to CPU's.

Regards,

Chris
ID: 1829792 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1830083 - Posted: 12 Nov 2016, 16:33:18 UTC - in response to Message 1828648.  

I got the last bits to fully spool up the S2600CP2J with E5-2670's today. After a quick BIOS version update to add the turbo option it's cooking 32 CPU AVX tasks at 3.0GHz. The Noctura coolers seem to be keeping the CPU temps below 50ºC at the moment as well. A few hundred tasks should tell how it is going to settle in long term.



If you have the time, please try it w/o HT, too. In this case, my dual E5-2670s seemed to do about the same amount of work with or w/o HT. My hypothesis is that all threads are running the same software, so whatever thread blocking occurs because of HT is at the max (?). Or, maybe, I am delusional. In any event, would be nice to have a second opinion data point. Thanks!

Here is some data on run times for the E5-2670's at 3.0GHz with the AVX app.
32 threads: VLARs ~120m, Normal AR ~150m
16 threads: VLARs ~75m, Normal AR ~90m
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1830083 · Report as offensive
Cruncher-American Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor

Send message
Joined: 25 Mar 02
Posts: 1513
Credit: 370,893,186
RAC: 340
United States
Message 1830090 - Posted: 12 Nov 2016, 17:24:40 UTC

Here is some data on run times for the E5-2670's at 3.0GHz with the AVX app.
32 threads: VLARs ~120m, Normal AR ~150m
16 threads: VLARs ~75m, Normal AR ~90m


So HT gains 25% in total work done then:

In 5 hours (600 minutes) for VLARs

HT on - 120min/WU = 5/Tx32 = 160 WUs done
HT off - 75min/WU = 8/Tx16 = 128 WUs done

160/128 = 1.25 = 25% gain.

And similarly for Normal ARs.


A little better than I expected, actually.
ID: 1830090 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1831564 - Posted: 20 Nov 2016, 13:35:15 UTC - in response to Message 1830090.  

Here is some data on run times for the E5-2670's at 3.0GHz with the AVX app.
32 threads: VLARs ~120m, Normal AR ~150m
16 threads: VLARs ~75m, Normal AR ~90m


So HT gains 25% in total work done then:

In 5 hours (600 minutes) for VLARs

HT on - 120min/WU = 5/Tx32 = 160 WUs done
HT off - 75min/WU = 8/Tx16 = 128 WUs done

160/128 = 1.25 = 25% gain.

And similarly for Normal ARs.

A little better than I expected, actually.


I normally take the time and come up with a tasks per day value.
So something along the lines of:
32 threads: VLARs ~384, Normal AR ~307
16 threads: VLARs ~307, Normal AR ~256
However with sorties and other in between work the count seems to fall more in the 450-550 task a day range.
Extracting the host_total_credit values from the statistics_setiathome.berkeley.edu.xml it looks like it is generating ~29K credit a day with the current work.
So a pair of small-medium GPUs would likely double it's output. AS I recall my 750ti FTW would do ~10k/day with normal CUDA tasks. I expect with GBT data it would be a bit less. So perhaps a pair of 1050ti FTWs would be a good choice to about double the systems output without ramping up the KW/h's.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1831564 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1831576 - Posted: 20 Nov 2016, 14:38:08 UTC - in response to Message 1831564.  

have a Gigabyte variant of the 1050ti on the way, intended first to cut some power usage around here, get the Windows variant of Petri's Pascal code operational (having some issues with my 980, blocking development), then eventually use it with win10 on my Mac Pro via an ssd. Could be interesting to see what these can do, given the things 'look like' a repeat of the 750ti success. Since I'm still running 1080p monitors, could end up sidegrading the other machines to try get the temps down for summer.
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1831576 · Report as offensive
Profile Shaggie76
Avatar

Send message
Joined: 9 Oct 09
Posts: 282
Credit: 271,858,118
RAC: 196
Canada
Message 1831577 - Posted: 20 Nov 2016, 14:39:15 UTC

On thing I've wondered about hyper-threading is if you get more mileage out of it when an NVidia OpenCL task is paired with a CPU task on the same core -- from what I can tell there's a fair amount of polling involved when waiting for GPU results and I would hope that if this is done nicely (ie: _mm_pause) it should let the other half of the core run more or less unimpeded.

I've been too fussy about rearranging my hardware lately so my stats haven't settled but I'd like to try core affinity for the GPU tasks to see if it improves the CPU throughput when hyper-threaded. I've got a HT hex cruncher with 3 double-tasked GPUs so it's 6 GPU tasks and 6 CPU tasks so it would be a perfect machine to check.
ID: 1831577 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 1831581 - Posted: 20 Nov 2016, 15:00:56 UTC - in response to Message 1831577.  

Yeah there seems to be a state of flux with the NV Windows drivers at the moment. From what I can tell NV seem to be scrambling to get dx12, vulkan and multithreading sorted out. I'm guessing you'll either see the expected behaviour setting affinity, or otherwise some kindof issues with some kernel time trying to spread the load badly. Going by some gaming analyses anyway.
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 1831581 · Report as offensive
AMDave
Volunteer tester

Send message
Joined: 9 Mar 01
Posts: 234
Credit: 11,671,730
RAC: 0
United States
Message 1831623 - Posted: 20 Nov 2016, 20:13:01 UTC - in response to Message 1831577.  
Last modified: 20 Nov 2016, 20:14:09 UTC

@Shaggie & @jason

Start here.
ID: 1831623 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1849
Credit: 268,616,081
RAC: 1,349
United States
Message 1831952 - Posted: 23 Nov 2016, 1:21:06 UTC

Another thought for cutting costs. A lot of the Xeons and the mobos that support them will support either registered or unregistered memory, though you cannot mix the different types.
Why hassle with it? There are a ton of tested server pulls for sale on eBay right now for totally dirt cheap. I just got 6 ea. DDR3 4gb PC-10600 ECC RDIMMS for $25.50 including shipping, which will work quite nicely in my Z600. What I could possible need 24 gig of RAM for, I dunno, but at $4.25 a stick, worth keeping in mind.
I suspect I'll be able to sell the 12 gig (6x DDR3 2gb 10700 ECC UDIMM) I pull out for a bunch more than $25.00
ID: 1831952 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1831963 - Posted: 23 Nov 2016, 2:01:21 UTC - in response to Message 1831952.  
Last modified: 23 Nov 2016, 2:03:07 UTC

Another thought for cutting costs. A lot of the Xeons and the mobos that support them will support either registered or unregistered memory, though you cannot mix the different types.
Why hassle with it? There are a ton of tested server pulls for sale on eBay right now for totally dirt cheap. I just got 6 ea. DDR3 4gb PC-10600 ECC RDIMMS for $25.50 including shipping, which will work quite nicely in my Z600. What I could possible need 24 gig of RAM for, I dunno, but at $4.25 a stick, worth keeping in mind.
I suspect I'll be able to sell the 12 gig (6x DDR3 2gb 10700 ECC UDIMM) I pull out for a bunch more than $25.00

I usually do pretty well at finding bargain memory deals on old stuff. Like 32GB(8x4GB) of DDR2 PC2-5300 ECC FBDIMMs I picked up to put in an older Xeon system for well under a $1/GB. I'm only using 16GB of it right now. As that stuff gets HOT and eat up power. So I mostly look at it as having spares in the event I loose a DIMM or two.
Also for my dual E5-2670 system I took half of the 128GB(16x8GB) it came with out. Since I'm just doing some bench SETI@home testing and I don't think it has broken 6GB of total memory used.
Additionally it is now running 1 DIMM per channel instead of two. As there have been tests showing reduction in memory speeds when more than one DIMM per channel is used I figured why not give it a try. So far it doesn't look like there is any significant difference when running SETI@home tasks on the system. The difference may only be apparent when running synthetic benchmarks or under more intense memory loads.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1831963 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1849
Credit: 268,616,081
RAC: 1,349
United States
Message 1831977 - Posted: 23 Nov 2016, 3:11:16 UTC

Thanks, Hal. Good info.
I have to wish M$ hadn't pulled the performance metrics stuff out of Win10. Would have been fun to see what a difference it will make one way or another.
On the Z600, if you populate the second CPU, you must also populate the second memory bank, so that part won't apply.
Do wish there were some better techniques for measuring comparisons.
ID: 1831977 · Report as offensive
baron_iv
Volunteer tester
Avatar

Send message
Joined: 4 Nov 02
Posts: 109
Credit: 104,905,241
RAC: 0
United States
Message 1832178 - Posted: 24 Nov 2016, 13:45:16 UTC

I know it's a bit of an older thread, but I wanted to comment on the heat issues. I've found that for those first couple of days, you will see MUCH higher temperatures after reapplying (or applying) thermal paste. It takes the paste some time to get down into the cracks and imperfections. For those first 2 days, I typically run the CPUs much less than I normally would, due to the heat. After that, I shut the machine down, let it all cool off, then restart it and run it as I normally would. I use Arctic Silver 5, the same thermal paste I've used for what seems like an eternity and it's never let me down. If you take the heatsink off of a CPU after it's been hot, you will find the AS 5 to be quite thin and liquid-like, almost, which is what you want, this helps it get into the nooks and crannies.

On my 6700K system (4.6ghz), I was getting temps of around 70-75C that first day, down to about 65-70 on day two and now that I'm about a week in, it's down to 55-60C, with occasional temps up into the upper 60s when it gets hot in the house (this is on 1.35V with the old Corsair H70). I've switched to all liquid cooling.

If you're still experiencing temps in the 90s, then something isn't right. Either your motherboard is giving the CPUs too much voltage or your heatsinks aren't drawing heat away from your CPUs fast/well enough. I realize those are different than cooling a 4/8 core/thread CPU, but I've run CPUs up to 24 cores (48 threads) for other people and the method I outlined above has always worked flawlessly.

I've been at this SETI "game" since 2001 and I've run all kinds of different computers, from consumer CPUs to server CPUs, and GPUs of all shapes and sizes. I don't run any of my systems at 100% CPU load. Part of this is because it produces extra heat and heat kills components. I've been using my core i7 980x "extreme" edition 24/7/365 for 6 years, overclocked at 4.2ghz and it's never failed me, because I keep CPU temps down below 65C at all times. It has 12 threads, 6 cores. Currently it has 2 GPUs (dual gtx 1070s) and I run 1 thread per GPU and 50% of the cores active (6). This keeps my temps pretty low and has ensured that the system has lasted for me. I've run Linux on it a good deal of the time and I've found that Linux is considerably faster for Nvidia GPUs and will be returning shortly, I was just doing a test to see the difference between Win10 and Linux. So I would urge you to run somewhere around 50% of your cores, for longevity and better thermals.
-baron_iv
Proud member of:
GPU Users Group
ID: 1832178 · Report as offensive
Previous · 1 . . . 7 · 8 · 9 · 10 · 11 · 12 · Next

Message boards : Number crunching : Building a 32 thread xeon system doesn't need to cost a lot


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.