Message boards :
Number crunching :
Building a 32 thread xeon system doesn't need to cost a lot
Message board moderation
Previous · 1 . . . 7 · 8 · 9 · 10 · 11 · 12 · Next
Author | Message |
---|---|
OVERTAKE_ME Send message Joined: 20 May 16 Posts: 21 Credit: 458,341 RAC: 0 |
Hi Jim, ... I just assumed there would be a size property setup for the tag used in this forum. Some forums have this tag attribute; I think they would have to add it to the SETI forum in order for this to work. Thanks, Chris. |
Cosmic_Ocean Send message Joined: 23 Dec 00 Posts: 3027 Credit: 13,516,867 RAC: 13 |
With thermal paste, I never ever do that "make a glob in the center and call it done" approach. I follow the instructions that Arctic Silver gave years ago.. put a small glob in the center, and then use a razor blade or a credit/gift card to spread it out across the whole surface. Ideally, you want a very thin layer of compound, because the whole point of the compound is just to fill-in the pits in the porous surface of both the CPU's heat spreader and the base of the heatsink. If the CPUs report they are 90C, the heatsink should be too hot to touch, even all the way up at the highest point. If the CPU is reporting 90c but the heatsinks are warm but not instant-pain when you touch them, heat is not being transferred to them properly. But if in doubt: Arctic Silver's instructions for Intel processors. I see some methods are vertical line, horizontal line, middle dot, or surface spread. Linux laptop: record uptime: 1511d 20h 19m (ended due to the power brick giving-up) |
Jimbocous Send message Joined: 1 Apr 13 Posts: 1853 Credit: 268,616,081 RAC: 1,349 |
With thermal paste, I never ever do that "make a glob in the center and call it done" approach. I follow the instructions that Arctic Silver gave years ago.. put a small glob in the center, and then use a razor blade or a credit/gift card to spread it out across the whole surface. Ditto. Also, the other commonly available tool for spreading this is a plastic bread bag closer. Always found it interesting that the instructions vary based on CPU type. |
OVERTAKE_ME Send message Joined: 20 May 16 Posts: 21 Credit: 458,341 RAC: 0 |
Hi Cosmic_Ocean, Thanks for the reply. I currently using Nocturas Heat sinks and Fans, so I have just bought some Nocturas CPU cleaner, and Nocturas thermal paste. I have just read what you wrote with great interest; hopefully this next I seat the heat sinks/fans I will do a better job. I think all avenues are pointing to the heat sink/fans not being done correctly due to non-excessive heat coming from them (when cpu's at 90c). Thanks for your reply and advice, Chris |
HAL9000 Send message Joined: 11 Sep 99 Posts: 6534 Credit: 196,805,888 RAC: 57 |
Hi Cosmic_Ocean, I would say if you can place your hands on the heatsinks while the CPU cores are indicating 90ºC and keep them there. Then you may not have good thermal transfer to the heatsinks. I don't really go for anything fancy when I apply thermal compound anymore. There were times when I spent a great deal of time trying to get that perfectly thin layer or 100% coverage. In the end I found unless you are shooting for some high end OC and need the last 0.0001% of thermal transfer it is a bunch of wasted effort. My basic steps are: 1. Apply a 3-4mm ball of thermal compound to the middle of the CPU heat spreader. 2. Moosh the compound with the heatsink working it back and forth 3-4 times. (I twist clockwise and back.) 3. Pull the heatsink to see if it is covering ~80% of the CPU thermal spreader. 4. If coverage looks OK bolt the heatsink down. Since the Xeons are a bit larger than a normal CPU I used a bit more thermal compound. Perhaps a 5-6mm ball. I'm thinking I might grab a 2nd board so the temps you get will be good to know. I might go for a pair of the NH-D9DX i4 3U's instead of the NH-U12DXi4's if they get the job done for less money. EDIT: Also I should add that my method of "cleaning" the CPUs. After clearing any large amount from the CPU. I tear a paper towel into ~2"x2" squares. Then rub the CPU spreader until the paper towel stops picking up any old thermal compound. SETI@home classic workunits: 93,865 CPU time: 863,447 hours Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[ |
Grant (SSSF) Send message Joined: 19 Aug 99 Posts: 13746 Credit: 208,696,464 RAC: 304 |
With thermal paste, I never ever do that "make a glob in the center and call it done" approach. I follow the instructions that Arctic Silver gave years ago.. put a small glob in the center, and then use a razor blade or a credit/gift card to spread it out across the whole surface. Agreed. Metal to metal gives the best heat transfer. However as the surfaces aren't perfectly smooth there is still a large amount of surface area that isn't in contact with the other surface. The idea of heatsink compound is to fill in these air gaps to transfer the heat (air is actually a pretty good insulator when it comes to heat transfer- look at how well it works for seals & penguins when trapped in their fur & feathers). It's not there to stop the 2 bits of metal from actually making contact with each other. If it were possible to get a perfect finish on the face of the heatsink & the heat spreader, heatsink compound wouldn't be necessary. Grant Darwin NT |
rob smith Send message Joined: 7 Mar 03 Posts: 22225 Credit: 416,307,556 RAC: 380 |
All heatsink compound age, so it is worth considering replacing them after a few years of service. Also, they become more mobile at high temperatures, so it is worth checking the tightness of the mounting screws, while the system is still hot, after a few hours of use. Bob Smith Member of Seti PIPPS (Pluto is a Planet Protest Society) Somewhere in the (un)known Universe? |
Cruncher-American Send message Joined: 25 Mar 02 Posts: 1513 Credit: 370,893,186 RAC: 340 |
Ideally, you want a very thin layer of compound, because the whole point of the compound is just to fill-in the pits in the porous surface of both the CPU's heat spreader and the base of the heatsink. I use Vaseline (straight petroleum jelly, no scent or other additives) for thermal compound, and a very small amount at that (just get the cpu surface barely sticky with it) - I just put a dab on my fingertip and smear it on the cpu. Never had a problem with cpu temps. It fills the microscopic cracks, doesn't evaporate or dry out, doesn't run out. And is easy to clean up with a tissue. And, it is very cheap compared to the thermal compound you buy as thermal compound. I got the idea from this guy: http://www.dansdata.com/goop.htm (An oldie, but a goodie. Originally published in 2002. But still relevant.) He used some other oddball things, and I realized from what he was saying that petroleum jelly should work, and by gum, it does. Just make sure your heatsink is properly seated on the cpu! |
OVERTAKE_ME Send message Joined: 20 May 16 Posts: 21 Credit: 458,341 RAC: 0 |
Hi HAL9000,
I will give you an update on temps once I re-seat the Heat sinks/fan. Currently, there are around 40c on idle, and 90c+ on 100% load! It would be interesting to hear what temp's you get on those D9DX's.. Thanks for all the information and advice you have given. Hi Grant (SSSF), rob smith, and Cruncher-American, Thanks for the information about attaching heat sinks / fans to CPU's. Regards, Chris |
HAL9000 Send message Joined: 11 Sep 99 Posts: 6534 Credit: 196,805,888 RAC: 57 |
I got the last bits to fully spool up the S2600CP2J with E5-2670's today. After a quick BIOS version update to add the turbo option it's cooking 32 CPU AVX tasks at 3.0GHz. The Noctura coolers seem to be keeping the CPU temps below 50ºC at the moment as well. A few hundred tasks should tell how it is going to settle in long term. Here is some data on run times for the E5-2670's at 3.0GHz with the AVX app. 32 threads: VLARs ~120m, Normal AR ~150m 16 threads: VLARs ~75m, Normal AR ~90m SETI@home classic workunits: 93,865 CPU time: 863,447 hours Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[ |
Cruncher-American Send message Joined: 25 Mar 02 Posts: 1513 Credit: 370,893,186 RAC: 340 |
Here is some data on run times for the E5-2670's at 3.0GHz with the AVX app. So HT gains 25% in total work done then: In 5 hours (600 minutes) for VLARs HT on - 120min/WU = 5/Tx32 = 160 WUs done HT off - 75min/WU = 8/Tx16 = 128 WUs done 160/128 = 1.25 = 25% gain. And similarly for Normal ARs. A little better than I expected, actually. |
HAL9000 Send message Joined: 11 Sep 99 Posts: 6534 Credit: 196,805,888 RAC: 57 |
Here is some data on run times for the E5-2670's at 3.0GHz with the AVX app. I normally take the time and come up with a tasks per day value. So something along the lines of: 32 threads: VLARs ~384, Normal AR ~307 16 threads: VLARs ~307, Normal AR ~256 However with sorties and other in between work the count seems to fall more in the 450-550 task a day range. Extracting the host_total_credit values from the statistics_setiathome.berkeley.edu.xml it looks like it is generating ~29K credit a day with the current work. So a pair of small-medium GPUs would likely double it's output. AS I recall my 750ti FTW would do ~10k/day with normal CUDA tasks. I expect with GBT data it would be a bit less. So perhaps a pair of 1050ti FTWs would be a good choice to about double the systems output without ramping up the KW/h's. SETI@home classic workunits: 93,865 CPU time: 863,447 hours Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[ |
jason_gee Send message Joined: 24 Nov 06 Posts: 7489 Credit: 91,093,184 RAC: 0 |
have a Gigabyte variant of the 1050ti on the way, intended first to cut some power usage around here, get the Windows variant of Petri's Pascal code operational (having some issues with my 980, blocking development), then eventually use it with win10 on my Mac Pro via an ssd. Could be interesting to see what these can do, given the things 'look like' a repeat of the 750ti success. Since I'm still running 1080p monitors, could end up sidegrading the other machines to try get the temps down for summer. "Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions. |
Shaggie76 Send message Joined: 9 Oct 09 Posts: 282 Credit: 271,858,118 RAC: 196 |
On thing I've wondered about hyper-threading is if you get more mileage out of it when an NVidia OpenCL task is paired with a CPU task on the same core -- from what I can tell there's a fair amount of polling involved when waiting for GPU results and I would hope that if this is done nicely (ie: _mm_pause) it should let the other half of the core run more or less unimpeded. I've been too fussy about rearranging my hardware lately so my stats haven't settled but I'd like to try core affinity for the GPU tasks to see if it improves the CPU throughput when hyper-threaded. I've got a HT hex cruncher with 3 double-tasked GPUs so it's 6 GPU tasks and 6 CPU tasks so it would be a perfect machine to check. |
jason_gee Send message Joined: 24 Nov 06 Posts: 7489 Credit: 91,093,184 RAC: 0 |
Yeah there seems to be a state of flux with the NV Windows drivers at the moment. From what I can tell NV seem to be scrambling to get dx12, vulkan and multithreading sorted out. I'm guessing you'll either see the expected behaviour setting affinity, or otherwise some kindof issues with some kernel time trying to spread the load badly. Going by some gaming analyses anyway. "Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions. |
AMDave Send message Joined: 9 Mar 01 Posts: 234 Credit: 11,671,730 RAC: 0 |
|
Jimbocous Send message Joined: 1 Apr 13 Posts: 1853 Credit: 268,616,081 RAC: 1,349 |
Another thought for cutting costs. A lot of the Xeons and the mobos that support them will support either registered or unregistered memory, though you cannot mix the different types. Why hassle with it? There are a ton of tested server pulls for sale on eBay right now for totally dirt cheap. I just got 6 ea. DDR3 4gb PC-10600 ECC RDIMMS for $25.50 including shipping, which will work quite nicely in my Z600. What I could possible need 24 gig of RAM for, I dunno, but at $4.25 a stick, worth keeping in mind. I suspect I'll be able to sell the 12 gig (6x DDR3 2gb 10700 ECC UDIMM) I pull out for a bunch more than $25.00 |
HAL9000 Send message Joined: 11 Sep 99 Posts: 6534 Credit: 196,805,888 RAC: 57 |
Another thought for cutting costs. A lot of the Xeons and the mobos that support them will support either registered or unregistered memory, though you cannot mix the different types. I usually do pretty well at finding bargain memory deals on old stuff. Like 32GB(8x4GB) of DDR2 PC2-5300 ECC FBDIMMs I picked up to put in an older Xeon system for well under a $1/GB. I'm only using 16GB of it right now. As that stuff gets HOT and eat up power. So I mostly look at it as having spares in the event I loose a DIMM or two. Also for my dual E5-2670 system I took half of the 128GB(16x8GB) it came with out. Since I'm just doing some bench SETI@home testing and I don't think it has broken 6GB of total memory used. Additionally it is now running 1 DIMM per channel instead of two. As there have been tests showing reduction in memory speeds when more than one DIMM per channel is used I figured why not give it a try. So far it doesn't look like there is any significant difference when running SETI@home tasks on the system. The difference may only be apparent when running synthetic benchmarks or under more intense memory loads. SETI@home classic workunits: 93,865 CPU time: 863,447 hours Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[ |
Jimbocous Send message Joined: 1 Apr 13 Posts: 1853 Credit: 268,616,081 RAC: 1,349 |
Thanks, Hal. Good info. I have to wish M$ hadn't pulled the performance metrics stuff out of Win10. Would have been fun to see what a difference it will make one way or another. On the Z600, if you populate the second CPU, you must also populate the second memory bank, so that part won't apply. Do wish there were some better techniques for measuring comparisons. |
baron_iv Send message Joined: 4 Nov 02 Posts: 109 Credit: 104,905,241 RAC: 0 |
I know it's a bit of an older thread, but I wanted to comment on the heat issues. I've found that for those first couple of days, you will see MUCH higher temperatures after reapplying (or applying) thermal paste. It takes the paste some time to get down into the cracks and imperfections. For those first 2 days, I typically run the CPUs much less than I normally would, due to the heat. After that, I shut the machine down, let it all cool off, then restart it and run it as I normally would. I use Arctic Silver 5, the same thermal paste I've used for what seems like an eternity and it's never let me down. If you take the heatsink off of a CPU after it's been hot, you will find the AS 5 to be quite thin and liquid-like, almost, which is what you want, this helps it get into the nooks and crannies. On my 6700K system (4.6ghz), I was getting temps of around 70-75C that first day, down to about 65-70 on day two and now that I'm about a week in, it's down to 55-60C, with occasional temps up into the upper 60s when it gets hot in the house (this is on 1.35V with the old Corsair H70). I've switched to all liquid cooling. If you're still experiencing temps in the 90s, then something isn't right. Either your motherboard is giving the CPUs too much voltage or your heatsinks aren't drawing heat away from your CPUs fast/well enough. I realize those are different than cooling a 4/8 core/thread CPU, but I've run CPUs up to 24 cores (48 threads) for other people and the method I outlined above has always worked flawlessly. I've been at this SETI "game" since 2001 and I've run all kinds of different computers, from consumer CPUs to server CPUs, and GPUs of all shapes and sizes. I don't run any of my systems at 100% CPU load. Part of this is because it produces extra heat and heat kills components. I've been using my core i7 980x "extreme" edition 24/7/365 for 6 years, overclocked at 4.2ghz and it's never failed me, because I keep CPU temps down below 65C at all times. It has 12 threads, 6 cores. Currently it has 2 GPUs (dual gtx 1070s) and I run 1 thread per GPU and 50% of the cores active (6). This keeps my temps pretty low and has ensured that the system has lasted for me. I've run Linux on it a good deal of the time and I've found that Linux is considerably faster for Nvidia GPUs and will be returning shortly, I was just doing a test to see the difference between Win10 and Linux. So I would urge you to run somewhere around 50% of your cores, for longevity and better thermals. -baron_iv Proud member of: GPU Users Group |
©2024 University of California
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.