Building a 32 thread xeon system doesn't need to cost a lot

Message boards : Number crunching : Building a 32 thread xeon system doesn't need to cost a lot
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 . . . 12 · Next

AuthorMessage
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1785076 - Posted: 6 May 2016, 0:03:36 UTC - in response to Message 1785068.  

Fortunately or unfortunately depending on how you look at it, they are mostly Intel boards (Sturdy/stable, but feature poor), and the extent of their BIOS adjustments is pretty much setting the fan to either normal or aggressive. Nothing in the way of fine tuning there, and of course I set them to aggressive, since they are running 2-3 of 4 cores full bore 24x7 so I'd think the cooler the better.

One thing you probably never thought about, and neither did I till I was sitting there with Speedfan open and sort of daydreaming for a minute while configuring it. I noticed that one of the fans was running 3900 RPMs and I was thinking about longevity.

3900rpm x 60min x 24hours x 365days = over 2 Billion revolutions a year. Pretty amazing they don't self destruct sooner than they do when you think about it.

Oh well, I will keep using it for now, until I find some better software that controls the CPU fan. I wonder if HP has some utility that I can try tuning the fans with? I'll have to put that on the list to look into.

Yeah some motherboards don't have a lot in the way of fan configuration in the BIOS. Sometimes you don't even get Auto or Manual settings.

For CPU fans I generally don't run them run any faster than is needed. AS I found that the old way of running high volume fans at full tilt constantly really didn't accomplish anything. Other than making a lot of noise. Now I tend to go with low RPM high volume fans.
For the dual Xeon X5470 that I have half put together. Once I have the heatsinks mounted so they properly draw heat from the CPUs I'm planning to remove the fan from the CPU coolers. Then use either 2 140mm or 3 120mm fans to blow air across the MB. Possibly with a shroud, likely made from cardboard, to better channel the air over the bits that need cooling. Similar to the layout in 1U & 2U rack servers.
At the moment I am using several 120mm or 140mm NZXT fans in my systems. Which run 1200 RPM, 59 CFM, & 27 dBA at full tilt. They spend most of their time around 800 RPM. As that seems to be as slow as they operate.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1785076 · Report as offensive
Al Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Avatar

Send message
Joined: 3 Apr 99
Posts: 1682
Credit: 477,343,364
RAC: 482
United States
Message 1785098 - Posted: 6 May 2016, 2:52:03 UTC - in response to Message 1785076.  

I've been recently playing around with that as well, starting the GPU fan at 100%, and seeing under load what the operating temps are, and then backing them down maybe 5-10% per step and noting the temp differences. I've found in my limited testing so far that I can drop it from 100% to 70% and only pick up a degree or 2, and at 50% maybe 5-10 degrees, so I've decided to try running them between 60-70% for the time being, as the sound levels at 75% and below are much less noticeable.

I've done the wind tunnel thing with thin cardboard on my Megahelms that I had bought 5 years ago when I started this farm project, I think they do make a little difference, but it's been a while since I've played with them. Well, if nothing else, this is fun playing with new systems and processors. As hobbies go, this one isn't too bad in either terms of initial or longer term outlays, but as the farm comes online, my opinion on that may change...

ID: 1785098 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1853
Credit: 268,616,081
RAC: 1,349
United States
Message 1785101 - Posted: 6 May 2016, 3:18:43 UTC - in response to Message 1784968.  
Last modified: 6 May 2016, 4:00:46 UTC

Jim, I have both Coretemp64 and Speedfan running on this system, have all the fans cranked up to 11(!), and am not very impressed with HP's extremely expensive (for what it appears you get) CPU coolers. Maybe the AS5 thermal paste needs to set up a little, but in core temp, they are running at 71-77* on both procs. This seems quite high to me, compared to other systems I have running, but to be honest, this is the only one in a case, the rest are just boards on trays out in the open. Oh, one other thing I should add, Speedfan and Coretemp don't agree on the CPU temps, Coretemp is giving the 70* readings on the cores, Speedfan only lists one temp, about 55* but doesn't break it out per CPU/core like Coretemp does, so I guess I don't know what to believe...

What are your temps in your Z600 running, are they comparable, or a bit more 'normal'? Thanks for any suggestions!

The Z600 has 2xE5620 @ 2.40GHz. HWinfo reports Core Max average at 63 and 55, with max of 69 and 62. First core will always seem to run about 5 higher, as the second core cooler blows right into it. When I got the machine, I did remove the fan from each cooler and pull out all the carpet fuzz that had built up over the years, and hit the vanes with compressed air. BTW, it is possible to remove the fans without removing the cooler from the CPU. Remove the top screws, loosen the lower ones if possible, then rotate the vertical bars enough out of the way that the fan can be slid up. Could try replacing the fans on the factory heat pipes, perhaps with Arctic PWM fans @$20 ea.
My Z400's, unfortunately, are running in the hi 70's consistently. TJmax on these Xeons is somewhere north of 95c, so I'm not too concerned about anything under 80.
I have these in the basement for the summer, and fans are cranked to the max in BIOS. In the winter, they heat my living room:)
ID: 1785101 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1853
Credit: 268,616,081
RAC: 1,349
United States
Message 1785108 - Posted: 6 May 2016, 3:56:29 UTC - in response to Message 1785068.  

I wonder if HP has some utility that I can try tuning the fans with? I'll have to put that on the list to look into.

I've looked, so far without success. Ever find anything, let me know. Seems silly that there isn't, but HP has the BIOS locked down so tight on these it's lucky we can look at it ;)
ID: 1785108 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1853
Credit: 268,616,081
RAC: 1,349
United States
Message 1785109 - Posted: 6 May 2016, 4:02:12 UTC

I'm running Malik's HWInfo64, as it lets me watch all machines across the network and gives me a bunch of info about each machine, including GPU data. Might be worth a look. Very decent, especially for a freebie.
I especially like that I can turn on widgets for the tool bar with color-coded data for each machine.
ID: 1785109 · Report as offensive
Al Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Avatar

Send message
Joined: 3 Apr 99
Posts: 1682
Credit: 477,343,364
RAC: 482
United States
Message 1785204 - Posted: 6 May 2016, 14:17:40 UTC - in response to Message 1785101.  

My Z400's, unfortunately, are running in the hi 70's consistently. TJmax on these Xeons is somewhere north of 95c, so I'm not too concerned about anything under 80.
I have these in the basement for the summer, and fans are cranked to the max in BIOS. In the winter, they heat my living room:)

I can certainly believe that, I can't believe how much heat that thing puts out. Maybe it is just appearances because it is concentrated coming out the back side of the case instead of sitting in the open, but even my dual X5690s don't feel like they are generating that much heat. Now that it is apparently working properly, I think I am going to NNT it for now, and when it's dry shut her down for the season. Once the free cooling starts again in about 5-6 months, I'll fire the ol girl back up and add her back into the mix.

ID: 1785204 · Report as offensive
Phud Redux

Send message
Joined: 20 Apr 16
Posts: 270
Credit: 2,976,272
RAC: 1
United States
Message 1787202 - Posted: 13 May 2016, 3:51:28 UTC

i bought one E5-2660 over a month ago for 54 bucks. three weeks later i got a second one for 49 bucks. as i see it the only drawback is finding an X79/C602/C606 board. the seti system is paired with one Gtx 760. which is pumping out really well.
ID: 1787202 · Report as offensive
Phud Redux

Send message
Joined: 20 Apr 16
Posts: 270
Credit: 2,976,272
RAC: 1
United States
Message 1787329 - Posted: 13 May 2016, 19:33:48 UTC - in response to Message 1787202.  

going to start the other E5-2660 on sunday, i think. our niece is graduating from university tomorrow, so sunday will be better.
ID: 1787329 · Report as offensive
Profile Zombu2
Volunteer tester

Send message
Joined: 24 Feb 01
Posts: 1615
Credit: 49,315,423
RAC: 0
United States
Message 1787331 - Posted: 13 May 2016, 20:13:33 UTC

just grabbed a 96 core opteron system on ebay runs good and stable
I came down with a bad case of i don't give a crap
ID: 1787331 · Report as offensive
Profile ML1
Volunteer moderator
Volunteer tester

Send message
Joined: 25 Nov 01
Posts: 20267
Credit: 7,508,002
RAC: 20
United Kingdom
Message 1787368 - Posted: 13 May 2016, 23:28:09 UTC - in response to Message 1787331.  

just grabbed a 96 core opteron system on ebay runs good and stable

Yeah!!! Way to go!!

Slightly jealous...

I'm running some 32 core opteron systems to good effect. Very solid and consistent and reliable for server work even if not the hottest for number crunching.

But then again: GPUs are always going to be the winners for number compute power...


Happy fast crunchin
Martin
See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)
ID: 1787368 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13732
Credit: 208,696,464
RAC: 304
Australia
Message 1787375 - Posted: 14 May 2016, 0:01:55 UTC - in response to Message 1787331.  

just grabbed a 96 core opteron system on ebay runs good and stable

What's it's power load?
Grant
Darwin NT
ID: 1787375 · Report as offensive
Profile Zombu2
Volunteer tester

Send message
Joined: 24 Feb 01
Posts: 1615
Credit: 49,315,423
RAC: 0
United States
Message 1787389 - Posted: 14 May 2016, 1:18:52 UTC - in response to Message 1787368.  
Last modified: 14 May 2016, 1:19:47 UTC

just grabbed a 96 core opteron system on ebay runs good and stable

Yeah!!! Way to go!!

Slightly jealous...

I'm running some 32 core opteron systems to good effect. Very solid and consistent and reliable for server work even if not the hottest for number crunching.

But then again: GPUs are always going to be the winners for number compute power...


Happy fast crunchin
Martin


yeah but still 32 cores is nice too
i got the newer 16 core opterons coming 2.5ghz 16 cores whith a grand total of 128 cores

they where cheap too 35$ a piece ...yay for amd products and their rapid depriciation

power load is 2.8kw at 100% cpu but the psu's are 2 phase 220v 9.8Amp per psu
I came down with a bad case of i don't give a crap
ID: 1787389 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13732
Credit: 208,696,464
RAC: 304
Australia
Message 1787394 - Posted: 14 May 2016, 1:59:51 UTC - in response to Message 1787389.  

power load is 2.8kw at 100% cpu but the psu's are 2 phase 220v 9.8Amp per psu

2.8kW!
Is that actual load when running, or maximum rating for the unit?
Grant
Darwin NT
ID: 1787394 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1787395 - Posted: 14 May 2016, 2:22:05 UTC - in response to Message 1787394.  
Last modified: 14 May 2016, 2:22:39 UTC

power load is 2.8kw at 100% cpu but the psu's are 2 phase 220v 9.8Amp per psu

2.8kW!
Is that actual load when running, or maximum rating for the unit?

The spec sheet for the Dell PowerEdge C6145 says it can be equipped with either 1100W or 1400W redundant PSUs. I'm guessing they have the 1400w PSUs. All of my Dell servers typically would run 50-60% of the PSU rating when at full load & filled with HDDs. That is normally in the peak efficiency range for PSUs. So it makes sense the power supplies would be spec'd to run in that range.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1787395 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13732
Credit: 208,696,464
RAC: 304
Australia
Message 1787398 - Posted: 14 May 2016, 2:42:49 UTC - in response to Message 1787395.  

The spec sheet for the Dell PowerEdge C6145 says it can be equipped with either 1100W or 1400W redundant PSUs. I'm guessing they have the 1400w PSUs. All of my Dell servers typically would run 50-60% of the PSU rating when at full load & filled with HDDs. That is normally in the peak efficiency range for PSUs. So it makes sense the power supplies would be spec'd to run in that range.

So the actual load would be up to around 840W. That's still a lot of power for a single system.


Before one of my video cards died, I had 2 systems with 4 video cards & 3 monitors; total load through the UPS was around 500W max.
Grant
Darwin NT
ID: 1787398 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1853
Credit: 268,616,081
RAC: 1,349
United States
Message 1787426 - Posted: 14 May 2016, 4:46:04 UTC

Pretty wild when we get to where we lowly number crunchers have more under the hood than that project servers do :)
ID: 1787426 · Report as offensive
Profile bloodrain
Volunteer tester
Avatar

Send message
Joined: 8 Dec 08
Posts: 231
Credit: 28,112,547
RAC: 1
Antarctica
Message 1788881 - Posted: 20 May 2016, 1:42:17 UTC - in response to Message 1776402.  

wow price to performance is. very very good.
ID: 1788881 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1853
Credit: 268,616,081
RAC: 1,349
United States
Message 1788908 - Posted: 20 May 2016, 5:37:08 UTC - in response to Message 1788881.  

wow price to performance is. very very good.

??
ID: 1788908 · Report as offensive
Profile George 254
Volunteer tester

Send message
Joined: 25 Jul 99
Posts: 155
Credit: 16,507,264
RAC: 19
United Kingdom
Message 1789211 - Posted: 21 May 2016, 10:07:35 UTC - in response to Message 1788881.  

Very tempted to get a tower unit instead of a stack of laptops.

Yes purchase price to performance is good, but compared to laptops, running costs are high. My stack of 6 laptops use between 3 & 20 watts so with average of say 15 watts, that should give total usage of around 90 watts. Tasks vary between 30 and 65 per day with credits between 3,600 and 3,900 per day. Two machines run Rosetta as well as SETI@Home and I have just added Enigma to a separate machine as well.
ID: 1789211 · Report as offensive
Cruncher-American Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor

Send message
Joined: 25 Mar 02
Posts: 1513
Credit: 370,893,186
RAC: 340
United States
Message 1789216 - Posted: 21 May 2016, 11:52:00 UTC - in response to Message 1789211.  

Very tempted to get a tower unit instead of a stack of laptops.

Yes purchase price to performance is good, but compared to laptops, running costs are high. My stack of 6 laptops use between 3 & 20 watts so with average of say 15 watts, that should give total usage of around 90 watts. Tasks vary between 30 and 65 per day with credits between 3,600 and 3,900 per day. Two machines run Rosetta as well as SETI@Home and I have just added Enigma to a separate machine as well.


Yes - but one 32-thread machine (with no GPU work at all) will run 500+ WUs/day, drawing about 400 watts. So roughly 10 times the work done for about 5 times the cost.
ID: 1789216 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 . . . 12 · Next

Message boards : Number crunching : Building a 32 thread xeon system doesn't need to cost a lot


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.