Hello group,

I've been repurposing old hardware and putting it to work for SETI, but now I think I'm running into cooling issues. I need help with some ideas on how to approach the problem.

The setup: I have a cluster of six (soon to be seven) small Dell desktop machines sitting next to one another. They are mini-ATX-ish size, or something like that. They each use their own power supply that is in the case. Each case has a fan on the CPU and another one built into the power supply. Everything is running Ubuntu with a base install and nothing special going on with the hardware (no overclocking, etc.).
I'll try to post a photo this evening when I get home.

The problem: My power bill has been increasing noticeably, and I think that a big part of it is the cooling requirements. All the individual power supply and case fans run 24-7. I've taken to leaving the side panel off of the cases, which seems to help a little bit. I need to come up with some kind of cooling plan for the system as a whole though, rather than relying on each system to cool itself.
My hope is that some kind of overall cooling for the whole cluster will help dramatically, and allow the individual case fans to slow down. Hopefully that will be more efficient that having all the individual fans running continuously. I'm accumulating the hardware to bring online a second identical cluster in a month or two, so I want to solve this issue before that.

The design: not sure yet. Any ideas?
I'm leaning towards building some kind of enclosure that I can mount the desktops into, and then use a few USB fans to blow air over the whole cluster. My hope is that would help remove heat more quickly and allow the case fans to slow down, with a net increase in electrical efficiency.
Eventually I would like to get something like a truck tool box and locate the cluster outdoors. I live in Alaska and we get about 4-5 months where the temperature never gets above freezing. For the fall-through-spring seasons that might be a good way to keep things cool. That doesn't help me for the summer months though.

Other (possibly nuts) ideas:
-Remove everything from the individual cases and build some kind of structure to hold the motherboards and associated hardware out in the open, kind of like a mining rig. (Could really help with cooling, but would be quite an intricate project too. Could be expensive, and definitely would be time-consuming.)
-Do the above, but submerge the whole setup in some kind of fluid cooling. (Not sure what to use for the working fluid though.)
-Run more than one machine off larger, externally-located power supplies. (Would this give me more efficient use of power, rather than having one power supply for each system? How would the wiring work?)
-Buy a chest-high freezer at Costco, bribe my girlfriend with jewelry so I can set it up at her house, and house the cluster inside it. (The jewelry purchase might be more expensive than the current electricity cost though.)

What do you guys think?
Brandon

One thing that would help with your cooling problem, with the overall performance of your computers and to a lesser extent with the power bill is to stop using the built-in GPUs for processing.
The Intel built-in GPUs cause problems in that they share a lot of resources with the CPU, thus "hobble" the CPU, at the same time their computing performance is very limited, and proportionately they consume a fair chunk of power.
There are several places you can turn off using the Intel GPUs, first is from your web settings (https://setiathome.berkeley.edu/prefs.php?subset=project) - this only works if you are using the web settings. Second is to use the BOINC manager option - go to the "advanced view", then go to the "activity" or "options" menu, and set "use GPU" to off (I can't recall the exact location, because I'm away from access to may main crunchers and those beside me just now don't have GPUs, so the options don't show).

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

Why don't you get a box fan to blow across the cluster. $20 at the home repair stores. Spend a little more ($40) and purchase one of the whole house type fans. They move an incredible amount of air. I have one that sits on the floor at the doorway of a bedroom that houses two crunchers and moves 4230 CFM on HIGH. Only 18" in diameter. It will be loud though even on MEDIUM.

---

Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)

Rob,
That's interesting about the built-in Intel GPUs not being all that useful. I've come across posts to that effect before, but haven't pursued it further. I have one salvaged laptop with a built-in Nvidia card that seems to be a good processor. It shows up as "NVIDIA Quadro K2100M (1999MB) driver: 390.11 OpenCL: 1.2" in my computers listing, and is one of the best performing systems. I should probably leave that one turned on?
The only other two machines that are doing GPU computing are two windows laptops. I'll try turning off the GPU computing on those machines and see if it makes much difference in the computing work.
Problem is that in the main cluster (I still need to get a photo up) all of the boxes are running Ubuntu, and so none of those are doing GPU computing. They are all little machines that don't even have a PCI slot on the boards.

Keith,
I may start out with the fan idea. We've recently had an especially "warm" summer here in Anchorage (mid 70's) so all the local stores have been sold out of fans, lol. I should be able to scare up something though.

All,
I did some more looking around online and came across mineral oil-submerged systems. It's probably overkill for what we do, but that would be pretty cool. I'm going to do some more research and then build two proof-of-concept systems. First I'll use a raspberry pi, and then I'll use another one of these salvaged desktops that I have. If it all works I might be able to move everything into a big tank and then use a fish tank bubbler to bubble cool air through the oil, as well as set up basic circulation. I'l post about it when I get further along.

Oil immersed cooling is good, but one has to be very careful when choosing just about every component used. Depending on the oil you can have problems with connectors, cables, pcbs, resistors, capacitors inductors, integrated circuits etc - indeed just about everything used in all consumer level computers. When we were specifying the beast for which I've just finished commissioning the cooling system we looked at phase-change, water, oil-immersed, aerosol, and ended up with a dry-air-blast closed loop system. This beast draws over 90kw "from the wall", and has to run 24/7/365 - and we keep the devices in the 35-40C range. There were just too many unknowns with all the other cooling systems - even those that are commonly used in other fields.
For "normal" domestic life I would seriously look at pre-built liquid cooling (which relies on plate-separation between the hot stuff and the coolant and remotely located radiators) or "simple" air cooling - depending on the local ambient temperature (liquid above ~25 to 28C, air below).

The big problem is with the Intel built-in GPU, which is actually on the same die as the CPU. In the case of your system with a Qudro that is a separate device probably integrated into the motherboard. While "old hat" compared to things like the nVidia RTX2080 etc it is certainly better than any of the Intel GPU of that era (and indeed many of their newer ones).

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

I was wondering about that myself: which components might be affected by the oil. Most oils act as a solvent to some degree, and it's an unknown as to which components might be affected by immersion. I wonder if the board could be sprayed with some kind of sealant before immersion to reduce the effects. Even something simple like a clear krylon spray paint might provide a decent barrier to any solvent action, but should still transfer heat pretty well.
I read somewhere that anything with moving parts needs to be kept out of the oil. I'm guessing that fans and hard drives built for use in air end up burning out when trying to churn through the much heavier oil?

Yeah, on that laptop the graphics are a separate card. It is integrated into the motherboard, but is indeed a separate device. (I had to take that machine apart to replace the CPU and GPU cooling paste, and it was an ugly project to get inside there.) That laptop is another of the salvaged ones I picked up at my prior job.

For the cooling project I'm going to pursue two different routes simultaneously:
1. Open up the computers that are currently running and remount them inside some kind of enclusure where I can use external fans to force air through the whole enclosure. That should help with the "microclimate" around the systems and allow the intrgrated fans to relax a bit.
2. Take a few of the other systems I haven't put into use yet and experiment with running them in oil.
2.1 Remove the fans from a system (CPU and power supply) and see if it will even work in an oil bath.
2.2 If 2.1 is successful for a week or two, take another system and spray paint everything I can get to with a clear coat. Then see if one system dies before the other, or if they both happily crunch along with the air-cooled
systems.

Now I just have to figure out how to get a raspberry pi connected to temperature sensors and in-line ammeters. Then I could do an apples to apples comparison of average temperature, life cycle, and power use to see if two oil-cooled systems do better than air-cooled ones. lol.

To do oil cooling one really has to start from scratch. Most coatings just aren't up to the job, and those that work are in the "£$%&* expensive (and/or toxic) category.....
Also oil can be horrendously messy - you need to make sure that the tank is oil tight (oil will often find its way out through joints that water doesn't). Further you have to drain and clean everything before you can start work, then dispose (or store) the oil.
One thing many folks don't realise is that most oils are hygroscopic, and this can cause no end of problems in the medium to long term.

Obviously one would certainly have to keep all fans out of the stuff, and hard drives, but also the majority of connectors used on "domestic" electronics are susceptible to oil-creep and so go intermittent. It is highly probable you would need to have oil circulating pumps and stirrers to get the hot oil away from hot components to where it can be cooled.

Another thing to consider is that oil is not as good a thermal conductor as water, and being more viscous is not as good at getting into tight corners as air.

Two weeks is far too short a time to see what's going to happen. A far cheaper way to see what's going to fail is to get samples of everything that's in a computer (bits of PBB, surface mount capacitors and resistors, bits of wire and plugs, a few chips etc) and leave them in jars of various different oils for a few weeks in a very warm place (you need at least 50C/110F). For connectors have both "open" and "mated" pairs with cable attached - make sure the remote ends of the wire are well outside the oil - measure the voltage drop and resistance across the connectors before putting in the oil, then at weekly intervals for a couple of months.

For air cooling:
Certainly opening the case on the majority of smaller cases I've seen would help. For example I run mid-size ATX cases, and have at least one side off each just to stop too much warm air going where it shouldn't.

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

Looking back through the thread I see that Keith suggested air cooling very early on.
Thinking aloud here for a moment.
Air cooling is cheap and reliable, and in your situation it will be far easier to setup and maintain.
First you need to lay out the computers in a "sensible" array so that air can be "blasted" through them - simple shelving will be sufficient to start with.
Remove all the cases, so you've just got the bottom, front and rear panels. (I think you are talking about the old Dell "pizza box" computers)
Depending on the space available either two layers of three or three layers of two would get you going. Create some simple ducting so the "cool" air goes into the front of the array, and exits through the back. Space the computers an inch or two apart horizontally, and 3 or 4 vertically (it's best to get the front of the array more or less square - you'll see why in a moment). Make sure there is about an inch between PSU inlet and any shelf - simple blocks can be used to lift the computers up to get clearance if needed.
Find your input fan(s) - these should be slightly larger diameter than the diagonal of the array. Position the fan approximate 1 fan diameter in-front of the computers, centered on the centre of the front face. The input fan should be mounted in a simple cylindrical duct as long as the fan is deep, with the fan mounted centrally, leading into a flat square plate with a suitable size hole cut in it - this plate acts the interface between the circular fan and the square ducts that follows.
Now for the ducting - this can be just about anything - wood, ply, chipboard, thick corrugated card, preferably not plastic (you don't want to create too much static electricity). The input part is a truncated pyramid tapering from the fan diameter to the size of the front face of the computer array. then next part is a very simple rectangular, open ended box to enclose the computers. Finally is the exhaust stage - this should open out in about half the length of the inlet duct to about 10% bigger than the inlet duct.
All told this unit will be about four feet long - plus say six inches at each end to allow the air in and out.
Enhancements - a simple net "scrim" over the inlet to filter out dead leaves and stop the cat investigating is a "good idea" - a second exhaust fan will help, but if you do that don't forget to increase the length of the exhaust duct by the thickness of the fan.

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

You are going to have to cool them, no other way around it. I have re-purposed 7 hp refurbs, in a small room downstairs. (my office, etc) most have 3 or 4 graphics cards added in mining frames. Rather than spending hours and hours gaming solutions, i got a cheapy 110v air conditioner from walmart and threw it in the window. Works great, yes will add to the bill (and a 230V ac unit will use less power in the long run) But I went with quick and dirty.... also can add more if I want by just adding them on the shelf. The room is 12x12 and used for nothing else usually........

---

"Don't worry about it, nothing is gonna be O.K. anyway.........."

Hello group,

I've been trying to get an image of the cluster posted but have been having difficulty getting it to show up. Stay tuned on that . . . .

I agree about starting off with air cooling for the time being. I picked up some USB powered fans and have those loosely extracting warm air from the area of the computers, and it is making a noticeable difference. I could feel the heat buildup between the cases before, but now the air space between computers is back to room temperature. The combination of leaving the case doors off and adding some extra fans is indeed helping.

I'm going to see if I can make some kind of enclosure around the boxes next, and position the fans at each end. That will make the whole assembly a lot less unsightly, as well as facilitate cooling. I also like the idea of keeping some computer components in jars of oil for a while to see how they perform. I may go that route as well.

I'll see if I can get the photos up in the near future as well.
Thanks everyone!