Originally posted on the 1st farm thread, ( http://www.setiathome.ssl.berkeley.edu/forum_thread.php?id=51094 ) but decided to start a new one of my own. Here is my post there...

Hi, thinking about doing the same thing, but in a slightly different way, wondering if a couple ideas I had might work.

I am getting a number (4-6) Intel DG33FBC motherboards in the next couple weeks, and thought about setting up a farm of my own, but not in the 'normal' way. I think this board is ideal, as it has everything integrated, and is pretty stable. Here is what I though I would attempt, after reading this thread:

Set them up with quad core procs, thinking about Q9550s as they are 45nm and should use less energy. I plan on doing them caseless, with a KVM switch, using the on board video.

After reading this thread, and considering the pros and cons, I think I would like to do a diskless, server based (as opposed to USB) setup. Like the original poster, I have very very limited experience with Linux, and seeing there is now a 'pre-packaged' distro out there helps a bunch.

One of the things I am trying to do to keep space and (somewhat) costs down, it to possibly rig this stack up with a single Power Supply, instead of individual case/PS's. I know, this is where it might get interesting. I have a extra PCP&C 1KW power supply currently sitting around unused, which (from reading the specs) should easily run 4-5 of these boards, the CPU maxes out at 90w, and can't imagine the board takes 1/2 of that? But of course, there are some issues...

I know that I will have to do some interesting wiring, getting a hold of some harnesses to supply them. That shouldn't be _too_ hard, but then there is the issue of individually controlling them. I know modern PS's are controlled with momentary switches, I suppose I could try and find some old 180w Dell PS's that had a mechanical switch, but how much fun would that be, and of course, it would increase power consumption. Does anyone have any thoughts on setting up multiple identical boards using a single PS, and the best way to control them individually?

I didn't see in the post, might have missed it, is there a setup for the server to boot all these WS's from? I saw the USB one mentioned. Also, how much memory would be enough to make this setup work the most efficiently? Is 1 gig enough, would 2 be better, or would 4 be best?

Any thoughts anyone might want to add to this project I am considering would be greatly appreciated. It sounds like fun, and will do quite a bit of crunching, once the headache of the setup/config is taken care of.

Thanks!

---

Take a simple OS to control your machines. With simple I mean small, without all the fancy graphics some major OS supplier thinks he has to add to their program.
Try Linux, you won't regret it. It is stable, it is fast and therefore won't give you headaches.
The amount of ram is not important. I just saw that my 4 WU' each take between 32 and 37MB of memory. At the moment the complete computer uses 573MB of ram with zero in swap.
The other one, running one WU, only uses 181 MB.
When you have 1GB per machine you have more than enough, especially when you also have a 1GB cache.
Your questions about running diskless and the power supplies have to be answered by others, here I can not help you with.
Good luck with your machines, let us know when it's all working, maybe with some nice pictures.

---

______
DeMus

Thanks for the thoughts, I am right now putting together a system from an old P4 1.6 to serve as my server, 16 gig HD, from what I read, this should be fine, just need to be sure to have enough memory. Still waiting to hear from others on my Linux questions, and if my idea to use 1 PS for 4-6 boards is feasable. I know it's Sunday, so maybe the experts are taking a day of rest... ;-) I've got time, and it will take a bit of it to figure out the server install, not to mention the diskless workstations. Have a bit to learn, I know, time to start reading!

---

Thanks for the thoughts, I am right now putting together a system from an old P4 1.6 to serve as my server, 16 gig HD, from what I read, this should be fine, just need to be sure to have enough memory. Still waiting to hear from others on my Linux questions, and if my idea to use 1 PS for 4-6 boards is feasable. I know it's Sunday, so maybe the experts are taking a day of rest... ;-) I've got time, and it will take a bit of it to figure out the server install, not to mention the diskless workstations. Have a bit to learn, I know, time to start reading!

If you're going to try to use one power supply for more than one board (not recommended), you would need to get at least a 1KW PSU. For the price of those, it is cheaper to get 4 450's depending on the brand. Not only that, but you would have to do some cutting/splicing to attach multiple ATX connectors to the same harness. It's just messy and bad things are likely to happen.

Now I have gone the other way around... hooking up multiple PSUs to one system. That works just fine, you just have to bond the grounds, bright green, and grey wires together so both PSUs will turn on when you press the button, and the bonded grounds just make sure things don't get out of whack.

---

Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)

Cosmic_Ocean, thanks for the reply. I did mention I had a 1KW PC Power & Cooling power supply lying around unused, and was hoping to put it to good, if very unconventional, use. I had planned on getting 3-4 harnesses, I have the soldering ability, that and a little heat shrink tubing, it shouldn't be too difficult. The big question for me is how to individually control each board on a single PS.

I believe the PS cuts the power to the board upon a reboot, not sure, but if so, each time it reboots, it would take down all the systems attached. That's why I posted here, wondering if anyone had any ideas as to how, if it's even possible, to control multiple boards individually from the one PS. Doing this more for fun/learning than actual cost savings, I know I can find some low power PS's around, I probably have a few lying around, but would prefer to do it with one, if I can.

Thanks again!

---

One of the things I am trying to do to keep space and (somewhat) costs down, it to possibly rig this stack up with a single Power Supply, instead of individual case/PS's. I know, this is where it might get interesting. I have a extra PCP&C 1KW power supply currently sitting around unused, which (from reading the specs) should easily run 4-5 of these boards, the CPU maxes out at 90w, and can't imagine the board takes 1/2 of that? But of course, there are some issues...

I've spent the last few weeks reviewing low-power processor/motherboards and it is amazing how much the motherboard can draw.

I would not recommend relying on your imagination.

Also, 90w is the "Thermal Design Power" and while it is proportional to actual consumption, it's really about heatsink and cooling design.

Get one motherboard and CPU, and get a Kill-a-watt or something similar and measure the actual draw.

Your 1000w supply should comfortably deliver about 400w with margin. Try to push past 600w and you'd have problems.

I have my doubts about using one supply for multiple boards, though.

---

I have no expertise with running multiple boards with single power supply. But I have come across such setups. Prime example is the Hillbilly Supercomputer seti stack run by Luke Mester. He has posted a couple of pictures on his site. Mayb contacting him might be a good idea.
______________

---

Thanks for the suggestion, I just sent Luke an email asking him to take a peek at this thread, and to then chime in with any suggestions he may have for me. Hopefully the info he has will apply to my PS, as he is using an old AT one, big, strong, but very simple compared to today's, on old P100-200 boards. Who knows, I'll keep my fingers crossed...

I will def take your advice though, as I have a couple Kill-A-Watts, I'll grab one and plug it in, see what it is drawing with just the one board running full tilt, that will help me assess what I might be able to do.

I may have not stated clearly my initial assumptions on the power requirements, I meant to say that the CPU takes 90w and the board might take 1/2 again of that, for a total of ~150w, again just a guess at this point, of course. Kill-a-watt will give me a more complete picture.

Here's the specs of my PS, it seems to have enough capacity, and they have a rep for overbuilding their products.

Output: +5V @ 30A
+12V @ 72A (80A peak)
-12V @ 0.8A
+3.3V @ 24A
+5VSB @ 3.5A
continuous power = 1000W
peak power = 1100W

Regulation: 2% (+3.3V, +5V, +12V)
5% (-12V)
Ripple: 1% (p-p)
Hold Time: 32ms
PG Delay: 300ms

Compatibility: NVIDIA® SLI™ / ATI CrossFire™ Certified
M/B Connectors: 24-pin, dual 8-pin, 4-pin
Graphics Connectors: two 6-pin and two 6/8-pin PCI-E
Drive Connectors: 15 (6 SATA, 8 Molex, 1 mini)
MTBF: 120,000 hours
Warranty: 5 Years


Thanks again guys.

---

Well, got the parts in, and have set it up temporairly in the family room, but need to get it working and moved into the basement, per the Boss... Still working on the power supply harness issue, and haven't heard back yet from the person who had done this before with the AT PS's and the P100 & 133's. They do boot up, and search for a network connection to boot from, then time out, so far so good.

Now all I need to do is to figure out how to configure the server to have them boot the OS from there. More reading to do...

Oh, and I have a couple pics of the current setup, but don't know how to post them, I don't see a link to add them to the posting. Anyone give me some advice on this? Thanks much!

---

Oh, and I have a couple pics of the current setup, but don't know how to post them, I don't see a link to add them to the posting. Anyone give me some advice on this? Thanks much!

What I would suggest, go to Photo Bucket and create a free account, you can upload tons of pictures, and then link the pictures to here.

If you hover over a picture in the album (after it has been uploaded), the way I do it is click in the "Direct Link" portion (a left-click automatically copies it to the clipboard), and then add it here like:

[url=paste URL here]some text here[/url]

That saves people on slow connections from having to load the pictures when they view threads, and just makes things look a little more organized, in my opinion.

---

Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)

That works, a little bit more messing around, but if it makes it easier on ppl with slow connections, it's worth it, as I don't post that many pics to make it much of a hassle. Here they are:

The Setup

Closeup

I plan on organizing it a bit better when I get it installed downstairs, but 1st I need to get it working. I have some questions, I printed out the instructions for the setup, is there a way to do it non-command line, thru the GUI instead? Also, I plan on having this as a seperate segment in the network, I have a D-Link gamer router that will keep it segmented, but have some confusion about setting both the server up and the router.

I know the IP of my main router, as well as the Gateway. Wouldn't this router get it's own IP from the main router, and then give it's own IP's to each of the workstations via it's own DHCP internally, doing the translation out to the main network?

It's been a long time since I have configured my router, and set things like this up. When you don't use it, you loose it, I guess. Any thoughts/suggestions would be appreciated.

Thanks

---

A few years ago I had 11 motherboards running S@H. This was with Pentium 200 processors. I was able to run several boards from one power supply. I doubt that this would work very well with modern MPU's. They take a lot more current. I found that the limiting factor on how many boards I could connect to one ps was the voltage drop in the wiring harness. I needed longer & longer wiring to reach each motherboard as more were added. With the high currents and low voltages involved you will quickly go below the minimum voltage on the 5V supply. You should measure all power supply voltages at the motherboard power connector & make sure they are within tolerance. You will also need to measure the current draw to make sure that you don't exceed the ps rating. A low resistance shunt will be needed to measure current. A standard multimeter has to much voltage drop to be used.

---

Hey Luke, thanks for the reply. It looks like I am going to be building my harnesses, and will be using heavier than standard wires for them. I plan on soldering each connection to limit resistance as much as possible, and have heard about possible timing issues, so I plan on making them all the same length, even though they will range from 1-3 feet in actual distance from the board to the PS. I have a Fluke 189 multimeter, so I should be able to measure things pretty accurately, and have an amp probe for it was well, so I'll be able to tell if it is getting close to it's limits.

One concern I have is about controlling them individually, as they don't work like they used to, the board controls the PS, as opposed to the other way around back in the good ol days. I can hook up a reset button (momentary switches) from some old cases I have lying around to each board, but I think if I do a power one, it will turn off the power to the whole group, and this can't be good. That will be one hurdle I will need to find a way to address if I plan on getting this setup working stablely.

Well, it will be an interesting process, working Way outside the box on this one, we'll have to see how it goes, I can always get a number of 300 watt PS's and hook one up to each, but besides the cost, I would have to think it would also use more energy. Keep an eye here, if you're interested, I plan on documenting how it's working. I just connected my 8 port KVM switch to the boards, so it should be not too long till I get them running, just wish I knew Lunix better. Thanks again for the post!

---

Modern PSU's run most efficiently between approximately 30% and 80% of full rated output. I suggest you get a cheap wattmeter, like a "Kill a Watt" or similar. You could then test your setup with one, then two MB on it and see where the wattage takes you. Then you could predict the load of more MB on that PSU and decide just how nasty that conversion cable needs to be.

So your 1,000watt PSU would be most comfortable indicating less than 800w on the wattmeter, I think. Any Electrical Engineers care to clarify this?

Here are some watt draw readings from systems I tested last year on SETI@home:

All systems were running SETI MB opti apps with very small graphics card, no CUDA. The graphics cards drew between 10 and 30 watts. All PSU are rated 80% efficient:

OC QX9770 water cooled tower, water pump plus fan draw 25w: Total 280w
OC Q9450 air cooled (lots of fans): Total 250w
Small form factor Q9650 slight OC from 3.0 to 3.2GHz: Total 155w
Dual AMD Opteron system with 4 drive RAID and minimal graphics: Total 220w

The same QX9770 running nothing but BOINC on the CPU (no CPU SETI processing) plus 3 GTX295 cards running CUDA is currently averaging around 920watts. If the CPU cores are loaded up with 4 tasks they take a total of 100 watts more at the wall outlet. This seems to infer approximately 246watts drawn by each heavily loaded GTX295.
Calculated as (1020watts - 280watts) / (3xGTX295) = 246watts per GTX295.

I hope this helps.

Bob Mahoney

Just as an aside I liked the closeup best,especially of the good taste in WINE from Newcastle. LOL

---

Bob:

Modern PSU's run most efficiently between approximately 30% and 80% of full rated output. I suggest you get a cheap wattmeter, like a "Kill a Watt" or similar. You could then test your setup with one, then two MB on it and see where the wattage takes you. Then you could predict the load of more MB on that PSU and decide just how nasty that conversion cable needs to be.

I have a killawatt, and planned on doing just as you suggested. I spoke with a tech as PCP&C, one thing he suggested was to utilize another 12v line or 2 from a different cable, due to the increased amp draw that will be required by this unusual setup.

So your 1,000watt PSU would be most comfortable indicating less than 800w on the wattmeter, I think. Any Electrical Engineers care to clarify this?

Of course, if money was no object and I didn't already have a 1KW PS just sitting around, I could go with something like this: Koolance 1300/1700W Liquid-Cooled Power Supply, Rev 1.5 and run it on 220v power to get the full output of it...

Here are some watt draw readings from systems I tested last year on SETI@home:

All systems were running SETI MB opti apps with very small graphics card, no CUDA. The graphics cards drew between 10 and 30 watts. All PSU are rated 80% efficient:

OC QX9770 water cooled tower, water pump plus fan draw 25w: Total 280w
OC Q9450 air cooled (lots of fans): Total 250w
Small form factor Q9650 slight OC from 3.0 to 3.2GHz: Total 155w
Dual AMD Opteron system with 4 drive RAID and minimal graphics: Total 220w

Here is my final setup, now that all the hardware pieces are finally in place: 1 QX9550, 1 QX6600, & 6 QX9450's. The 9550 also has a mid range GPU on it (no external 4 pin power plug on it, so it can't be _too_ powerful), to be able to play with CUDA for fun if I choose. All the systems are bare Intel boards, with CPU cooling fans - no overclocking, no case fans(no cases), no HD's, and (mostly) no GPU's (on board video, LAN, etc) - which should help keep the power draw to a minimum. I plan on documenting it as I go along, it will be an interesting experiment to see how what effects what. My initial expectation is that they will draw between 125-175 watts each, depending on CPU utilization, but this is just a SWAG, the proof will be in the testing.

I am still working out the configuration for the farm, with Dotsch and his Dotsch/UX - A USB/Diskless/Harddisk BOINC Linux Distribution. I have been struggling a bit with it (I am a green Linux newb), but he has provided invaluable help in getting it close to working. Hopefully within a week we'll be getting it up and running in test mode, so I can see what the power situation will be. The only other hardware issue I am still researching is controlling multiple boards with one PS, that has been a harder nut to crack, but I have a few ideas.

The same QX9770 running nothing but BOINC on the CPU (no CPU SETI processing) plus 3 GTX295 cards running CUDA is currently averaging around 920watts. If the CPU cores are loaded up with 4 tasks they take a total of 100 watts more at the wall outlet. This seems to infer approximately 246watts drawn by each heavily loaded GTX295.
Calculated as (1020watts - 280watts) / (3xGTX295) = 246watts per GTX295.

I hope this helps.

Bob Mahoney

Bob, it sure does, thanks for taking the time to respond, and I will be sure to post comparitive info on my setup.


FiveHamlet:

Just as an aside I liked the closeup best,especially of the good taste in WINE from Newcastle. LOL

Hehe thanks, you should see it now with all 8 boards together, they're everywhere, it's driving my neat nut wife crazy (we're pretty much Oscar & Felix, guess which one I am...)

Oh, & thanks for noticing, I believe it was vintage early 2009, a very good year... ;) That's pretty much all I drink. Once you drink Newcastle, most others don't compare very well, it is quite good, and unique. I just wish more places carried it on tap, as there's nothing better than a tall cold glass filled with a pour from a fresh keg. Yum! Bottles are just placeholders in between, and I have a spare fridge that was scrapped from work (we got it in, could find nothing wrong with it. mfg's are nuts, you wonder why new appliances are so expensive?) that might just get converted into a kegerator, if I don't either build one myself from scratch, or purchase a pre-made one. I've been looking at True models, but they are $1000-1400 new, and I can throw one together for about 1/2 of that, but again, all it takes is free time...

---

I think you are going in wrong direction. I think a better way to go is to buy some cheap board, stick cheap CPU and then fill the PCI and PCI-e slots with nVidia cards, run CUDA.

I have old Socket 370 board, it got 4 PCI slots that means I can run four video cards with nVidia 8400GS, about 50 dollars per card. Each of those intel G33 boards will cost you about 100 dollar each. And you need to get all the guts for them, like CPU, RAM, hard drives, some kind of power supply.

---

I think you are going in wrong direction. I think a better way to go is to buy some cheap board, stick cheap CPU and then fill the PCI and PCI-e slots with nVidia cards, run CUDA.

I have old Socket 370 board, it got 4 PCI slots that means I can run four video cards with nVidia 8400GS, about 50 dollars per card. Each of those intel G33 boards will cost you about 100 dollar each. And you need to get all the guts for them, like CPU, RAM, hard drives, some kind of power supply.

Yep, they were about 100 each, but the good thing is I am running them bare, with nothing but a CPU, memory and a cooling fan. I think I have about $340-350 in each. I can always drop in Vid cards, like a 295 in the PCI-E and a few PCI cards in the others, but would like to see if I can get them running diskless 1st. It looks like I've bit off a wee bit more than I could chew here, as the hardware turned out to be the easy part, all it took was some $... Getting it to work together properly, in the way I would like it to work, well, that's a whole 'nother thing. Should be interesting, though.

---

I think you are going in wrong direction. I think a better way to go is to buy some cheap board, stick cheap CPU and then fill the PCI and PCI-e slots with nVidia cards, run CUDA.

I have old Socket 370 board, it got 4 PCI slots that means I can run four video cards with nVidia 8400GS, about 50 dollars per card. Each of those intel G33 boards will cost you about 100 dollar each. And you need to get all the guts for them, like CPU, RAM, hard drives, some kind of power supply.

Yep, they were about 100 each, but the good thing is I am running them bare, with nothing but a CPU, memory and a cooling fan.

Oh yeah, I was going to ask, how are you running it diskless? USB sticks?

---

Well, I am trying to run it Diskless, but am having some problems with getting it networked correctly. Quite frustrating, as I have all these boards laying around not being used at this time. I did break down today and toss a HD on my fastest one, and loaded a copy of Win 7 Beta on it, for giggles. I wanted to see how it would work on decent hardware, without a lot of ram. Works fine as far as I can tell, but the interface is driving me nuts. I am truely hoping that they will have an option for Windows classic desktop/taskbar, like they did in XP (and, I've heard) Vista. I can't imagine the nightmare most medium to large sized businesses are going to have migrating to this, as most I believe are running XP. The user retraining is going to be enormous...

Anywho, I actually am going to take your previous advice, and try something out just for fun. I ended up buying 3 EVGA GeForce 8400 GS PCI 512 MB DDR2 64-bit 567MHz DirectX 10 SLI Video Cards (they were about $50 ea with free shipping) to see what I might be able to do to create a crunching demon. I plan on putting them into my Win 7 box, (I loaded BOINC on it to see how it runs under 7, and to stress test the hardware a little) not exactly sure how it will react to having 4 video cards in in it, nor how to tell BOINC/CUDA how to see/utilize them, but if nothing else, just like this Linux farm project, it will be interesting. I just wish there was someone near me who knew Linux backwards & forwards, I would be happy to buy them dinner (or maybe spring for a little cash) for some help getting this farm working. My wife is getting about to the end of her patience with this, and I haven't had a lot of luck making it connect. I tried the new version, I was hoping it would clear up the issues I was having, but it brought along a few new ones. I'll have to PM Lars with my current findings, I am glad he likes hearing from a newbie, I can imagine it must be frustrating dealing with such seemingly simple (to him) problems. I'm sure I'll get it working one of these days...

---