Buying a Supercomputer?

Message boards : Number crunching : Buying a Supercomputer?
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · Next

AuthorMessage
C Horse

Send message
Joined: 25 Jul 18
Posts: 5
Credit: 10,361,509
RAC: 1
United States
Message 1950659 - Posted: 19 Aug 2018, 3:58:37 UTC

Hello,

I'd like to buy a "supercomputer" for the main purpose of contributing to research like SETI and medical projects. I'm also somewhat interested in using it for VR, because why not.

I have an approximate budget of 20-30k, and have a few questions:

-How wasteful of my funds is this? Would building my own Linux cluster be 2x cheaper? 5x?

-What computer should i buy to get the most bang for my buck? I was looking at the "Silverdraft Power Resolve Workstation"... but not sure if I could stretch my dollars further with a different computer.

-Does this invite a security risk? I was once told by a friend that having such a powerful computer can attract the attention of hackers who would like to take control of the machine for mining bitcoin or whatever else.

Thanks for the help!

C
ID: 1950659 · Report as offensive
Profile arkayn
Volunteer tester
Avatar

Send message
Joined: 14 May 99
Posts: 4438
Credit: 55,006,323
RAC: 0
United States
Message 1950662 - Posted: 19 Aug 2018, 4:42:17 UTC - in response to Message 1950659.  

Hello,

I'd like to buy a "supercomputer" for the main purpose of contributing to research like SETI and medical projects. I'm also somewhat interested in using it for VR, because why not.

I have an approximate budget of 20-30k, and have a few questions:

-How wasteful of my funds is this? Would building my own Linux cluster be 2x cheaper? 5x?

-What computer should i buy to get the most bang for my buck? I was looking at the "Silverdraft Power Resolve Workstation"... but not sure if I could stretch my dollars further with a different computer.

-Does this invite a security risk? I was once told by a friend that having such a powerful computer can attract the attention of hackers who would like to take control of the machine for mining bitcoin or whatever else.

Thanks for the help!

C


Looking at that machine that they want $33,500+Tax for, I decided to price out the components and add a few other items as well.









I can build pretty much that same computer, add two 31.5" 4K monitors, and faster ram for a quarter of the price they want. Buy another of the same case, more GPU's and another PSU and you still will not be even half their price.

ID: 1950662 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1950663 - Posted: 19 Aug 2018, 4:44:30 UTC - in response to Message 1950662.  

+1
ID: 1950663 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1950685 - Posted: 19 Aug 2018, 10:23:06 UTC
Last modified: 19 Aug 2018, 10:26:39 UTC

Excellent choises i only wish to add:

- Instead of the Air cooled EVGA 1080Ti, use Hybrids of the same model:

https://www.amazon.com/EVGA-GeForce-HYBRID-GAMING-Technology/dp/B074D7S8HR/ref=sr_1_2?ie=UTF8&qid=1534673406&sr=8-2&keywords=evga+hybrid+ftw3+1080ti&dpID=512khJjnmQL&preST=_SY300_QL70_&dpSrc=srch

They cost a little more but runs a lot cooler and with less air flow problems due the use of the 4 GPU's.
Their radiators fits perfect on the top of the Thermaltake X9 Cube making your entire system running a lot cooler.

or you can wait until the next week announcement of the RTX2080 series...... who are expected to be a lot faster GPU.

- You realy not need more than 16 GB to put this baby to crunch SETI, specialy on Linux (top SETI crunchers runs on Linux), so 64 GB are not realy necesary , unless you have another use for it.

- A Top SETI cruncher runs on Linux, so you not need to buy a Windows 10 unless you plan to use it for another use.

My 0.02
ID: 1950685 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1950690 - Posted: 19 Aug 2018, 10:56:03 UTC - in response to Message 1950685.  

I was going to say that 4 air cooled cards will never fit in 2.0 slot width, without water blocks, HyroCopper or AIO cards.

Full dual loop water cooling with 3 x 360 rads (or 4) would be nice and cool. But add ~4k to that bill.
ID: 1950690 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1950693 - Posted: 19 Aug 2018, 11:56:42 UTC - in response to Message 1950659.  

Hello,

I'd like to buy a "supercomputer" for the main purpose of contributing to research like SETI and medical projects. I'm also somewhat interested in using it for VR, because why not.

I have an approximate budget of 20-30k, and have a few questions:
---------------------------------cut-------------------------------------
-Does this invite a security risk? I was once told by a friend that having such a powerful computer can attract the attention of hackers who would like to take control of the machine for mining bitcoin or whatever else.

Thanks for the help!

C


The first thing I would like to ask is exactly which Medical Projects?

A purely Seti oriented machine would be heavily biased towards Gpu processing because we have really fast processing across multiple OS as well as things like the CUDA90 project aka: secret sauce.

Nearly all the "medical projects" I have looked at are CPU-based. So the more fast cores/hyper-threading you have the more production you would get.

You need to ask yourself if you are a "price is no object" person or if you are a "I want the most bang for the least buck" person.

This is a link from the Seti website to the top producing machines as well as the top producing individuals.
https://setiathome.berkeley.edu/stats.php

In the other responses you received a top end (price is no object) system was proposed.

You may find that used locked slower cpus with high core counts (8c/16t or 10c/20t or 12c/24 etc) will actually give you excellent results.

Used gtx 1080Ti's can be had on eBay for $500-$600.

Used e5-2670v1 2.6Ghz CPU's (8c/16t) can be had for $100~ on eBay Run them on dual CPU (server) ~$400 MB's (Please note, this is my "hobby" horse so I am offering biased opinion here).

My own experience with SSD has been a massive speed up in booting up speeds.

If you want it to run as cool as possible you may want to go with an "open air" Coin mining "case." You can buy them or get busy and make them out of wood. This would also allow you to run (probably) more than 4 gpus. They don't look especially pretty however.

You can also buy "turn key" machines like you previously mentioned. These have the benefit of you not having to assemble all the parts and some kind of warranty. There are "server" companies out there that will build a custom server to your specifications.
---------------------------------------
You asked about security and being a target for hackers etc. If you don't go around bragging about your Super Computer (or Super Computer Cluster) and you use adequate security firewalls or even "security appliances" you should be OK. A lot of trouble is caused by going to "bad" websites and gaining an infection from them.
---------------------------------------
VR is certainly interesting but you may find that your machine is just a little stressed when you try that at the same time you are Seti processing. Btw, you should NOT setup multiple gpus in "Crossfire" (I think that is one name for linking gpus for better gaming) for Seti processing. It is possible that VR benefits from "Crossfire" so you would have to choose. The "Crossfire" I am thinking of is purely hardware based with links between the gpus.

You may find yourself with high electrical bills.

HTH,
Tom
A proud member of the OFA (Old Farts Association).
ID: 1950693 · Report as offensive
C Horse

Send message
Joined: 25 Jul 18
Posts: 5
Credit: 10,361,509
RAC: 1
United States
Message 1950796 - Posted: 19 Aug 2018, 21:47:26 UTC

Thank you all so much for the help. I think i'll probably look into some local help assembling the pieces, since these computers can be built for 1/4 the price of of buying one pre-assembled one (as mentioned in the first response I received).

I'm going to do some research into who might be able to assemble something in my neighborhood. Thanks again

C
ID: 1950796 · Report as offensive
C Horse

Send message
Joined: 25 Jul 18
Posts: 5
Credit: 10,361,509
RAC: 1
United States
Message 1950798 - Posted: 19 Aug 2018, 21:59:48 UTC - in response to Message 1950693.  

I'm mostly interested in SETI. Running VR was an after thought, and it's ok if the machine isn't optimal for medical research.

I look for server companies like you mentioned. Hopefully they aren't as much of a rip off. This is the first one I found:

https://www.pssclabs.com/servers/hpc-servers/?gclid=Cj0KCQjwquTbBRCSARIsADzW88x1B9Xsg9Yaoi0ChnxpgNKKpBEKiO43qkBRWP-61_OeqeRsL1Koh_AaApHXEALw_wcB

If you were to build an ideal machine for SETI, that is likely in the neighborhood of 20-30k, what components would you choose? I'm hoping im not paying a 4x premium from these guys... I'm fine with paying a little extra to avoid building it, but 4x seems a bit much.

Thanks for the help!
ID: 1950798 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1950800 - Posted: 19 Aug 2018, 22:09:40 UTC - in response to Message 1950798.  

You could likely get 80% of what you need right from EVGA.com, but that Thremaltake X9 case is a goodie for a big build.
Also a good liquid cooler for a big CPU if you go huge. AIO coolers are fine, but not really designed for high power 24/7.
ID: 1950800 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1950804 - Posted: 19 Aug 2018, 22:58:13 UTC - in response to Message 1950798.  
Last modified: 19 Aug 2018, 23:01:33 UTC

I'm mostly interested in SETI. Running VR was an after thought, and it's ok if the machine isn't optimal for medical research.

I look for server companies like you mentioned. Hopefully they aren't as much of a rip off. This is the first one I found:

https://www.pssclabs.com/servers/hpc-servers/?gclid=Cj0KCQjwquTbBRCSARIsADzW88x1B9Xsg9Yaoi0ChnxpgNKKpBEKiO43qkBRWP-61_OeqeRsL1Koh_AaApHXEALw_wcB

If you were to build an ideal machine for SETI, that is likely in the neighborhood of 20-30k, what components would you choose? I'm hoping im not paying a 4x premium from these guys... I'm fine with paying a little extra to avoid building it, but 4x seems a bit much.

Thanks for the help!


How about multiple machines that are "a bit less than $20 a pop" and then buy/make several :)

I remember someone proposing a seti oriented box that had a fast i5 4 core no hyper-threading and 4 gtx 1080Ti's plus all the stuff you need like a big box, lots of cooling, big PSU etc. Possibly as little as 16 GB of memory? Basically it is recommend you use one core per video card in the NVidia world.

Now the 1080's would be running 3 tasks in parallel each. So that would give you 12 gpu-based processing routes.

To make it run "red hot" (aka really, really fast) you would need to be trying to run the "all in one Cuda" that Tbar has. I don't think it is bleeding edge being 2-3 minor updates behind but it has been shown to be easily installed under Lubuntu (that is what I am running and will be running for my bigger box once its fixed) and several other versions of Linux. If you are a skilled/experience Linux person you shouldn't have a problem running it (yes I saw Darwin).

I personally would prefer a lot of cores as well as a lot of gpu's. That way if I want to start doing much medical processing I have the cores to do it and if not, yes Seti CPU processing is still a major stream in the entire programs production.

I have see several 32 core machines running with no GPU at all ("headless"?). They were "suffering" but still generating respectable numbers.

I am guessing if you wanted to buy a single machine you either need to go with the top end Threadripper 2 MB cpu with 32c/64t or investigate the dual cpu MB for the very expensive/high end Intel cpus.

What is confusing is the really high count CPU's that Intel sells mostly live in "Server Racks" where they have no space for a GPU at all. The server company URL you posted has that issue. Its boxes don't have any place for a Gpu so you would need to see how many gpu slots they have and run the gpus via "riser" cables (not necessarily big flat ribbons).

Would it be possible to repurpose a Bit Coin Mining MB? ASUS has released one.
https://www.newegg.com/Product/Product.aspx?Item=N82E16813119028&ignorebbr=1&nm_mc=KNC-GoogleAdwords-PC&cm_mmc=KNC-GoogleAdwords-PC-_-pla-_-Motherboards+-+Intel-_-N82E16813119028&gclid=EAIaIQobChMInOiV8Zn63AIVELnACh0hFwjYEAYYASABEgJPRPD_BwE&gclsrc=aw.ds
I do wonder if you could really run Seti GPU processing using the 19 gpu MB from Asus. ( As I understand it, you can only run 16 gpus (8 Nvidia, 8 AMD/Ati) because of the driver limits. The cpu socket is late model so you could run a high end, fast, large core count intel cpu. But you might need to spread the gpus over more than one core.

Lets see. 6 X Gtx 1080 Ti X 3 parallel tasks = 18 parallel tasks. 6 X AMD Radeon (TM) R9 390 Series X 3? parallel tasks = 18 parallel tasks plus probably some CPU tasks.

Its enough to make you drool ;)

You might go with the original proposed system above except buy used Gtx 1080 Ti for near $500 rather than list price new ones at $1,000+

HTH,
Tom
A proud member of the OFA (Old Farts Association).
ID: 1950804 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 1950805 - Posted: 19 Aug 2018, 23:09:06 UTC
Last modified: 19 Aug 2018, 23:11:11 UTC

I was going to say that 4 air cooled cards will never fit in 2.0 slot width, without water blocks, HyroCopper or AIO cards.

Full dual loop water cooling with 3 x 360 rads (or 4) would be nice and cool. But add ~4k to that bill.


you can do it with all 4 cards in a single loop, with a single radiator.

Watercool MO-RA3 (either 360 [9x120] or 420 [9x140] version)
I'm using one to cool 2x1080ti and 2xE5-2697v2 12-core all in the same loop.

----------------------------------------------------------------------------------------------------------------------

but in response to the general replies of "build your own" here:

looks into used server hardware. server hardware is extremely reliable, and it can be had cheap on the secondary markets (ebay, reddit, etc) for CHEAP because of lack of demand.

if you want to run 4 cards plugged into a motherboard, you need water cooling ideally, or at the least some hybrid AIO cards. if you try to run anything more powerful than a 1050ti in a 4x config stacked next to one another, you will roast the cards. i have 2 systems with 2x1060s size by side and i have to use extremely powerful and load server exhaust fans + 80+% fan speed on the top card just to keep it below 80C. it will only get worse with more powerful cards.

if you really want to maximize SETI output from a single box, at maybe the expense of speed on other projects (depending how much they care about PCIe bandwidth), just run an open air rig, and put the cards on risers. you can get 6-8 cards on a single board. might need to run multiple power supplies and it wont look pretty.

i have no problems feeding SETI WUs all the way down to a PCIe x1 interface, even on a 1080ti.
like i said, other projects might care more about it, but SETI does not.

my most powerful machine is actually a system with

Linux OS + optimized SETI app
a single 4c/8t xeon e3 CPU
single stick of 8GB ram
cheap server case (but long) Rosewill RSV-L4500
1x1080ti
5x1060
mining risers carrying PCIe signals over USB 3.0 cable to a 1x interface
2x HP 750w server PSUs to power the cards and CPU 8-pin plug
1x 160w picoPSU to power the MB 24-pin
6x noctua iPPC 2000 fans to pull air across the GPUs

basically this: it does like 300,000 credits/day


Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 1950805 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1950816 - Posted: 20 Aug 2018, 0:14:21 UTC

I think Rosetta@Home is the only cpu only based medical distributed computing project still around. Folding@Home, World Community Grid and GPUGrid.net for example have gpu based applications as well as cpu. Gpu applications are much more productive on a watt per hour basis compared to cpu unless you factor in low power computing on Raspberry Pi type of hardware. So I only can see a mega core cruncher being useful at Rosetta while a gpu based cruncher would be useful at many more medical focused projects.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1950816 · Report as offensive
mmonnin
Volunteer tester

Send message
Joined: 8 Jun 17
Posts: 58
Credit: 10,176,849
RAC: 0
United States
Message 1950822 - Posted: 20 Aug 2018, 0:22:09 UTC - in response to Message 1950816.  

I think Rosetta@Home is the only cpu only based medical distributed computing project still around. Folding@Home, World Community Grid and GPUGrid.net for example have gpu based applications as well as cpu. Gpu applications are much more productive on a watt per hour basis compared to cpu unless you factor in low power computing on Raspberry Pi type of hardware. So I only can see a mega core cruncher being useful at Rosetta while a gpu based cruncher would be useful at many more medical focused projects.


WCG does not have a GPU app. Years ago it did but not any more.
ID: 1950822 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1950853 - Posted: 20 Aug 2018, 4:10:31 UTC - in response to Message 1950798.  

If you were to build an ideal machine for SETI, that is likely in the neighborhood of 20-30k, what components would you choose?

That would build up to 10 high performance systems. No single system is worth that much money unless it is being used in a data centre with very specific duties.
Grant
Darwin NT
ID: 1950853 · Report as offensive
Profile tullio
Volunteer tester

Send message
Joined: 9 Apr 04
Posts: 8797
Credit: 2,930,782
RAC: 1
Italy
Message 1950866 - Posted: 20 Aug 2018, 8:21:18 UTC - in response to Message 1950693.  

GPUGRID is a medical project and makes use of GPUs (nVidia) both in Windows and Linux. Besides this,it runs Computer Chemistry tasks on CPUs, only in Linux.
Tullio
ID: 1950866 · Report as offensive
Ghia
Avatar

Send message
Joined: 7 Feb 17
Posts: 238
Credit: 28,911,438
RAC: 50
Norway
Message 1950873 - Posted: 20 Aug 2018, 10:03:17 UTC

Wouldn't a NVMe disk (or two) be better than SATA ?
The EVGA X299 Dark has room for 2 of them...I know because I have that mobo.
And btw...would you recommend 1 large disk, or 2 smaller ? Like 1 for the OS and one for apps ?
Humans may rule the world...but bacteria run it...
ID: 1950873 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1950881 - Posted: 20 Aug 2018, 12:06:06 UTC - in response to Message 1950873.  

Wouldn't a NVMe disk (or two) be better than SATA ?
The EVGA X299 Dark has room for 2 of them...I know because I have that mobo.
And btw...would you recommend 1 large disk, or 2 smaller ? Like 1 for the OS and one for apps ?


When I can afford it, I like to run paired disks on Raid1. Then when one dies its less trouble to "rebuild" the drive.

Tom
A proud member of the OFA (Old Farts Association).
ID: 1950881 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1950882 - Posted: 20 Aug 2018, 12:24:30 UTC - in response to Message 1950663.  

+1

A proud member of the OFA (Old Farts Association).
ID: 1950882 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1950883 - Posted: 20 Aug 2018, 12:30:51 UTC - in response to Message 1950805.  

basically this: it does like 300,000 credits/day



Thank you for the picture.

Now I get it. That "partition" in the middle of the case is the fans "drawing air from the gpus" that you referenced.

It must have hit a "bump" its doing "only" 150,000+- RAC at the moment ;)

And its not listing the Gtx 1080Ti at all?


Tom
A proud member of the OFA (Old Farts Association).
ID: 1950883 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1950884 - Posted: 20 Aug 2018, 12:31:30 UTC - in response to Message 1950853.  

If you were to build an ideal machine for SETI, that is likely in the neighborhood of 20-30k, what components would you choose?

That would build up to 10 high performance systems. No single system is worth that much money unless it is being used in a data centre with very specific duties.



+1
A proud member of the OFA (Old Farts Association).
ID: 1950884 · Report as offensive
1 · 2 · 3 · Next

Message boards : Number crunching : Buying a Supercomputer?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.