New rig project

Message boards : Number crunching : New rig project
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Profile Akio
Avatar

Send message
Joined: 18 May 11
Posts: 375
Credit: 32,129,242
RAC: 0
United States
Message 1759489 - Posted: 26 Jan 2016, 22:20:37 UTC
Last modified: 26 Jan 2016, 22:22:16 UTC

Well, I'm getting ready to make the leap into building a new 24/7 rig by planning out all the hardware. Zalster has been and continues to be extremely helpful in this process due to my limited build experience; in fact, Zalster has given me a ton of knowledge on newer hardware that I had little of before.

That being said, I'd like to share my Working New Rig hardware list to get some more feedback from you guys too.

Please check it out and let me know what you guys think so far.

(Note: in the list you will notice I have an M.2 2280 120GB SATA SSD that I'd like to use *only* for Windows / booting. I'm toying with this idea.) Overkill? ...Thoughts on utilizing M.2?

Thanks for your help, folks.

- Akio
ID: 1759489 · Report as offensive
woohoo
Volunteer tester

Send message
Joined: 30 Oct 13
Posts: 972
Credit: 165,671,404
RAC: 5
United States
Message 1759507 - Posted: 26 Jan 2016, 23:52:21 UTC

Although I am a big fan of SSD and I use them, they don't help your RAC. If you were going to go the m.2 route, I think the 850 EVO is a waste because it tops out at regular sata speeds. An sm951 or 950 pro would be a better pick. Also m.2 would use pci-express bandwidth, I didn't bother researching whether using it in conjunction with a four pack of video cards would cause a bottleneck or not.

The video card that I like is GIGABYTE GeForce GTX 980Ti 6GB XTREME GAMING WATERFORCE
Theoretically if you sandwich four of these cards together you might not have to worry about three blocked fans.

The case I like is Thermaltake Core X9 Black E-ATX Stackable Tt LCS Certified Cube Chassis CA-1D8-00F1WN-00
It doesn't cost that much and it seems to have a lot of room.
ID: 1759507 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1759522 - Posted: 27 Jan 2016, 0:55:09 UTC - in response to Message 1759507.  
Last modified: 27 Jan 2016, 1:08:58 UTC

I'm going to confess to not seeing the M.2 when I saw the list earlier.

I'm not a fan as of yet. There is also the issue of support for it with that chip.

Support won't come until the next generation of Intel code name Broadwell as seen in this article

Intel's new Broadwell CPUs and their chipsets include native support for M.2 and PCI Express boot drivers—


The following comes from ASrock webpage on this board.


1 x M.2_SSD (NGFF) Socket 3, supports M.2 SATA3 6.0 Gb/s module and M.2 PCI Express module up to Gen2 x4 (20 Gb/s)****
****If M2_1 is occupied by a SATA-type M.2 device, S_SATA3_0 will be disabled. If you install a M.2 PCI Express module and the SATA_EXP0 is in use, the M2_1 will downgrade to x2 mode.


If people choice to use a SSD, that is fine,I have used either SSD or HDD in some of my machines. I think it's a personal choice.
ID: 1759522 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1759529 - Posted: 27 Jan 2016, 1:20:31 UTC - in response to Message 1759489.  

Well, I'm getting ready to make the leap into building a new 24/7 rig by planning out all the hardware. Zalster has been and continues to be extremely helpful in this process due to my limited build experience; in fact, Zalster has given me a ton of knowledge on newer hardware that I had little of before.

That being said, I'd like to share my Working New Rig hardware list to get some more feedback from you guys too.

Please check it out and let me know what you guys think so far.

(Note: in the list you will notice I have an M.2 2280 120GB SATA SSD that I'd like to use *only* for Windows / booting. I'm toying with this idea.) Overkill? ...Thoughts on utilizing M.2?

Thanks for your help, folks.

- Akio

If the machine is going to be a dedicated cruncher the SSD is overkill considering you are also including two 1TB drives. I'm guessing they will be RAID1?
If you planning to use RAID1 then you could configure a 100GB volume in the RAID for the OS. Then have the remainder of the 1TB for storage.
For a dedicated cruncher either 2 SSDs or two HDDs in RAID1 seems like an ideal configuration. That way a drive can die & you don't have to start all over.

In my HTPC I use a 120GB SSD for a boot drive. I have ~100GB partitioned for the OS & apps. Then the remaining 10GB I have partitioned as a temp drive. Which is also where I put my BOINC data. That machine also has 3 2.5" 1TB drives in RAID5 for my media storage. However I have chosen not to have my BOINC data there so that the drives will run in their low power state more often. To save a few watt hours.

Otherwise depending on what else you are planning to do with the machine it should be more than ample for SETI@home use.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1759529 · Report as offensive
Profile ML1
Volunteer moderator
Volunteer tester

Send message
Joined: 25 Nov 01
Posts: 20140
Credit: 7,508,002
RAC: 20
United Kingdom
Message 1759534 - Posted: 27 Jan 2016, 1:26:14 UTC - in response to Message 1759522.  
Last modified: 27 Jan 2016, 1:28:42 UTC

I'm going to confess to not seeing the M.2 when I saw the list earlier.

I'm not a fan as of yet. There is also the issue of support for it with that chip.

Support won't come until the next generation of Intel code name Broadwell as seen in this article

Intel's new Broadwell CPUs and their chipsets include native support for M.2 and PCI Express boot drivers—


The following comes from ASrock webpage on this board.


1 x M.2_SSD (NGFF) Socket 3, supports M.2 SATA3 6.0 Gb/s module and M.2 PCI Express module up to Gen2 x4 (20 Gb/s)****
****If M2_1 is occupied by a SATA-type M.2 device, S_SATA3_0 will be disabled. If you install a M.2 PCI Express module and the SATA_EXP0 is in use, the M2_1 will downgrade to x2 mode.


If people choice to use a SSD, that is fine,I have used either SSD or HDD in some of my machines. I think it's a personal choice.

Minor brag:

An M2 is shown tested on Linux. Note the special format connector needed on the motherboard...


Quite a big power mix there for your specs. Excellent.

I'll agree with the water cooling for the GPUs. Also water cooling is a good idea for the CPU. Then you just need to avoid dust choking your cooling radiator (easier to clean than GPUs!). But... Takes longer to assemble and set up. Aside and to state the obvious, you will still need some airflow to cool the motherboard and disks.

Putting the OS on a separate drive is always a good idea. If you have enough RAM, you can gain a small speedup for s@h by using a ram drive (Linux tmpfs) for Boinc.


Let us know when online :-)

Happy fast crunchin'!
Martin
See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)
ID: 1759534 · Report as offensive
Profile Louis Loria II
Volunteer tester
Avatar

Send message
Joined: 20 Oct 03
Posts: 259
Credit: 9,208,040
RAC: 24
United States
Message 1759536 - Posted: 27 Jan 2016, 1:42:31 UTC - in response to Message 1759534.  

Putting the OS on a separate drive is always a good idea. If you have enough RAM, you can gain a small speedup for s@h by using a ram drive (Linux tmpfs) for Boinc.


Let us know when online :-)

Happy fast crunchin'!
Martin[/quote]

RAM Drive...that's a term I hadn't heard in a very long time... I am a SSD convert myself. The load time for applications is amazing, I don't wait for long. The RAM drive is interesting though, but can it really help? Even with an image to be loaded on boot (because RAM is volatile of course) it still needs the CPU and GPU for compute operations. Now you have me curious. With 16gigs of RAM though, I could try this....

Peace and happy crunching!
ID: 1759536 · Report as offensive
Profile Cactus Bob
Avatar

Send message
Joined: 19 May 99
Posts: 209
Credit: 10,924,287
RAC: 29
Canada
Message 1759537 - Posted: 27 Jan 2016, 1:44:10 UTC

I agree with the other posters on the M.2 card. You could get a .5 Gig M.Sata card for the same price and end up with the same speed.

About 60% of you costs are your GPU's, this will give you an awesome RAC. If that is your goal than your good to go. The I7 is needed if your doing the 4 GPUS but knock that back to 2 and you could use an I5. That alone would probably knock about $2.000 off the sticker price.

Bottom line is what your budget is. and if you want to shave any cost off your list.

Bob
Sometimes I wonder, what happened to all the people I gave directions to?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
SETI@home classic workunits 4,321
SETI@home classic CPU time 22,169 hours
ID: 1759537 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 34744
Credit: 261,360,520
RAC: 489
Australia
Message 1759579 - Posted: 27 Jan 2016, 5:24:58 UTC
Last modified: 27 Jan 2016, 5:26:00 UTC

Just in case you go with those drives.

Those WD Blue drives only have a 2yr warranty on them, shy not pay a little bit more for the Black versions as they carry a 5yr warranty and are a little faster. ;-)

I've just recently did a did a very rare RMA on a Black drive that was 4yrs & 10mths old (it spent its life under heavy use). :-)

Cheers.
ID: 1759579 · Report as offensive
Profile Tuna Ertemalp
Volunteer tester
Avatar

Send message
Joined: 15 Nov 99
Posts: 18
Credit: 74,971,084
RAC: 0
United States
Message 1759601 - Posted: 27 Jan 2016, 6:47:23 UTC
Last modified: 27 Jan 2016, 7:11:10 UTC

I have recently built two machines, one with Quad Titan X, one with Dual Titan Z (specifically for PrimeGrid given the Double Precision prowess of the Z). Both 24/7 BOINC machines. So, I'll tell you what I used in the Quad X:

ASUS X99-E WS/USB 3.1 Motherboard. 4-way, all Gen3 x16, M.2 doesn't share anything with PCI, nor is limited by anything.

SSD 950 PRO M.2 512G boot drive. I am extremely happy. For a BOINC-only machine, that should be enough storage for OS+BOINC Data and way more.

SSD 850 PRO 1T SSDs, if you need more storage, as many as you need (board allows up to 12), and then use Windows 10 Storage Spaces to set them up as parity or striping. Don't use the onboard RAID since you'll lose the TRIM functionality on the SSD. These guys are beasts, never had one fail, they are in all of my machines.

Fo optical, I used LG Internal SATA 16x Super Multi Blue WH16NS40 and put it into an external enclosure (VANTEC ST-530S3-BK Aluminum 5.25" SATA USB 3.0 External 5.25" Optical Drive Enclosure USB 3.0 External Interface). Why external? Read on to see my comments about airflow.

For GPUs, I prefer EVGA. I get them individually sealed water cooled, resulting in as many radiators as GPUs. For EVGA, that is either 'EVGA GeForce GTX TITAN X HYBRID GAMING' or 'EVGA GeForce GTX TITAN X SC GAMING' combined with 'EVGA All-in-One Hybrid Water Cooler for Select EVGA GTX TITAN X Graphics Cards' plus some elbow grease.

For CPU, I love Intel® Core™ i7-5960X Processor Extreme Edition (20M Cache, up to 3.50 GHz). 8 cores, 16 threads, and I was able to push it to 3.9GHz.

To cool that, I use CORSAIR Hydro Series H110i GT Extreme Performance Water/Liquid CPU Cooler.

To accommodate all these 4+2=6 fans to exhaust, plus accommodate further intake and exhaust fans, I use Thermaltake Core x9 case. It is big, but has lots of space for anything you might want to do in there. I get two 200 fans as intake for the front, a 140 in the back for exhaust, plus the 4+2=6 fans exhausting at the top. Also the GPUs end up exhausting in the back with their on-card fans in addition to the liquid loop exhausting at the top of the case. And, the PSU also sucks in internal air and exhausts in the back. I removed as much of the stuff as possible inside, like many of the harddisk cages, put the SSDs at the bottom of the case, all in the name of unblocking the airflow in the case, especially that from the front intakes.

I maxed the RAM at 64G (max for that CPU) to be able to use "Keep projects in memory while switching" since I churn for 50 projects. What works with that MoBo and is really good and can be overclocked by the ASUS board automatically: Corsair Dominator® Platinum Series 64GB (8 x 8GB) DDR4 DRAM 2400MHz C14 Memory Kit (CMD64GX4M8A2400C14)

Here is the twist: When I build such an expensive top gear, I want to put it on a UPS, something from APC, to avoid hurting it through flickers and brownouts. Unfortunately, at max CPU+GPU use this beast draws about 1.2KwH. The 1500VA UPS units max out at about 860w. So, I would have needed to get a 2200VA unit which are really expensive and incredibly heavy (i know, because I got one for the dual Titan Z rig, and my jaw fell to the floor when I saw the huge wooden crate FedEx delivered). So, instead, given the case has spots for two PSUs, I got one Antec HCP-1300 Platinum 1300W and one Antec HCP-1000 Platinum 1000W. I put GPUs on 1300w, everything else (MoBo, fans, SSDs, etc.) on 1000. Then I was able to get two 1500VA UPS boxes, put one between the wall and each PSU; cheaper, lighter, plenty room. Both UPS show about 600-650w draw, well below their peak capacity, and well balanced. And, I used Antec since they come with an OC-link cable to link both PSUs so that your On/Off button on the case will turn on/off both PSUs at the same time, and do other things that two PSUs need to agree on before things on your motherboard start acting up due to getting powered by different PSUs. In other words, you connect them, and you have 2300w; the rest just works. When I first had them on one EVGA 1600 (which I later moved to the Titan Z machine), even though the math works out, with all the fans & overclocking, GPUs were getting limited by power/voltage etc. Now, they have all the juice they need even when they are pushing it.

I am also using MSI Afterburner 4.2 software on the machines to watch the GPU and CPU temps, as well as create custom on-GPU fan profiles. I don't actively push for overclocking. At stock, they are plenty fast. But, this whole hardware sort of auto-overclocks until it reaches some thermal/power limit. When that happens, MoBo and GPU fans become important. The factory settings are such that GPU fans don't really exceed 40%. That is nothing when the heat reaches 80C and your cards are thermal limited. In another of my Dual Titan X machines that is not liquid cooled, just by applying the fan profile to kick in high gear at 60-70C, the machine is now running at 50-60C instead of 80-85C, getting more calculation cycles done. Of course, it is louder, and the room is much warmer. :)

Why did I use Titan X instead of 980 Ti? First, just wanted to. Second, and more importantly, wanted this machine to be future proof in case some project started really using the GPU memory, given that I churn for all BOINC projects, including ones that'll come along in the future. So far, that was a waste of money. Max I saw in MSI Afterburner was about 2-3GB use of the GPU memory by some project; probably PrimeGrid. So, 980 Ti with 6G memory should be more than enough, and save you bunch of money, taking the sting away.

Side note: Titan X is able to do 4K with 4:4:4. A good "monitor" to take full advantage of that is actually a TV! I got a Samsung UN40JU6700 Curved 40-Inch 4K Ultra HD Smart LED TV (2015 Model) from Amazon which was an amazing deal in December at $650, including a BluRay player to boot! I am very very happy with it.

Good luck!
Tuna

Disclaimer: No, I am not a hardware geek. If you start drilling into me with detailed questions, I might not be able to say more than above. But, I have a friend who really knows this stuff and enjoys building things for his friends, so I learned quite a bit from him over the last few months. Still, my knowledge is very superficial compared to what he knows. A lot of the above is him holding my hand through the hardware jungle...
ID: 1759601 · Report as offensive
Profile Akio
Avatar

Send message
Joined: 18 May 11
Posts: 375
Credit: 32,129,242
RAC: 0
United States
Message 1759737 - Posted: 27 Jan 2016, 18:15:54 UTC

I appreciate you guys' input! You have certainly given me a lot to think about over the next couple of weeks. Please continue to comment on this while I'm still in planning mode ;)

I suppose you guys are right about the M.2 idea. I am probably looking at getting the Samsung 850 Pro 128GB SATA SSD for my boot drive and using a standard 1TB SATA drive for storage. The latter is cheap and easy to get :)

I'm going to forgo purchasing a CD/DVD drive of any kind and use my external one I have for installing the OS and all that during setup. I really have no need for a permanently installed one.

Other than that, I'll start with the two GPU's to get the build together, and then get the other two down the road when funds are replenished.

Updated part list. Still in the planning stages while I mull over other hardware.
ID: 1759737 · Report as offensive
Profile Todderbert
Avatar

Send message
Joined: 17 Jun 99
Posts: 221
Credit: 53,153,779
RAC: 0
United States
Message 1760256 - Posted: 29 Jan 2016, 5:50:58 UTC
Last modified: 29 Jan 2016, 6:04:33 UTC

I built a similar rig using a CaseLabs Mercury Case, its amazing. Tons of room for the hybrid coolers, great support for the gpus, and its made from thick aluminum. If your going to spend 300+ for a case, check it out.

http://setiathome.berkeley.edu/forum_thread.php?id=73909&postid=1757218
ID: 1760256 · Report as offensive
Profile Akio
Avatar

Send message
Joined: 18 May 11
Posts: 375
Credit: 32,129,242
RAC: 0
United States
Message 1762442 - Posted: 5 Feb 2016, 12:31:27 UTC - in response to Message 1760256.  

I built a similar rig using a CaseLabs Mercury Case, its amazing. Tons of room for the hybrid coolers, great support for the gpus, and its made from thick aluminum.


That case is sweet! After checking the pricing for it I was bummed, as it put me over my starting budget. It would be an amazing case to have though, but I had to stay with the Corsair 900D for this project.

I've had the new rig up and running GPU tasks for about five days or so, and so far all seems to be running well. I'll be adding some CPU tasks soon too. So far I'm pretty happy with the two hybrid 980 Ti's; they stay pretty cool under load and are very quiet. I'm already saving up for the last two, but that will take a very long while. I'll be having to tweak the app_config a little more for optimal performance.

The rig went together better and faster than I expected. It makes a huge difference when you're working with a tower that in no way lacks room to move your hands and fingers around in. I'm pretty happy so far with how it's turned out. Come to find out I actually like putting rigs together; it might be worth looking into for pay.
ID: 1762442 · Report as offensive
Profile cybrdaze

Send message
Joined: 26 Sep 11
Posts: 6
Credit: 14,266,184
RAC: 0
United States
Message 1762456 - Posted: 5 Feb 2016, 13:32:41 UTC - in response to Message 1762442.  

Hey Guys,

Just started to make a few purchases on a new rig when I came across this thread. My build is very close to this one with a few changes. Could you please take a look and give me your recommendations.

https://pcpartpicker.com/user/kdicks/saved/#view=jFYkcf

Thanks,
Kevin
ID: 1762456 · Report as offensive
Profile Akio
Avatar

Send message
Joined: 18 May 11
Posts: 375
Credit: 32,129,242
RAC: 0
United States
Message 1762459 - Posted: 5 Feb 2016, 13:54:28 UTC - in response to Message 1762456.  

Hey Guys,

Just started to make a few purchases on a new rig when I came across this thread. My build is very close to this one with a few changes. Could you please take a look and give me your recommendations.

https://pcpartpicker.com/user/kdicks/saved/#view=jFYkcf

Thanks,
Kevin



I like it overall, Kev. That's a lot of memory, though. And I'd be a little worried about the airflow restrictions between those graphics cards all sandwiched together on the MB (an observation noted by a poster in regards to my build too, and worth pondering. See posts above).
ID: 1762459 · Report as offensive
Profile cybrdaze

Send message
Joined: 26 Sep 11
Posts: 6
Credit: 14,266,184
RAC: 0
United States
Message 1762465 - Posted: 5 Feb 2016, 14:27:08 UTC - in response to Message 1762459.  

Thanks. Not planning to purchase the RAM all at once, just what is needed. Will likely start with 16 GB. The whole build is going to be piece by piece.

I like the EVGA Hybrid cards but, as an engineer, I do not believe that the sealed system will remain maintenance free for the life of the card nor do I want to keep them full time crunching on the small stock radiators. When the build reaches the 3rd card I plan to water cool.

I read your new build is up and running. Are you performing a slow burn-in on the parts?

Thanks,
Kevin
ID: 1762465 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22158
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1762466 - Posted: 5 Feb 2016, 14:30:36 UTC

CPU cooling is not as much an issue as keeping those four '980TIs cool - I would look at the cost of liquid cooling them as the centre two will almost certainly not get enough cool air for their needs.

Its a shame Asus don't do a '980TI version of their '970 with "through the board" cooling (http://www.asus.com/uk/Graphics-Cards/TURBO-GTX960-OC-4GD5/) as they run pretty cool, even those in the middle of a stack get plenty of air.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1762466 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1762490 - Posted: 5 Feb 2016, 15:40:45 UTC - in response to Message 1762456.  

Hey Guys,

Just started to make a few purchases on a new rig when I came across this thread. My build is very close to this one with a few changes. Could you please take a look and give me your recommendations.

https://pcpartpicker.com/user/kdicks/saved/#view=jFYkcf

Thanks,
Kevin



I agree that is a lot of RAM, you could go with a lower amount, say 16 and build up if you need it.

But I doubt you would need much more.

The main Issue I see is the cards themselves.

There is no way you will get all of those cards onto that board.

Those ASUS cards are too thick to be stuck in 4 way SLI

They are double and a half slot cards. Meaning, they will pushing on the card above and below them, so you will not be able to make them fit into the slots on the boards.

You could do it if you only run 2 cards and make sure to skip the second PCIe slot and go directly to the 3rd slot.

If you are looking for a 4 GPU machine, then you need to look at a different card.



Zalster
ID: 1762490 · Report as offensive
Profile cybrdaze

Send message
Joined: 26 Sep 11
Posts: 6
Credit: 14,266,184
RAC: 0
United States
Message 1762502 - Posted: 5 Feb 2016, 16:01:22 UTC

Thanks, made some changes.


Kevin
ID: 1762502 · Report as offensive
Profile Angela Special Project $75 donor
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 16 Oct 07
Posts: 13130
Credit: 39,854,104
RAC: 31
United States
Message 1789964 - Posted: 24 May 2016, 5:48:55 UTC

Thread unlocked at originator's request.
ID: 1789964 · Report as offensive
Profile Cactus Bob
Avatar

Send message
Joined: 19 May 99
Posts: 209
Credit: 10,924,287
RAC: 29
Canada
Message 1789972 - Posted: 24 May 2016, 6:32:16 UTC

I just saw this was unlocked. Have you built this rig yet Akio?

The money for the 980's might be better spent on 1080's or 1070's now. A lot has changed since you posted your picker part list.

Curious to see what you built (or will build)

Bob
Sometimes I wonder, what happened to all the people I gave directions to?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
SETI@home classic workunits 4,321
SETI@home classic CPU time 22,169 hours
ID: 1789972 · Report as offensive
1 · 2 · Next

Message boards : Number crunching : New rig project


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.