Okay. I have all the pieces ordered. So now I can go back to trying to sell a TR4 MB, a case and a cpu cooler while I am waiting for the pieces to show up.

I already have a workbench/table setup and I have ordered another KVM keyboard/monitor switch so I can run this in a different room with a different power circuit. :) I already have a 2400G in there so I have to share the monitor/keyboard/mouse.

Tom

Some of the parts have trickled in. I have a MB but the cpu is still out there someplace. And the "Miners Rig" frame seems to have but put on a slow boat errr.... truck.

Tom

---

A proud member of the OFA (Old Farts Association).

Okay. I have all the pieces ordered. So now I can go back to trying to sell a TR4 MB, a case and a cpu cooler while I am waiting for the pieces to show up.

I already have a workbench/table setup and I have ordered another KVM keyboard/monitor switch so I can run this in a different room with a different power circuit. :) I already have a 2400G in there so I have to share the monitor/keyboard/mouse.

Tom

Some of the parts have trickled in. I have a MB but the cpu is still out there someplace. And the "Miners Rig" frame seems to have but put on a slow boat errr.... truck.

Tom

Well I have some good news and some bad news. The good news is I have the cpu, the motherboard, PSU and lots of video cards.

The bad news is the cpu won't boot. The MB diagnostic light says the cpu is not functioning properly. And the LED that displays the bio codes keeps cycling through the same 4-5 different codes. :(

I have a working AM4 MB and a working AM4 cpu but I am very reluctant to take them apart so I can figure out which "other" part is broken. The MB or the cpu.

The cpu was "new(other)" and showed up in a sealed package. The MB was used but presumably working. :(((((

Tom

---

A proud member of the OFA (Old Farts Association).

Well IMO, used motherboards are about the biggest gamble you can make. If you buy one from an alleged computer dealer your chances are much better that it will at least boot. If the price is right I will buy a used board, from a computer dealer. For the used CPUs it's pretty safe to buy one of the Locked versions, at least you know someone hasn't tried to clock a Locked CPU to the moon just to see where it crashed. The best choice for me is a used, Locked, CPU and a New board. I did chance a used board recently, just because the new version was still way too high. This is what I thought I bought;

USED ASUS Z170-WS Z170 Socket LGA 1151 DDR4 ATX Motherboard
Condition: Used
“Used condition. Sold as is. NO ACCESSORIES INCLUDED”

I thought I'd get a working board in bubble wrap and if lucky, maybe a backplate as well. As long as it worked, I would chance $70.
What I got, from an alleged computer dealer, was a newish box, complete with ALL the ACCESSORIES in unopened packages, and a packing slip that said....ASUS Z170 WS Refurbished Motherboard. Not quite what was advertised. I Never buy Refurbished simply because for the most part the board is too complicated to be fixed when something goes wrong, and a working used board is usually better. Well, the thing works for the most part, with just a few quirks that could have been reason for someone paying MSR to return it. I'm not sure if I should be happy or mad, but, the only one of those accessories I've used was the backplate...and the rather large manual.
We'll see how it works out. For now it's running as a Mac, and just moved up to number 13.

I'm cleaning off the heat paste. Will try again after that. It is possible that I just need to wait longer :( I thought I did but maybe this is as bad as my threadripper.

Otherwise, I suppose I could buy a new cheaper AM4 cpu and see if MB will boot for that or else buy a new MB of same make/model.

Tom

---

A proud member of the OFA (Old Farts Association).

I'm cleaning off the heat paste. Will try again after that. It is possible that I just need to wait longer :( I thought I did but maybe this is as bad as my threadripper.

Otherwise, I suppose I could buy a new cheaper AM4 cpu and see if MB will boot for that or else buy a new MB of same make/model.

Tom

Well that sucks. Now the MB is completely dead. It won't light up the power switch on the board with two different PSU's. And with or without the cpu.

I left Gigabyte a message asking for a RMA. Notified the seller that it had quit early.
My memory on it was wrong. It was listed as "New(other)" which means the box had been opened but nothing had been touched.

:(

What other AM4 Motherboards have been running reliably with upto 4 gpus on them?

Just got a retention bonus so maybe I can afford a different MB.

Tom

---

A proud member of the OFA (Old Farts Association).

I have two more AM4 motherboards on the way. One has 5 slots, the other has 6 (crossing fingers). I also ordered another AM4 cpu so I will actually have enough cpus to run all my AM4 motherboards.

So probably next week sometime I will be able to post the good/bad news. I am guessing that I might manage 5 gpus based on my Intel experience. But I really would like to know if it is only the Threadrippers who have enough pcie resources to run a lot of gpus.

Then I suppose I will have to breakdown and buy one of the recommended TR4 MB's and the cheapest Threadripper CPU I can find for the next experiment.

Tom

---

A proud member of the OFA (Old Farts Association).

Ah, Ian&SteveC pointed me to a very high count pcie lane dual cpu server motherboard that I could reuse most of my Lga 2011 hardware on (that is the e5-2690v2/gtx 1070ti box. Maybe I could hang all my gtx 1060 3GB's off it along with my gtx 1070ti's :)

Its only $500 or so for a non-standard MB (have to go to an open air setup). His is on the way so I am waiting with baited breath to see how many gpus he can really hang off of it.

One experiment at a time.
I think I may get a least one new AM4 MB on monday X (fingers crossed).

Tom

---

A proud member of the OFA (Old Farts Association).

it should be able to do 11+ (only more than 11 if you use splitters or Dual GPU cards)

I'm moving ahead on this system, the only thing i'm waiting on at the moment is a bit of acrylic to mount the motherboard to the open air frame. and i'll need to modify the open air rig to support some more GPUs. also this motherboard uses Narrow-ILM socket, so you cannot use your existing CPU heatsink coolers that use the Square-ILM standard. I have brackets that came with my Supermicro CPU heatsinks that i can use to convert the Square-ILM to Narrow-ILM, but you will need to see if that is possible for whatever cooler you have, or just get new ones.

the only "unknown" at the moment, will be the video output. the BIOS supposedly has provisions for video out via a PCIe card, but with a lack of any PCIe x16 slots, i'm not sure which one will output the video. I will have to test it and see.

While the board does have Matrox onboard VGA, I'm fairly confident that i cannot use this for video out while having nvidia drivers installed to use the GPUs for compute work. i think there is some conflict in doing that.

---

Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

it should be able to do 11+ (only more than 11 if you use splitters or Dual GPU cards)

I'm moving ahead on this system, the only thing i'm waiting on at the moment is a bit of acrylic to mount the motherboard to the open air frame. and i'll need to modify the open air rig to support some more GPUs. also this motherboard uses Narrow-ILM socket, so you cannot use your existing CPU heatsink coolers that use the Square-ILM standard. I have brackets that came with my Supermicro CPU heatsinks that i can use to convert the Square-ILM to Narrow-ILM, but you will need to see if that is possible for whatever cooler you have, or just get new ones.

the only "unknown" at the moment, will be the video output. the BIOS supposedly has provisions for video out via a PCIe card, but with a lack of any PCIe x16 slots, i'm not sure which one will output the video. I will have to test it and see.

While the board does have Matrox onboard VGA, I'm fairly confident that i cannot use this for video out while having nvidia drivers installed to use the GPUs for compute work. i think there is some conflict in doing that.

I have a matrox gpu on my biggest box. The screen update is extermly slow. I don't recommend it at all :/

Try the card in the slot at the "bottom" first. I have consistently had the signal show up on the last card in my x16 slots so I am betting the card at the bottom is likely.

The limited test I did using the quad pcei to gpu cards I have didn't appear to slow down at all, as long as I was using the -nobs parameter. The same was true with my 2990wx. I could run all cpus (or nearly so) and the cpu processing would slow way down. But is long as I had -nobs on the gpu command line the gpus kept perking along.

Might or might not work.

Tom

Tom

---

A proud member of the OFA (Old Farts Association).

All this leads to the question of how many pcie lanes does a gpu use?

I can argue for 2, one for data and one for "control". Maybe 2 for data and 1 for control.

Anyone got a URL with a lead on this?

Tom

---

A proud member of the OFA (Old Farts Association).

All this leads to the question of how many pcie lanes does a gpu use?

I can argue for 2, one for data and one for "control". Maybe 2 for data and 1 for control.

Anyone got a URL with a lead on this?

Tom

A read at Wikipedia on the PCIe standard will answer your question.
https://en.wikipedia.org/wiki/PCI_Express

---

Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)

All this leads to the question of how many pcie lanes does a gpu use?

I can argue for 2, one for data and one for "control". Maybe 2 for data and 1 for control.

Anyone got a URL with a lead on this?

Tom

A read at Wikipedia on the PCIe standard will answer your question.
https://en.wikipedia.org/wiki/PCI_Express

So each slot has its own point to point wiring to wherever the "head" is? Or is this "logical"?

Why is it possible to run out of PCIe "resources when not all slots are populated? Can a bios upgrade increase the number of PCIe resources? Or are the PCIe resources strictly a hardware issue?

Tom

---

A proud member of the OFA (Old Farts Association).

Mostly hardware

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

The infrastructure is hardware limited to how many PCIe lanes it can service between the CPU and the PCH. That is why server boards have PLX chips to increase the number of PCIe lanes the motherboard can service.

---

Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)

Maybe the plx chips are more common on single socket boards, but all of the dual socket server boards I’ve ever used have a plethora of available PCIe lanes (80+) direct from the CPUs.

---

Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

I have been reading about "insufficient pcie resources" and apparently you need to enable the "above 4G"? option. My biggest box had it set to auto. And the other idea was to disable all the Sata devices you are not using.

Sounds like another test for the big box. It would be nice if I could run more than three gpu's :)

Tom

---

A proud member of the OFA (Old Farts Association).

above 4g decoding setting is always beneficial for multi-gpu setups. but that setting has to do with memory addressing, it doesnt change anything with pcie resources available.

---

Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

OK, my misnomer then. My workstation board I guess is not a server board per se. And it is single socket. Two PLX chips plus the CPU and the PCH to give X16 lanes to all slots.
https://www.newegg.com/Product/Product.aspx?Item=N82E16813132901

---

Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)

you can only get full 16x to slots 1,3,5,7. which is likely where you have all your cards plugged in.

slots 2,4,6 are only capable of 8x, if you look closely at the pictures you can see the [2,4,6] physical 16x slots are only electrically 8x, pins for lanes 9-16 are missing completely.

if you were to hypothetically run 7 GPUs on that board, you would be limited to 16x-8x-8x-8x-8x-8x-8x as per the manual.

if all of your GPUs are registering as PCIE gen 3 (8GT/s), then the PLX chips must be working on the CPU PCIE lanes only. research of the X99 block diagram is showing that the X99 chipset only provides 8 lanes of PCIe gen2. none at gen3.

---

Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

OK, my misnomer then. My workstation board I guess is not a server board per se. And it is single socket. Two PLX chips plus the CPU and the PCH to give X16 lanes to all slots.
https://www.newegg.com/Product/Product.aspx?Item=N82E16813132901

By the specification this board is capable to run 7 PCIe at 16x8x8x8x8x8x8 plus a M2 slot just not say with what processor it could do that magic.

Now imagine a host with the configuration and running 8 x 2080 Ti's ... something like a > 1.1 MM credit per day beast.

---