That was interesting. I was having a very slow laggy screen up to a few minutes ago. I reset the two places I can set the PCIe speed to "auto" from "gen1" and it boots faster, the snap crackle and pop are back...

Tom

Now where was I when the system quit?

Oh, yeah I was experimenting with running 10 gpus to see if it would "crash".

Well it did but apparently it was an OS bug/feature that was reporting something odd with the P100-102 cards (my guess).
Once I was lead to the solution it is up and running again.
It ran all night/day with 7-8 gpus. And I have just plugged the other two in (another p100-102 and a gtx 1070).

Tom

---

A proud member of the OFA (Old Farts Association).

Oh, yeah I was experimenting with running 10 gpus to see if it would "crash".

Ok, once again into the "fire" of 10 gpus.

I am running a mix of 1 Gtx 1660 Super, 1 Gtx 1070, 2 gtx 1060 3Gb and 6 p100-102's. On an MSI B360-F Pro with Pcie set to auto and an I9-9900K (I couldn't find a 9900) that was cheaper.

I think I have rooted out all the hardware failures in the riser system so now its back to that 2wk - 6wk non-stop test.

Then I will consider my next investment in a bigger mining frame so I can both setup 10 gpus properly and start expanding upwards.

I now have the i7w utility installed and it is tracking cpu temperatures and the load on the cpu cores nicely.

Tom

---

A proud member of the OFA (Old Farts Association).

If using mining board and a lot of GPUs beware of using AMD's ROCm drivers. They require you to have as many GEN3 lanes available to the CPU as you have GPU cards. That is a problem for a lot of motherboards and CPUs, not just the cheap ones.

Found out about it here after I lost 4 RX570 cards upgrading to ROCm
https://github.com/RadeonOpenCompute/ROCm/issues/589

ROCm probably works fine if you buy the best AMD motherboards and AMD GPU cards available. You will have a superhighway to the GPU for all your apps. Unfortunately, if you have commercial motherboards your "x1" unpaved backroads are not going to be used nor any GPU "rising" in them.

If using mining board and a lot of GPUs beware of using AMD's ROCm drivers. They require you to have as many GEN3 lanes available to the CPU as you have GPU cards. That is a problem for a lot of motherboards and CPUs, not just the cheap ones.

Found out about it here after I lost 4 RX570 cards upgrading to ROCm
https://github.com/RadeonOpenCompute/ROCm/issues/589

ROCm probably works fine if you buy the best AMD motherboards and AMD GPU cards available. You will have a superhighway to the GPU for all your apps. Unfortunately, if you have commercial motherboards your "x1" unpaved backroads are not going to be used nor any GPU "rising" in them.

I was under the impression that ROCm drivers didn't work for SETI crunching anyway.

---

Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

I came across these recently. might be something to look into for those of you that want to utilize a slot obstructed by a GPU.
Just use the slot adapter piece, and connect it to a regular x16 riser board.

USB Risers that fit under a GPU

---

Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

I was under the impression that ROCm drivers didn't work for SETI crunching anyway.

I didn't try SETI during the short time I had ROCm up and running. I did test Einstein as discussed here (WARNING SNOWFLAKE ALERT!)

The following image pretty much sums it up. The Pink color 2.07 app failed under ROCm. The successfully task, 2.08 was the beta rocx test app. Einstein now has an app that can run under ROCm or under the "normal" amdpro driver. The black box show that only the GPU in the x16 slots "had the right stuff"

I came across these recently. might be something to look into for those of you that want to utilize a slot obstructed by a GPU.
Just use the slot adapter piece, and connect it to a regular x16 riser board.

USB Risers that fit under a GPU

Nice, but they should have had a full x16 at the end or simply just sold the riser part.

I got a 4 pack of extenders
https://www.ebay.com/itm/153836826498?ViewItem=&item=153836826498

but they are only useful with the right angle risers like the ones that come with thee '3-in-1's here
https://www.ebay.com/itm/MintCell-VER-008S-16x-to-1x-USB-Riser-GPU-Adapter-Card-Mining-LOT-ETH-ZEC-XMR/192360550029?ssPageName=STRK%3AMEBIDX%3AIT&var=492330545161&_trksid=p2057872.m2749.l2649

I came across these recently. might be something to look into for those of you that want to utilize a slot obstructed by a GPU.
Just use the slot adapter piece, and connect it to a regular x16 riser board.

USB Risers that fit under a GPU

Nice, but they should have had a full x16 at the end or simply just sold the riser part.

It appears that this is intended to extend a x1 interface to some remote card, like a sound card or something, doesn't look like it was designed for GPU. but you can just use the USB-PCIe adapter with your normal GPU riser board.

but I don't understand the obsession with people trying to use ROCm drivers. are they better? are they faster? why re-invent the wheel when drivers that work already exist? seems like a waste of time.

---

Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

At least for SETI what they need instead of the ROCm drivers is a new AMD Petri33. Who could port the Special Sauce to the AMD world.

---

I came across these recently. might be something to look into for those of you that want to utilize a slot obstructed by a GPU.
Just use the slot adapter piece, and connect it to a regular x16 riser board.

That would allow you to run a "regular" gpu in the long slot an MSI B360-F Pro or probably any of the other mining motherboards that are laid out similarly and still use the higher performance slots next to the long slot.

That would be an argument for moving my RTX 2060 Super from another box to my "mining rig".

Tom

---

A proud member of the OFA (Old Farts Association).

I just realized that another difference between Mining system bios and "regular" overclock bios is the OC has a lot more parameters you can play with.
==edit==
And Intel server bio's.
==edit==
It occurred to me that if I increased the LLC parameter I might have less trouble.

And the MSI B360-F Pro (Intel) doesn't have an LLC parameter. I am pretty sure the Biostar (Amd AM4 socket) mining MB I had didn't have one either.

Tom

ps. just added "pci-nommconf" to the grub command line.

---

A proud member of the OFA (Old Farts Association).

On the general presumption that a higher count gpu system generates a higher peak demand after any/weekly maintenace I am using the following cc_config.xml file.

<cc_config>
 <log_flags>
   <sched_op_debug>1</sched_op_debug>
 </log_flags>
 <options>
   <use_all_gpus>1</use_all_gpus>
   <save_stats_days>180</save_stats_days>
   <max_tasks_reported>50</max_tasks_reported>
   <max_file_xfers>4</max_file_xfers>
   <max_file_xfers_per_project>2</max_file_xfers_per_project>
 </options>
</cc_config>

While this will slow filling my cache after maintenance it may also reduce the number of "http errors" that I get when trying to report 350+ tasks.

Tom

---

A proud member of the OFA (Old Farts Association).

ps. just added "pci-nommconf" to the grub command line.

Typo: pci=nonmconf

---

A proud member of the OFA (Old Farts Association).

It https://setiathome.berkeley.edu/show_host_detail.php?hostid=8894290 may have topped out at under 400,000. On the other hand it had a multi-day hesitation after maintenance last week too.

Don't you just hate it when you have to be patient?

"...I want more patience, and I want it RIGHT NOW!!!!..."

Anyway. The other question is if I should buy the 19 gpu rack so I can clean up the mess of the 6 gpu rack with 8 installed and two on the table....

Tom

---

A proud member of the OFA (Old Farts Association).

I was under the impression that ROCm drivers didn't work for SETI crunching anyway.

I didn't try SETI during the short time I had ROCm up and running. I did test Einstein as discussed here (WARNING SNOWFLAKE ALERT!)

The following image pretty much sums it up. The Pink color 2.07 app failed under ROCm. The successfully task, 2.08 was the beta rocx test app. Einstein now has an app that can run under ROCm or under the "normal" amdpro driver. The black box show that only the GPU in the x16 slots "had the right stuff"...

The clue there is for the no support for "atomics" for that hardware for ROCm....

I've had both Einstein and s@h working fine using ROCm, but PCIe3 atomics operation is needed for ROCm now. Or, only specific hardware/software combinations work using the AMDpro driver for running the openCL tasks...


Happy fast crunchin',
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)

Well, this experiment cracked 450,000 RAC but I am going to have to give it rest for a while. I hope to return with a bigger and better system later this spring. More gpu's. https://setiathome.berkeley.edu/show_host_detail.php?hostid=8894290

Tom

---

A proud member of the OFA (Old Farts Association).

Well, this experiment cracked 450,000 RAC but I am going to have to give it rest for a while. I hope to return with a bigger and better system later this spring. More gpu's. https://setiathome.berkeley.edu/show_host_detail.php?hostid=8894290

A quick glance and my first thought was that you're doing well for that graphics card until I noted the "[10]"!:

"[10] NVIDIA GeForce GTX 1660 SUPER (4095MB) driver: 440.59 OpenCL: 1.2"

Good fun!

And good going getting past the BIOS limiting the max number of connected GPUs...


Happy fast crunchin',
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)

he's using a mining board. there was no BIOS limit. at least no limit ~8-10 GPUs.

---

Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

And good going getting past the BIOS limiting the max number of connected GPUs...

Martin,
The whole reason we have a thread about "Bitcoin GPU-based Mining Machines...." is because some of these motherboards have 11, 12, 18 and 19 PCIe slots specifically setup for running video cards.

However, the MB's are geared from running those slots at Gen1 while in "Mining Mode". They can run out of mining mode and at least some of the slots will run Gen2 or even Gen 3.

Our most successful user of Mining Motherboards is Tbar. https://setiathome.berkeley.edu/show_host_detail.php?hostid=6813106 He is currently parked at 4th place with more than twice the Rac I had on my box.

I believe he is using an ASUS B250 Mining MB here with 19 gpu slots. He has 14 GPUs installed.

While he does have an RTX 2070 running on that system. The bulk of his GPUs are gtx 1070's and gtx 1060 3GB.

He also is using a pretty small CPU to generate an imposing Rac.

I don't think he processes in CPU tasks on that system.

I happen to wish to run CPU tasks as well as gpu tasks on my system so I chose an MB that would support a CPU with 8c/16t's. While the MB price is nearly the same as the Asus B250 the CPU was significantly more expensive and if I move into a rack that is big enough the minimal CPU cooler I was using will be upgraded to a much larger one.

Prior to using this MB, I have repeatedly bumped into the "Not enough PCIe resources" under Intel MB's and everything from flatly refusing to boot to system instability on some AMD motherboards.

The MSI B360-F Pro is the first MB I have had that was designed to run a very high count of GPUs while not spending Workstation Class money on it. I have played with several Intel server MB's as well as several different AMD AM4 socket motherboards in my twin pursuit of "more GPUs" while also being able to run CPU tasks.

I have an affliction called "Core Envy" which keeps me from following Tbar's path directly.

I took my system offline because I am being involved in a slow-motion family disaster that I have to get to a stable point where I can invest more money and electricity in this https://setiathome.berkeley.edu/show_host_detail.php?hostid=8894290 system.

My understanding is there are BOINC projects that require higher performing Pcie buses with their GPUs in order to work. Those projects clearly would not be a good candidate for the Bit-Coin type mining motherboard.

Tom

---

A proud member of the OFA (Old Farts Association).

Well that sucks. Do we have any other gpu-oriented BOINC projects that will run with a minimal PCIe load? Otherwise the entire experiment with Mining Motherboards becomes moot.

Tom

---

A proud member of the OFA (Old Farts Association).