Let's start with the easy things.
The "app_config.xml" file can do all sorts of things to the way the application performs the calculations....
Just a few - you can select specific applications to run, the number of concurrent tasks on each GPU and so on - too much to worry about just now.
So... You have a riser which has x1 for the connection to the motherboard - electrically it can only ever have a maximum of x1 to the GPU. That shouldn't matter too much so long as the x1 path continues all the way back to the CPU.
I've just found your computer with four Quadro - it is doing a "BLC VLAR" in the mid twenty seconds using the "stock" GPU application - a performance not to be sniffed at when you consider that a similar CPU task on the same machine took over 1000 seconds. I'd just sit back and let it run for a bit more as it is to see how the times settle down in a day or so (you need to get at least 11 "normal" tasks validated before the estimate run times really start to ave any meaning). To track these you need to be in the "advanced" view of the BOINC manager the "normal view" just doesn't respond fast enough; in the advanced view sort by "progress" and you will see the most complete tasks at the top of the list.

Edit - I like the concept of getting the sun to pay for your crunching :-)

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

Also, This app_config.xml file Tom keeps telling me about... Does this have any impact on performance or is it just tinkering with the textual numbers on the user interface display??

The app_config.xml file can have considerable impact on performance of video cards with larger numbers of cudas than you are using. I don't know if it will have significant impacts on your cards.

That said, I would put such a file like this in your setiathome dir and restart the the manager after a full task shutdown of 20-30 seconds.

File name: app_config.xml

<app_config>
<app>
<name>setiathome_v8</name>
<gpu_versions>
<gpu_usage>1.0</gpu_usage>
<cpu_usage>1.0</cpu_usage>
</gpu_versions>
</app>
<app>
<name>astropulse_v7</name>
<gpu_versions>
<gpu_usage>1.0</gpu_usage>
<cpu_usage>1.0</cpu_usage>
</gpu_versions>
</app>
</app_config>

This certainly can't hurt anything and it will devote a full cpu core to each single tasking video card.

I agree with Bob, let this run all day and if possible all night for several days.

Misc Notes:
The Cuda tasks will eventually give away to SOG tasks if the SOG tasks that are sent out result in higher Gflops# over the long haul than the Cuda task Gflop#'s.

The cpu tasks will try 8.00, 8.05 and 8.08(alt) that I am aware of. Whichever processes with a high Gflop# will be the one that you will mostly get fed on this machine.

---

A proud member of the OFA (Old Farts Association).

Also, I wish I had a Seti APP that would show me all my machines and their state at a glance.

There maybe such an animal under Google Play for your cellphone.

There are a number of "management" and/or keep an eye on, applications. Try: https://boinc.berkeley.edu/addons.php

---

A proud member of the OFA (Old Farts Association).

While running the "stock" applications there is no need for an "app_config" file.
Performance impact on display - only if the display is attached to one of your Quadros, if you have the display attached to an on-board video driver then no impact.

A tool to look at a whole "farm" at once? - A lot of people use "BoincManager", a third party tool. I've used it in the past, but prefer to use a remote login to my headless Linux systems as I find BM doesn't give me any information I need that I can't get that way, however I do have a "mature, stable" farm, not one that is in a continuous state of flux.

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

While running the "stock" applications there is no need for an "app_config" file.

I know the documentation for Nvidia cards in the SOG readme includes boilerplate that says there should be one cpu per task. On most cards that translates into 1 cpu core per card.

Over the years I have played with using things like .33 cpu's for the Nvidia card(s). And recently I had been using the "optimizing" idea that I was told about for even my smallest machine (4c/8t). I can't swear the gpu was running any faster.

So a little while ago I tried to drop the app_config.xml settings by renaming the file. For some reason, it decided to "stick" to the 2 cpus / gpu setting after a couple of different bonic restarts so I set the cpu # down to well below .01 since I remember something around that low.

The estimated time to process had settled around 16 minutes lately. And a quick look shows no individual differences yet.

Bob, thank you for the assertion that with stock seti (and probably a low end card?) I don't need the app_config.xml file. I have been working on auto pilot with the app_config.xml file for so long I need to revist early testing.

Tom

---

A proud member of the OFA (Old Farts Association).

At this stage you are trying to get a STABLE system running, not tuning it for maximum performance, that will come later.. Tuning in the manner suggested by Tom will mask any fundamental problems with your hardware configuration. Let the computer run for the few hours a day your solar power will allow for a couple of weeks, then see what the figures say, before starting to think about anything else. Remember, you are using a rare GPU in an even rarer motherboard, so you can expect to have problems (particularly if/when you ramp up to filling every PCI slot with Quadros - that will be some beast!).

One thing that does puzzle me just now is the lack of data actually being returned by the GPUs, totally swamped by that returned by the CPU - that needs some digging as there may be another problem lurking in the background :-( Thinking aloud - Could it be an issue with the way the PCI lanes are shared on the motherboard? If it is then that is an all but insoluble problem as its the physical board hardware that's in the way. Have your tried using just one GPU in that board, putting it into each slot and watching how long it takes to turn round a task, then do the same with two in use, three in use (the reported run times suggest this test shouldn't take very long, but it will be a pain to do with all the swapping of cards needed.)

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

A bit more digging - there are reports in "Number Crunching" that PCI v1 can cause issues with SETI by chocking data flows, resulting in poor "back-end" performance (reporting and getting new tasks between the CPU & GPU - remember all data communication with the outside world goes through the CPU)
Edit:
Have a look at this post https://setiathome.berkeley.edu/forum_thread.php?id=83337&postid=1954758, and the rest of the thread - it may give some insight into your current performance issues..

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

I tried the 1-CPU per GPU thing on the mining rig and it didn't seem to do very much.
I still get the sense I am not educated well enough in all of this to understand some of the things that have been said.
Bob Smith keeps referring to "stock" applications". What else is there?

What are the things we users are crunching?
Astropulse is one right? Astropulse just listens for broader-band, short-time pulses.

Seti@home is searching narrow-band pulses and is a separate application?

When a user joins and seti@home and downloads BOINC, what do he or she get?

Is there anything in any of the applications that steer one application to GPU's and others to CPU's or is that a mixed bag in of not why?

If a GPU has so many cores crunching on one work unit, why not force the GPU to use one core each for one work unit?
I'm assuming its the old GPU architecture argument here but that doesn't help me understand why some upper end GPU's with thousands of cores can crunch two or three units at a time in seconds...

I am VERY concerned that this ASUS B250 MINER mobo is NOT working for this application but at the same time, how can it work so well for Bitcoin mining ??
Its run several days now and the average GPU time to complete one unit is on the order of an hour each.

Tom offered this: "Misc Notes:
The Cuda tasks will eventually give away to SOG tasks if the SOG tasks that are sent out result in higher Gflops# over the long haul than the Cuda task Gflop#'s.
The cpu tasks will try 8.00, 8.05 and 8.08(alt) that I am aware of. Whichever processes with a high Gflop# will be the one that you will mostly get fed on this machine."

Wha the heck does all that mean? What is SOG?

What to do next?

Loads of questions, let's try and answer them (in no particular order)

"stock" are the applications you get by default. The alternative are "optimised" (aka "lunatics"), which are a whole load of applications that use various combinations of special compiler settings, special (high performance) programming techniques, to get these you have to download them, keep an eye open for updates, expect "issues" when installing as it is possible to install applications that will not run properly on your hardware (not for the nervous).

Astropulse is a different type of data-set and needs its own applications. Again there are "stock" and "optimised" applications. You may find that you get the odd one or two, they are very rare, and "pay" better than the ordinary data we get.

GPUs are used in a parallel processing mode because the cores are very simple, and it would take many hours to do a single task if they were used in "single thread" mode (if indeed they could do them).

All tasks are created equal, but when they are sent out to a user each one is dedicated to a particular type of processor.

I too am concerned that your system isn't behaving as well as it should, but, if you look at the thread I linked to earlier today you will see some comments that you are not alone in this. The run times and the number of tasks completed by your GPUs just don't make sense to me - there has to be something strange happening on that motherboard. You may have stumbled on it a few days ago when you said the PCI slots were identifying as v1 not the expected v2 or v3.

Why does Bitcoin hashing work well and SETI not on your system - its down to the amount of calculation and data processed - Bitcoins hashing is very "simple" in comparison to SETI, also Bitcoins only work with a very small amount of data, compared with the several mega bytes of a SETI task (after decompression).

SoG - "Signals on GPU", started life as an "optimised" application, but has been migrated into the "stock" world. It uses a processing system called "OpenCL", and relies heavily on the CPU, but does the bulk of work on the GPU - hence the comments about having one CPU core/thread for each GPU task being processed. Personally I would call SoG a "hybrid application" rather than a "GPU application" due the heavy reliance on the CPU.

When a user downloads BOINC they get a processing management environment, into which they have to load applications for specific projects - you have joined SETI@Home as a project, and are now getting work from it. After joining a project you have to register at least one computer with it to get work.

One day I'll have to let the beast I manage loose on SETI - its current iteration is four 24-core Xeon E7 with 48 Quadro P6000 GPUs - let that loose for a week and look out Petri your time at the top would be over (provided it got enough tasks every day). Trouble is it is due to go back any day now having nearly finished its job :-(

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

Those were excellent questions, Dave.

Bob, do you write for a living? Maybe you should look at it?

One day I'll have to let the beast I manage loose on SETI - its current iteration is four 24-core Xeon E7 with 48 Quadro P6000 GPUs - let that loose for a week and look out Petri your time at the top would be over (provided it got enough tasks every day). Trouble is it is due to go back any day now having nearly finished its job :-(

Those are amazing numbers. How did "you" get 48 gpus on there? Expansion cards with Risers or an improbably large number of gpu slots :)

I would turn some shade of color of envy except that sounds like possibly a several hundred thousand dollar rig.

"...due to go back...."? That sounds like it was leased :(

Tom

---

A proud member of the OFA (Old Farts Association).

I gotta go with Tom on this one!
Howd you get 48 GPU's in that machine Bob?
I too sense a leased cray type beastie for some kinda science or math crunching...

I'm worried the Bitcoin world is the Bitcoin world and although it was fun to play in that space, I dont see dedicated Bitcoin mining architectures working for BOINC yet.
At least not at my house.
I messaged that one guy Tom pointed out to me but he never answered back on how he got so many GPU's working on one system.

I wonder what the largest number of PCIeX16 slots on a single mobo a guy can get his hands on?
Humm.... Sell crappy ASUS Bitcoin miner crap and rack and buy a better mobo with a heap of slots on it...
Got 8 good GPU's coming...
Got three "BIG" systems to play with.

I cant get my dual Xeon machine to stay stable no matter what I do, which granted, isn't much on that AsRockRack server mobo.
I might have to break that thing apart and get those CPU's on regular Army mobo's to be useful.

Oh the pain... The pain...

I tried the 1-CPU per GPU thing on the mining rig and it didn't seem to do very much.
-------------------
What to do next?

Besides running the Miner w/o changing anything for a while may I suggest you download GPU-Z, a free utility for Windows that will allow you to see how hard the gpu's are being run on your system?

If they are not mostly staying above 90% then something isn't feeding them fast enough.

I expect it will be even worse if you were running GTX 1050Ti's but until you actually try them out, there is no guarantee. The problem could be MB bus speed, it could be gpu's, it could be a malset bios.

HTH,
Tom

---

A proud member of the OFA (Old Farts Association).

This BitMiner (I presume, it has 13 odd gpus) is running well: https://setiathome.berkeley.edu/show_host_detail.php?hostid=8568434 The machine is about 14-16 place on the Leaderboard.

I cant get my dual Xeon machine to stay stable no matter what I do, which granted, isn't much on that AsRockRack server mobo.
I might have to break that thing apart and get those CPU's on regular Army mobo's to be useful.

Oh the pain... The pain...

That is odd, I have 1 e5-2670 running on one of those generic X79 MB's they are selling from "overseas" although I bought mine in the US and got overseas shipping speed. The other two e5-'s are on a ASRock EP2C602. Neither are crashing.

You might download HW_info and peek around inside of it and see if there is anything odd looking. Is it running too hot? Might try "TThortle" as long as you are running on windows.

Might try a different video card. If the MB has VGA, run it headless and see if it is more stable.

I was hoping you could get between 2 and 4 gtx 1050's running on that machine once they arrived. But not if it is unstable.

HTH,
Tom

---

A proud member of the OFA (Old Farts Association).

External GPU rack system.
I manage it for an company that does serious FEA and CFD work, and as part of the agreement no pictures, no "other software" etc.
The hardware is similar to that used by Berkeley for acquiring the data from the telescopes, and I'm glad I don't have to pay the power bill!

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

The ASRock server motherboards are normally pretty stable, provided you use the correct memory and processor. Its too many years since I actully set one up, but there are a few running SETI - I think "Al" has at least one in his farm.

Looking at the figures for that high GPU count system, they aren't as well as I would expect. Last time I looked it had 12 or 13 GTX1080ti. It is however suffering a higher than expected error rate, so there may well be problems with its stability (loads of "segment violation" errors across all its GPUs).

One thought that has just crossed my mind - you have loaded the Quadro specific version of the driver, not the GTX version, and you have the nVidia sourced driver not an MS crippled version.

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

[quote]I tried the 1-CPU per GPU thing on the mining rig and it didn't seem to do very much.
-------------------
What to do next?

Besides running the Miner w/o changing anything for a while may I suggest you download GPU-Z, a free utility for Windows that will allow you to see how hard the gpu's are being run on your system?

If they are not mostly staying above 90% then something isn't feeding them fast enough.

I now remember that CUDAXX tasks (the name shows in the tasks panel of the Boinc Manager) don't necessarily pull 90%. It is only the SOG gpu tasks that should pull above 90%.

Sorry.

Tom

---

A proud member of the OFA (Old Farts Association).

One thought that has just crossed my mind - you have loaded the Quadro specific version of the driver, not the GTX version, and you have the nVidia sourced driver not an MS crippled version.

Bob,
I think you just said "Have you loaded the Quadro specific version of the Driver? Not the GTX version. And you are not using the Microsoft version of the driver?"

Very good points. Dave, we have had "mixed" sucess under Windows with Microsoft's version of the Nvidia drivers. Sometimes they don't work at all, sometimes they are less productive, sometimes they are "fine." So the standard recommendation is to get a copy of exactly what you need from Nvidia driver downloads and do a "custom install" which includes checking the box which says "clean install".

No one installs the "GE Force" app unless they are absent-minded or a gamer too.

HTH,
Tom

---

A proud member of the OFA (Old Farts Association).

Ha!

"No one installs the "GE Force" app unless they are absent-minded or a gamer too."
Funny. I guess that means I'm okay cause I don't even know what that is.

I will get the GPU analyzerware and see what these cards are doing.
As I offed in previous posts, they run red hot and high current so they are doing "something".
Whatever that is, they are NOT crunching the heck out of work units.

This morning I was able to get all eight cards working on the miner!
And yes, I do not let MS do much of anything in my universe so the nVidia drivers are all from their site direct.

I was hoping to find a comprehensive tech application that can measure the bandwidth of data to a GPU and its crunching prowess.
Tom mentioned one before in an older post and I wrote those down but I dont have them here at work with me today.

China delayed my order of gtx 1050Ti's but they will fulfill eventually. I hope they are legitimate.

I see I was able to inch up to the top 1%....

Dave

I just got an answer back from that guy running that killer GPU system that Tom found...

"It's a purpose built hashing rig yes, crypto is one of the use cases ;) . I believe a ASUS Miner Expert series."

BINGO!

Gotta figure out how he gets it to work!

Tom, Bob, are there "better" nVidia drivers available from "others" than those from nVidia?