How to use multiple GPU's

Questions and Answers : GPU applications : How to use multiple GPU's
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · Next

AuthorMessage
Profile Bravo6
Avatar

Send message
Joined: 28 Oct 99
Posts: 52
Credit: 44,947,756
RAC: 0
United States
Message 1957689 - Posted: 28 Sep 2018, 22:40:52 UTC - in response to Message 1954736.  

I understand something about the A-10 has died. Hope it isn't the motherboard/cpu :(


Haven't had time to mess with the a10, but:
I have started a rack with two GTX 1050s making a total of three 1050s in my one hp refurb, T0 on my computers page. After i got the third one up and apparently stable I noticed the WUs on that computer running in the CPUs have dramatically slowed down (like 5days instead of 1.5 hours), now I expected some difference, but not that much. In order for me to understand this process better, can someone explain to me what is actually happening here. (that computer has 8GB ram..)
thanx

actually getting the 2 additional gpus in the rack hooked up and running was not too hard, rebooting has been the best solution to many little glitches, (so far). and rack on and off first, then computer......(when I remember, of course)
"Don't worry about it, nothing is gonna be O.K. anyway.........."
ID: 1957689 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1957706 - Posted: 29 Sep 2018, 0:21:09 UTC - in response to Message 1957689.  


Haven't had time to mess with the a10, but:
I have started a rack with two GTX 1050s making a total of three 1050s in my one hp refurb, T0 on my computers page. After i got the third one up and apparently stable I noticed the WUs on that computer running in the CPUs have dramatically slowed down (like 5days instead of 1.5 hours), now I expected some difference, but not that much. In order for me to understand this process better, can someone explain to me what is actually happening here. (that computer has 8GB ram..)
thanx


Before we start offering possible diagnosis could you post your "app_config.xml" file and your MB*SOG.txt command line, please?

Thank you,
Tom
A proud member of the OFA (Old Farts Association).
ID: 1957706 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22158
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1957752 - Posted: 29 Sep 2018, 6:49:20 UTC

A couple of other things to consider
Are the CPU tas AstroPulse tasks? Thes are signified by the fiirst letters of the task name being "ap".
Next, using windows task manager sort the process list to get the max cpu use at the top of the list sand see what the top few running tasks are. I would expect to see several copies of each of the two SETI programs. (Two because each running task has some cpu contribution.)
The comman line file Tom asks for has a low impact on cpu usage, but the other file can have quite a significant impact if things aren't set just right.
One quick check, using Boinc manager's advanced view what % of your cpu have you set aside for boinc to use?
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1957752 · Report as offensive
Profile Bravo6
Avatar

Send message
Joined: 28 Oct 99
Posts: 52
Credit: 44,947,756
RAC: 0
United States
Message 1957801 - Posted: 29 Sep 2018, 17:51:41 UTC - in response to Message 1957706.  



Before we start offering possible diagnosis could you post your "app_config.xml" file and your MB*SOG.txt command line, please?

Thank you,
Tom

Do not find any such files in dir setiathome.berkeley.edu, have a AP_clFFTplan_GeForceGTX1050_32768_gr64_lr16_wg256_tw0_ls1024_bn16_cw16_r2887.bin_38813 file which appears to be a config statement in the title itself.
also a file ap_cmdline_7.10_windows_intelx86__opencl_nvidia_100.txt which is blank
"Don't worry about it, nothing is gonna be O.K. anyway.........."
ID: 1957801 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1957805 - Posted: 29 Sep 2018, 18:26:53 UTC - in response to Message 1957801.  
Last modified: 29 Sep 2018, 18:42:06 UTC



Before we start offering possible diagnosis could you post your "app_config.xml" file and your MB*SOG.txt command line, please?

Thank you,
Tom

Do not find any such files in dir setiathome.berkeley.edu, have a AP_clFFTplan_GeForceGTX1050_32768_gr64_lr16_wg256_tw0_ls1024_bn16_cw16_r2887.bin_38813 file which appears to be a config statement in the title itself.
also a file ap_cmdline_7.10_windows_intelx86__opencl_nvidia_100.txt which is blank


The "app_config.xml" file is one you would create. If you didn't create it, its going to be missing :)

The "MB*sog.txt" file will also be empty if you have not added anything. It will not be present until you have processed your first Sog task.

I haven't been able to find an AP task in your cpu task listing, which doesn't rule out the possiblity that your really high cpu processing time was associated with an AP task. I have seen "estimated time to completion" in times that didn't track what was actually going on. You have been processing AP tasks in your gpus.

The discussion about your CPU assumes we are talking about and have been talking about your i3 CPU 530 computer which currently has the 3 Gtx 1050's on it. We can't see the computer names only the Seti ID# and some technical details. The order of the machine listing is dynamic and can change.

All of your CPU processing speed numbers could be consistent for a 2 core with Hyperthreading cpu. And they are pretty low. I would normally expect them to be in the high teens or low twentys for Gflop#'s. I think my expectations are based on 4 core CPUs so I am not sure I would worry.

If you are not going to drive the gpus with "one core" per gpu, you could turn off hyperthreading (in the bois) and should get a fairly good bump in the cpu processing speed. It also might slow the total processing production on your cpu side of the house. Only experimentation can show one or the other.

Look for "readme" SOG for more information on how to use the command line. You are getting good production already so it could be a "don't fix what ain't broke" situation.

It is entirely possible that nothing is wrong, you just have a 2 core system.

Tom
A proud member of the OFA (Old Farts Association).
ID: 1957805 · Report as offensive
Profile Bravo6
Avatar

Send message
Joined: 28 Oct 99
Posts: 52
Credit: 44,947,756
RAC: 0
United States
Message 1957814 - Posted: 29 Sep 2018, 19:34:27 UTC - in response to Message 1957805.  
Last modified: 29 Sep 2018, 19:35:07 UTC


The "app_config.xml" file is one you would create. If you didn't create it, its going to be missing :)

The "MB*sog.txt" file will also be empty if you have not added anything. It will not be present until you have processed your first Sog task.


OK now I am remembering. Many yrs ago when first joined I was getting into those files and functions. Life got in the way. Have just started to get back into it. Can you give me a decent file to work with, or point me to some samples. i can see I need to get in the weeds again.
thanks
"Don't worry about it, nothing is gonna be O.K. anyway.........."
ID: 1957814 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22158
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1957818 - Posted: 29 Sep 2018, 20:07:09 UTC

By default on windows thesee files are in hidden folder, normally c:/programdata/boinc/setiathome (variations in capital letters do occur).
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1957818 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1957851 - Posted: 30 Sep 2018, 0:18:12 UTC - in response to Message 1957814.  
Last modified: 30 Sep 2018, 0:21:22 UTC


The "app_config.xml" file is one you would create. If you didn't create it, its going to be missing :)

The "MB*sog.txt" file will also be empty if you have not added anything. It will not be present until you have processed your first Sog task.


OK now I am remembering. Many yrs ago when first joined I was getting into those files and functions. Life got in the way. Have just started to get back into it. Can you give me a decent file to work with, or point me to some samples. i can see I need to get in the weeds again.
thanks


This is from a Gtx 750Ti machine so it "should" be in the right ballpark. If I am screwing this up Bob will point it out :)

app_config.xml
<app_config>
<app>
<name>setiathome_v8</name>
<gpu_versions>
<gpu_usage>1.0</gpu_usage>
<cpu_usage>1.0</cpu_usage>
</gpu_versions>
</app>
<app>
<name>astropulse_v7</name>
<gpu_versions>
<gpu_usage>0.50</gpu_usage>
<cpu_usage>1.0</cpu_usage>
</gpu_versions>
</app>
</app_config>

All of this down here is comments (after the </app_config>)

If you are running multiple projects and you want finer grained control over how many of what kind is being processed, you can
use the following inside the <app_config> of each project app_config.xml file.

<project_max_concurrent>2</project_max_concurrent>


One of the biggest simple headaches I have is creating this file the first time using Notepad. It has a tendency to save the file as "app_config.xml.txt" which is not recognized! :(

Comments on the app_config.xml file
The CPU setting controls the number of CPU threads that are devoted to driving a gpu card. If the number is less than 50% by some amount it will no longer dedicate a CPU thread to driving the gpu but will still drive the gpu. The documentation recommends 1 CPU per gpu for higher end cards. A gtx 1050 probably is not a higher end card :)

You can run multiple tasks in "parallel" on a gpu by setting the gpu number in the above file to say 0.50 for two tasks and .33 for three tasks. My understanding is the gpu version of the AP task doesn't use my whole gpu so I have it set up so that if two were to show up and the same time, they both get processed at the same time. AP tasks are rare enough that I don't count on them hitting at the same time like this.

My experience is neither a Gtx 750Ti or a Gtx 1060 3GB gains you anything by running more than one gpu task at a time. It won't hurt to try but it probably will slow your production on the GPUs by as much as 120%.

The following is an example of a not particularly aggressive command line for the SOG gpu task. Since the gtx 1050 series was the replacement for the gtx 750 series (as I understand it) and draws less power also, this command line probably can't hurt and might help. There are a bunch more things you could include that MIGHT speed up things, or might not. Somethings I have not included will often cause screen lag.

mb-cmdline*_sog.txt is the text file you are looking for.
-sbs 192 -spike_fft_thresh 2048 -tune 1 64 1 4


HTH,
Tom
A proud member of the OFA (Old Farts Association).
ID: 1957851 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22158
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1957893 - Posted: 30 Sep 2018, 8:07:56 UTC

Bravo6:
A quick look at your computer with the 3 gtx1050 gpus shows it to have a 4 cpu. The percieved wisdom is to have one cpu core for every concurrent gpu task being run, with the config file Tom has suggested you would be running 6 concurrent gpu tasks plus 4 cpu tasks, thus significantly over commiting your cpu. While this may bo ok with an i7 an i3 like yours will struggle.
The simplest solution is to not deploy that file but to use the "use max cpu"limit in the boinc manager on that cokputer, set it to something around 20 to 25 percent. You should then see one cpu task running in about 90min and three gpu tassks taking around 10min.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1957893 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1957910 - Posted: 30 Sep 2018, 14:04:05 UTC - in response to Message 1957893.  

Bravo6:
A quick look at your computer with the 3 gtx1050 gpus shows it to have a 4 cpu. The percieved wisdom is to have one cpu core for every concurrent gpu task being run, with the config file Tom has suggested you would be running 6 concurrent gpu tasks plus 4 cpu tasks, thus significantly over commiting your cpu. While this may bo ok with an i7 an i3 like yours will struggle.
The simplest solution is to not deploy that file but to use the "use max cpu"limit in the boinc manager on that cokputer, set it to something around 20 to 25 percent. You should then see one cpu task running in about 90min and three gpu tassks taking around 10min.


The reason it is the simplest solution is that you don't have to try to create the "app_config.xml" file. :)

If you do create the file. You need to set it something like this:
<cpu_usage>0.33</cpu_usage>
to avoid over-committing your CPU.

Tom
A proud member of the OFA (Old Farts Association).
ID: 1957910 · Report as offensive
Profile Bravo6
Avatar

Send message
Joined: 28 Oct 99
Posts: 52
Credit: 44,947,756
RAC: 0
United States
Message 1957938 - Posted: 30 Sep 2018, 19:51:41 UTC - in response to Message 1957851.  
Last modified: 30 Sep 2018, 19:54:26 UTC


This is from a Gtx 750Ti machine so it "should" be in the right ballpark. If I am screwing this up Bob will point it out :)
Tom


OK thanx, that is what I thought. Back to my shell programming and flang programming days ... except not case and esac, now I actually have to read the documentation to dig out all the applicable options and flags.
God do I hate reading documentation............ I am supposed to be able to take it all in by osmosis! :-(
(Prepare for irritating comments on efficacy of the BOINC manual......... I think it is probaby still required for all programmers to infuse their individual highly reasonable opinions on every piece of documentation visible))
(Still think UNIX is better, i cut my teeth on DOS and windows at version 1.something, hasn't improved much since Microsoft stole it from QD Dos and Radio Shacks DESKMATE)
"Don't worry about it, nothing is gonna be O.K. anyway.........."
ID: 1957938 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22158
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1957940 - Posted: 30 Sep 2018, 20:23:47 UTC

The amount of a GPU used by an AstroPulse tasks varies quite considerably, while many of the more recent ones have been "simple" to resolve and use little in the way of the GPU's resources, there are quite a number that do require a lot more computing power and will make a GTX1080 being fed by a Xeon sweat hard.
As with many things it is worth starting simple to make sure the system is behaving well before jumping in to fine tune it.
How are the CPU run-times settle now - are they still way up in the days, or have they dropped to a more reasonable couple of hours?

The options for the SoG application are not in the BOINC manual, but in the read.me file that comes as part of the Lunatics installer (or at least they used to be there...)

Move over to Linux and you gain access to a very fast CUDA application which uses OS features that are lacking in windows. (Also if you don't adopt the windows "must have the latest update" attitude the current distros are very easy to install and get BOINC & SETI running, my norm is about thirty minutes from a pile of cold bits to a system that is running and stable - excluding the time needed to re-find the USB stick with everything on it....)
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1957940 · Report as offensive
Profile Bravo6
Avatar

Send message
Joined: 28 Oct 99
Posts: 52
Credit: 44,947,756
RAC: 0
United States
Message 1957941 - Posted: 30 Sep 2018, 21:05:44 UTC - in response to Message 1957910.  
Last modified: 30 Sep 2018, 21:16:03 UTC

ok found the app config files but the following is confusing to my poor brain:
the below syntax is different from the docs for BOINC or are these updated?

<gpu_versions>
<gpu_usage>.5</gpu_usage>
<cpu_usage>.4</cpu_usage>
</gpu_versions>

Or as my amazing intellect just thought of :
does the .5 and .4 refer to that amount (%) of CPUs, gpus present in the computer .
.5 meaning 2 in a 4cpu/gpu system?

- is that syntax correct, CPU and GPU statement in the GPU versions block?

-Where does the accepted application_name (syntax) come from? is it just Seti@home?

-can all these options be in either the app_config.hml or cc_config.hml?
thanx
"Don't worry about it, nothing is gonna be O.K. anyway.........."
ID: 1957941 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1957951 - Posted: 30 Sep 2018, 22:56:22 UTC - in response to Message 1957941.  

ok found the app config files but the following is confusing to my poor brain:
the below syntax is different from the docs for BOINC or are these updated?

<gpu_versions>
<gpu_usage>.5</gpu_usage>
<cpu_usage>.4</cpu_usage>
</gpu_versions>

Or as my amazing intellect just thought of :
does the .5 and .4 refer to that amount (%) of CPUs, gpus present in the computer .
.5 meaning 2 in a 4cpu/gpu system?


Let's start with the first part of this .5 gpu usage means you are using half of a gpu to run 1 work unit. Thus you can run 2 work units per GPU or 4 work units on 2 GPUs, 6 on 3 GPUs or 8 on 4 GPUs

The second part cpu usage, is saying I would like each work unit to use only 0.4 of a cpu thread. I say, would like to, because in reality it's going to use as much as it wants despite what value you put in there. If you put less than it requires, it will use more. If you put more then it will use less. This part is hard for most to get their head around. Generally we tell must people just to assign 1 thread to each work unit, that way you don't deprive work of any time on a thread.

- is that syntax correct, CPU and GPU statement in the GPU versions block?


This is the older form of a app_config file. I generally use the following


<app_config>
  <app_version>
    <app_name>setiathome_v8</app_name>
    <plan_class>opencl_nvidia_SoG</plan_class>
    <avg_ncpus>1</avg_ncpus>
    <ngpus>0.5</ngpus>
    <cmdline></cmdline>
  </app_version>
   <app_version>
    <app_name>astropulse_v7</app_name>
    <plan_class>opencl_nvidia_100</plan_class>
    <avg_ncpus>1</avg_ncpus>
    <ngpus>0.5</ngpus>
    <cmdline></cmdline>
  </app_version>
<project_max_concurrent></project_max_concurrent>
</app_config>


Why? it allows me to add command lines in 1 place without having to alter the app_info. It also allows me to get a limit on how many work units I run total on the machine. Understanding that GPU work task will get preference over CPU work task until such time as there are no GPU work task anymore.

-Where does the accepted application_name (syntax) come from? is it just Seti@home?
Yes above

-can all these options be in either the app_config.hml or cc_config.hml?
thanx


They are in the app_config.xml in the same folder as the app_info.xml You must make sure the syntax at the end is xml and not txt. When you save it from notepad, it will always default to .txt so you have to remember to change the save file type as to *.*

The cc_config.xml is in another folder 2 levels up from setiathome.berkeley.edu folder. It's has different commands, usually for allowing usage of different GPU types and error reporting. I won't go in depth of those here.

Good Luck
ID: 1957951 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1958088 - Posted: 1 Oct 2018, 22:10:43 UTC - in response to Message 1957940.  

Move over to Linux and you gain access to a very fast CUDA application which uses OS features that are lacking in windows. (Also if you don't adopt the windows "must have the latest update" attitude the current distros are very easy to install and get BOINC & SETI running, my norm is about thirty minutes from a pile of cold bits to a system that is running and stable - excluding the time needed to re-find the USB stick with everything on it....)


I can tell you the speed up on my gtx 750Ti was amazing. And the Gtx 1050's really can "smoke".

I needed some things that were "handier" in Windows or I would still be running the 750ti under Linux and the Cuda90+ apps.

Tom
A proud member of the OFA (Old Farts Association).
ID: 1958088 · Report as offensive
Profile Bravo6
Avatar

Send message
Joined: 28 Oct 99
Posts: 52
Credit: 44,947,756
RAC: 0
United States
Message 1958434 - Posted: 4 Oct 2018, 23:24:40 UTC - in response to Message 1958088.  
Last modified: 4 Oct 2018, 23:26:15 UTC

It there a decent utility around for testing running video cards for performance, etc? or is it not worth it?
Got fourth card running on one computer and am not sure what I am seeing in re: performance diffs.
thanx
"Don't worry about it, nothing is gonna be O.K. anyway.........."
ID: 1958434 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1958444 - Posted: 5 Oct 2018, 0:01:48 UTC - in response to Message 1958434.  

It there a decent utility around for testing running video cards for performance, etc? or is it not worth it?
Got fourth card running on one computer and am not sure what I am seeing in re: performance diffs.
thanx


For Windows, I like a free utility called GPU-Z.

Tom
A proud member of the OFA (Old Farts Association).
ID: 1958444 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1958672 - Posted: 5 Oct 2018, 22:26:46 UTC - in response to Message 1958434.  
Last modified: 5 Oct 2018, 22:27:18 UTC

It there a decent utility around for testing running video cards for performance, etc? or is it not worth it?
Got fourth card running on one computer and am not sure what I am seeing in re: performance diffs.
thanx

I prefer SIV64X. Gives lots of data also boinctasks. Those 2 were my main before I went to Linux
ID: 1958672 · Report as offensive
Profile Bravo6
Avatar

Send message
Joined: 28 Oct 99
Posts: 52
Credit: 44,947,756
RAC: 0
United States
Message 1959351 - Posted: 9 Oct 2018, 1:11:49 UTC
Last modified: 9 Oct 2018, 1:12:53 UTC

Ok, adding gpus fairly easily ( reboot, reboot,reboot) in the status column (for GPUs) ie: Running (0.18 CPUs + 1 NVIDA GPU).... where is the 0.18 (in this instance as I see almost all pcs are different) configured, and what are the values sources? (if that makes sense)
THANX
"Don't worry about it, nothing is gonna be O.K. anyway.........."
ID: 1959351 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 1959355 - Posted: 9 Oct 2018, 1:27:03 UTC - in response to Message 1959351.  

Ok, adding gpus fairly easily ( reboot, reboot,reboot) in the status column (for GPUs) ie: Running (0.18 CPUs + 1 NVIDA GPU).... where is the 0.18 (in this instance as I see almost all pcs are different) configured, and what are the values sources? (if that makes sense)
THANX


The "app_config.xml" file that is located in the hidden ProgramData -> BOINC -> projects -> setiathome

Will allow you to change that value. What you are seeing is the default. And it is entirely possible you don't need to tinker with it. Documentation for the NVidia cards that are running SOG appears to suggest one cpu core / gpu task. If you have 1 gpu task on a card it would use 1 CPU core in this scenario.

Previously, I think we talked about how to set this in an "app_config.xml" file.

You set the cpu to 1 or .33 or even 0.18 and then set the gpu to 1 or 0.50 (two tasks) or even 0.33 (3 tasks).
Usually, 1 task/card for anything less than a gtx 1080 or 1080Ti seems to be about right.

Tom
A proud member of the OFA (Old Farts Association).
ID: 1959355 · Report as offensive
Previous · 1 · 2 · 3 · 4 · Next

Questions and Answers : GPU applications : How to use multiple GPU's


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.