How does BOINC resource sharing actually work?

Questions and Answers : GPU applications : How does BOINC resource sharing actually work?
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Tom Miller
Volunteer tester
Avatar

Send message
Joined: 28 Nov 02
Posts: 768
Credit: 18,726,464
RAC: 22,937
United States
Message 1873437 - Posted: 16 Jun 2017, 22:06:23 UTC

I am chiefly a Seti guy. However, I would LIKE to also run a little bit of World Grid, a little Cosmology and some SetiBeta.

But so far, everything I have tried allows World Grid to sit front and center and NEVER appears to be going away.

I would even be willing to devote a couple of cores to the "other" projects just so they would stay out of the way of my main interest.

I understand if push comes to shove I will stop accepting new work from the "other" ones and manually let some work in but I would rather automate it if I can.

Currently I have 100 as resource share for Seti, and have re-set everyone else to 5.

Any advice?

Thank you,
Tom
"You are entitled to your own opinion but not to your own facts." Senator and Professor Patrick Moynihan
---
https://GalensonConsulting.WordPress.com
ID: 1873437 · Report as offensive
rob smithProject Donor
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 15193
Credit: 251,275,517
RAC: 322,190
United Kingdom
Message 1873558 - Posted: 17 Jun 2017, 12:12:41 UTC

BOINC uses a long term average work done system to allocate which projects to get work from. However some projects are badly behaved when it comes to sending work - you request a small amount and they will send you many days of work, instead.
The range is 0 to 1000 for each project, not as many think 0-100. Zero has a special meaning, something like "only get a task for this project if there is no work available from any of my non-zero projects" - this is very useful to keep crunching if your prime project has regular outages or shortages of work.
WCG can be VERY badly behaved, in my experience setting other than zero will result in it pouring work in your direction without a thought for your desired balance.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1873558 · Report as offensive
Profile Tom Miller
Volunteer tester
Avatar

Send message
Joined: 28 Nov 02
Posts: 768
Credit: 18,726,464
RAC: 22,937
United States
Message 1873564 - Posted: 17 Jun 2017, 12:46:07 UTC - in response to Message 1873558.  

WCG can be VERY badly behaved, in my experience setting other than zero will result in it pouring work in your direction without a thought for your
desired balance.


Thank you.
Sigh, exactly (WCG). What is your recommendation given my Seti orientation? 0 or 5?

Tom
"You are entitled to your own opinion but not to your own facts." Senator and Professor Patrick Moynihan
---
https://GalensonConsulting.WordPress.com
ID: 1873564 · Report as offensive
Profile Tom Miller
Volunteer tester
Avatar

Send message
Joined: 28 Nov 02
Posts: 768
Credit: 18,726,464
RAC: 22,937
United States
Message 1873961 - Posted: 19 Jun 2017, 8:15:30 UTC - in response to Message 1873558.  

WCG can be VERY badly behaved, in my experience setting other than zero will result in it pouring work in your direction without a thought for your desired balance.


I finally ran out of the "Cancer Marker" programs yesterday (I have disabled getting new tasks) and the rest of the World Community Grid programs are much more polite (so far) taking up only part of the available cores.

So it is possible that the issue I have been having with WCG always sitting front and center is an artifact of the "Cancer Marker" program. I know you can choose on the WCG page what apps you want. So I suppose I could de-select the Cancer Marker program.

Hmmm....

Tom
"You are entitled to your own opinion but not to your own facts." Senator and Professor Patrick Moynihan
---
https://GalensonConsulting.WordPress.com
ID: 1873961 · Report as offensive
rob smithProject Donor
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 15193
Credit: 251,275,517
RAC: 322,190
United Kingdom
Message 1873983 - Posted: 19 Jun 2017, 12:43:08 UTC

I doubt that it is an artifact of a specific sub-project. Over the years many people have had similar issues with quite a number of their applications. This all points to a poor server-side implementation of BOINC, probably with a number of the checks and balances turned off.

On a similar vein I noticed that one project does not do "check pointing", thus leaving both the user and the project vulnerable in the event of a problem a task, an application or a cruncher - a lot of BOINC features are in place for a good reason, namely protection of the data integrity, and not p'ing the user off. (This project happens to be of interest to me, having spent too many hours in PG studies trying to work out crystal structures from X-Ray diffraction data...)
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1873983 · Report as offensive
Profile Tom Miller
Volunteer tester
Avatar

Send message
Joined: 28 Nov 02
Posts: 768
Credit: 18,726,464
RAC: 22,937
United States
Message 1873999 - Posted: 19 Jun 2017, 14:19:06 UTC - in response to Message 1873983.  

I doubt that it is an artifact of a specific sub-project. Over the years many people have had similar issues with quite a number of their applications. This all points to a poor server-side implementation of BOINC, probably with a number of the checks and balances turned off.


Is there another meta project out there (like WCG) that is cancer/current disease of the year oriented and is better behaved?

Or am I just going to have to turn on the new tasks every other week or so and then turn them off?

Thanks,
Tom
"You are entitled to your own opinion but not to your own facts." Senator and Professor Patrick Moynihan
---
https://GalensonConsulting.WordPress.com
ID: 1873999 · Report as offensive
Profile Tom Miller
Volunteer tester
Avatar

Send message
Joined: 28 Nov 02
Posts: 768
Credit: 18,726,464
RAC: 22,937
United States
Message 1874003 - Posted: 19 Jun 2017, 14:31:52 UTC

Bob,

Do you have any idea how the "local" Boinc client chooses which tasks to schedule when?

I am pretty sure it uses the "resource" level / project and the "deadline date/time" but what else does the local scheduler appear to look at as it decides which task(s) to switch to (in the switch tasks every X hours) as well as individually as each task expires and another needs to be started?

Thanks,
Tom
"You are entitled to your own opinion but not to your own facts." Senator and Professor Patrick Moynihan
---
https://GalensonConsulting.WordPress.com
ID: 1874003 · Report as offensive
rob smithProject Donor
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 15193
Credit: 251,275,517
RAC: 322,190
United Kingdom
Message 1874018 - Posted: 19 Jun 2017, 15:48:10 UTC

If you aren't trying to micro manage it is a First-in First-out system. Unless it appears that tasks are going to time out before they are due to be run, in which case they are run in "panic mode" ahead of anything else.
Task switching is a black art - I've got a few CPN tasks which ran for a few days during one of the longer outrages and are now sitting waiting for their delivery to be more urgent than June NEXT year!
Basically, don't worry, tamper or interfere and the local BOINC manager will do a pretty good job.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1874018 · Report as offensive
Profile Tom Miller
Volunteer tester
Avatar

Send message
Joined: 28 Nov 02
Posts: 768
Credit: 18,726,464
RAC: 22,937
United States
Message 1875720 - Posted: 29 Jun 2017, 19:23:09 UTC - in response to Message 1874018.  
Last modified: 29 Jun 2017, 19:24:07 UTC

Bob,
I was watching one of my boxes that has Seti (100), SetiBeta (5), World Community Grid (1) and Rosseta (1) with GpuGrid (0) on gpu backup. The numbers in the () are the priorities. I read someplace you can set your highest number to 1,000 but am not sure of the use of it.

I have noticed that "usually" unless it has run out of tasks from that project, the scheduler seems to switch from one project to another rather than "mixing" them in together. e.g. 5 Seti, 2 SetiBeta, 2 WCG and 2 Rosseta (this one has 12 cores, usually 1 is running a Seti gpu task).

I do understand the different deadlines will skew things around.

Is there any settings on the priorities that will increase a tendency to "mix" the projects?

I would like my cpu to get less bored by giving it a real mix of work ;)

Thank you,
Tom
"You are entitled to your own opinion but not to your own facts." Senator and Professor Patrick Moynihan
---
https://GalensonConsulting.WordPress.com
ID: 1875720 · Report as offensive
rob smithProject Donor
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 15193
Credit: 251,275,517
RAC: 322,190
United Kingdom
Message 1875807 - Posted: 30 Jun 2017, 5:01:40 UTC

So easy to use 1000, just enter it - the figures are NOT percentages but relative values. So with SETI set to 1000, and two other projects each set to 1 the ratios of work to be done is 1000:1:1
As each project obeys (or disobeys) the rules as to how much work they will send, and their deadlines the only thing you can do is try different values, but be aware the ratios are not instant, but LONG TERM (weeks to months) in even approaching your desired ratios.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1875807 · Report as offensive
Harri Liljeroos
Avatar

Send message
Joined: 29 May 99
Posts: 103
Credit: 48,514,348
RAC: 26,558
Finland
Message 1875842 - Posted: 30 Jun 2017, 9:26:37 UTC

If I remember correctly, Boinc is designed to avoid the project mixing. This was introduced around the time when version 7 of Boinc was launched. Please correct me if I am wrong. So the idea is to download a full cache from one project and crunch it, then move to next project etc. You can "force" some mixing using app_config.xml files but that does not reflect on the work requests.
ID: 1875842 · Report as offensive
rob smithProject Donor
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 15193
Credit: 251,275,517
RAC: 322,190
United Kingdom
Message 1875849 - Posted: 30 Jun 2017, 10:03:58 UTC

If you get the ratios right, and give it enough time BOINC will allow several projects to run simultaneously. But it does appear to take a lot of luck and time to get the balance just right, only to have it disturbed by one of the selected projects throwing a wobbly. And SETI is not alone in its ability to throw wobblies.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1875849 · Report as offensive
OzzFan
Volunteer tester
Avatar

Send message
Joined: 9 Apr 02
Posts: 15578
Credit: 55,060,703
RAC: 26,167
United States
Message 1875858 - Posted: 30 Jun 2017, 10:23:38 UTC - in response to Message 1875842.  

If I remember correctly, Boinc is designed to avoid the project mixing. This was introduced around the time when version 7 of Boinc was launched. Please correct me if I am wrong. So the idea is to download a full cache from one project and crunch it, then move to next project etc. You can "force" some mixing using app_config.xml files but that does not reflect on the work requests.


This isn't correct at all. BOINC sees the world in resource shares and project debts and deadlines. It doesn't care what project the work is from, it crunches them all indiscriminately.
ID: 1875858 · Report as offensive
Harri Liljeroos
Avatar

Send message
Joined: 29 May 99
Posts: 103
Credit: 48,514,348
RAC: 26,558
Finland
Message 1875881 - Posted: 30 Jun 2017, 12:35:17 UTC - in response to Message 1875858.  

If I remember correctly, Boinc is designed to avoid the project mixing. This was introduced around the time when version 7 of Boinc was launched. Please correct me if I am wrong. So the idea is to download a full cache from one project and crunch it, then move to next project etc. You can "force" some mixing using app_config.xml files but that does not reflect on the work requests.


This isn't correct at all. BOINC sees the world in resource shares and project debts and deadlines. It doesn't care what project the work is from, it crunches them all indiscriminately.


OK, so I got it wrong. Thank you for pointing that out. Back to the drawing board.
ID: 1875881 · Report as offensive
Profile Tom Miller
Volunteer tester
Avatar

Send message
Joined: 28 Nov 02
Posts: 768
Credit: 18,726,464
RAC: 22,937
United States
Message 1875949 - Posted: 30 Jun 2017, 19:23:42 UTC - in response to Message 1875881.  

OK, so I got it wrong. Thank you for pointing that out. Back to the drawing board.


OHHHHH, <claps breast pantingly> an ART major........









Sorry, (I think) its Friday and I am a little giddy for some reason.....

(I believe the "pantingly" is actually a "Tom Swiftly")

Tom :D
"You are entitled to your own opinion but not to your own facts." Senator and Professor Patrick Moynihan
---
https://GalensonConsulting.WordPress.com
ID: 1875949 · Report as offensive
Profile Tom Miller
Volunteer tester
Avatar

Send message
Joined: 28 Nov 02
Posts: 768
Credit: 18,726,464
RAC: 22,937
United States
Message 1876516 - Posted: 2 Jul 2017, 23:22:39 UTC

I was told about this trick and this may help to "manage" the World Community Grid a little more granularly. You need to put an "app_config.xml" file in the WCG project directory with this in it. This will limit the number of applications for each WCG project. In my case, I didn't like seeing the "Cancer Marker" processing (which are very long) tying up every cpu core I had. So "mcm1" is 2. You can find the app name you need in the client_state.xml file.

<app_config>
<app>
<name>zika</name>
<max_concurrent>2</max_concurrent>
<gpu_versions>
<gpu_usage>1.0</gpu_usage>
<cpu_usage>1.0</cpu_usage>
</gpu_versions>
</app>
<app>
<name>mcm1</name>
<max_concurrent>2</max_concurrent>
<gpu_versions>
<gpu_usage>1.0</gpu_usage>
<cpu_usage>1.0</cpu_usage>
</gpu_versions>
</app>
<app>
<name>oet1</name>
<max_concurrent>2</max_concurrent>
<gpu_versions>
<gpu_usage>1.0</gpu_usage>
<cpu_usage>1.0</cpu_usage>
</gpu_versions>
</app>
<app>
<name>fahv</name>
<max_concurrent>2</max_concurrent>
<gpu_versions>
<gpu_usage>1.0</gpu_usage>
<cpu_usage>1.0</cpu_usage>
</gpu_versions>
</app>
<app>
<name>scc1</name>
<max_concurrent>2</max_concurrent>
<gpu_versions>
<gpu_usage>1.0</gpu_usage>
<cpu_usage>1.0</cpu_usage>
</gpu_versions>
</app>
<app>
<name>fahb</name>
<max_concurrent>2</max_concurrent>
<gpu_versions>
<gpu_usage>1.0</gpu_usage>
<cpu_usage>1.0</cpu_usage>
</gpu_versions>
</app>
<app>
<name>hst1</name>
<max_concurrent>2</max_concurrent>
<gpu_versions>
<gpu_usage>1.0</gpu_usage>
<cpu_usage>1.0</cpu_usage>
</gpu_versions>
</app>

</app_config>

"You are entitled to your own opinion but not to your own facts." Senator and Professor Patrick Moynihan
---
https://GalensonConsulting.WordPress.com
ID: 1876516 · Report as offensive
Profile Tom Miller
Volunteer tester
Avatar

Send message
Joined: 28 Nov 02
Posts: 768
Credit: 18,726,464
RAC: 22,937
United States
Message 1876597 - Posted: 3 Jul 2017, 13:17:41 UTC
Last modified: 3 Jul 2017, 13:19:41 UTC

Lets suppose you have a 4 core cpu with hyper threading and only want to let World Community Grid use say upto 4 cores/threads? Additional information from the thread I got the my last post from says this works. I have implemented it and Boinc is not complaining and I am running at or less than 4 WCG tasks. I have left the "gpu_versions" in even though as of the moment none of the WCG sub-projects has any gpu-based processing.

Place the following after that last </app>
</app>
<project_max_concurrent>4</project_max_concurrent>
</app_config>


Here is the whole app_config.xml file for the WCG project directory.


<app_config>
<app>
<name>zika</name>
<gpu_versions>
<gpu_usage>1.0</gpu_usage>
<cpu_usage>1.0</cpu_usage>
</gpu_versions>
</app>
<app>
<name>mcm1</name>
<gpu_versions>
<gpu_usage>1.0</gpu_usage>
<cpu_usage>1.0</cpu_usage>
</gpu_versions>
</app>
<app>
<name>oet1</name>
<gpu_versions>
<gpu_usage>1.0</gpu_usage>
<cpu_usage>1.0</cpu_usage>
</gpu_versions>
</app>
<app>
<name>fahv</name>
<gpu_versions>
<gpu_usage>1.0</gpu_usage>
<cpu_usage>1.0</cpu_usage>
</gpu_versions>
</app>
<app>
<name>scc1</name>
<gpu_versions>
<gpu_usage>1.0</gpu_usage>
<cpu_usage>1.0</cpu_usage>
</gpu_versions>
</app>
<app>
<name>fahb</name>
<gpu_versions>
<gpu_usage>1.0</gpu_usage>
<cpu_usage>1.0</cpu_usage>
</gpu_versions>
</app>
<app>
<name>hst1</name>
<gpu_versions>
<gpu_usage>1.0</gpu_usage>
<cpu_usage>1.0</cpu_usage>
</gpu_versions>
</app>
<project_max_concurrent>4</project_max_concurrent>
</app_config>
End of code.  Everything below this is considered to be comments by Boinc.

Example for limiting for a single project/task within the app_config.xml
<name>zika</name>
<max_concurrent>2</max_concurrent>
<gpu_versions>
<gpu_usage>1.0</gpu_usage>
<cpu_usage>1.0</cpu_usage>
</gpu_versions>
</app>

Place this after the last </app> to limit the total instances running under that app_config.xml
</app>
<project_max_concurrent>4</project_max_concurrent>
</app_config>


HTH,
Tom
"You are entitled to your own opinion but not to your own facts." Senator and Professor Patrick Moynihan
---
https://GalensonConsulting.WordPress.com
ID: 1876597 · Report as offensive
rob smithProject Donor
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 15193
Credit: 251,275,517
RAC: 322,190
United Kingdom
Message 1876769 - Posted: 4 Jul 2017, 12:04:52 UTC - in response to Message 1876597.  

Tom,
the place to ask about wcg applications is their help desk not the seti help desk.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1876769 · Report as offensive
Profile Tom Miller
Volunteer tester
Avatar

Send message
Joined: 28 Nov 02
Posts: 768
Credit: 18,726,464
RAC: 22,937
United States
Message 1876822 - Posted: 5 Jul 2017, 3:26:55 UTC - in response to Message 1876769.  

Tom,
the place to ask about wcg applications is their help desk not the seti help desk.


Bob,
I refer you to the thread title "How does BOINC resource sharing actually work?" Since the topic is BOINC rather than Seti only, I claim this is completely on topic. In addition, I am posting an answer to a question I had about "managing Boinc projects in a more fine grained way" for years.

Since I have now posted a working example of exactly how to control the total number of tasks that are active a Boinc/Non-Seti project as well as the same information for Boinc/Seti I don't really have anything further to say.

I will say that the details I have posted here are not very explicit in the documentation which is why I am posting them.

Respectfully,
Tom
"You are entitled to your own opinion but not to your own facts." Senator and Professor Patrick Moynihan
---
https://GalensonConsulting.WordPress.com
ID: 1876822 · Report as offensive

Questions and Answers : GPU applications : How does BOINC resource sharing actually work?


 
©2017 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.