Work Unit Execution order (Bug?).


log in

Advanced search

Questions and Answers : Wish list : Work Unit Execution order (Bug?).

Author Message
walybare
Send message
Joined: 2 Mar 04
Posts: 5
Credit: 122,359
RAC: 0
United States
Message 701066 - Posted: 18 Jan 2008, 6:56:15 UTC
Last modified: 18 Jan 2008, 7:17:44 UTC

Is there a way to change the order in which the WU are selected for execution. WU should be selected for execution by the shortest "Report Deadline" - not by order received.

I get WU that are due two months down the line and then I get a bunch that are due in a week or a few days. But the WU that are due two months down the road (which were received first) are consistently selected before the shorter deadline WU's.

Since I don't leave my computer on all the time, some WU with short "Report Deadline" may not get finished in time.

Please change the execution order to select the WU with the shortest "Report Deadline" first. I see no reason to select WU using the order it was received.

PS. I know I can suspend WU so the other ones will execute, but I don't want to manually do this all the time.
____________

Profile Ageless
Avatar
Send message
Joined: 9 Jun 99
Posts: 12467
Credit: 2,690,735
RAC: 1,178
Netherlands
Message 701078 - Posted: 18 Jan 2008, 7:56:35 UTC - in response to Message 701066.

Please change the execution order to select the WU with the shortest "Report Deadline" first. I see no reason to select WU using the order it was received.

Ha no? So what if you also attach to a project that has 24 hour (or less) deadlines? You'd only be running those tasks then, until BOINC figures it can't finish your other tasks in time and will start switching (between and) to them exclusively.

For the rest of the answers, read this thread.
____________
Jord

Fighting for the correct use of the apostrophe, together with Weird Al Yankovic

Profile Pooh Bear 27
Volunteer tester
Avatar
Send message
Joined: 14 Jul 03
Posts: 3221
Credit: 2,640,394
RAC: 383
United States
Message 701111 - Posted: 18 Jan 2008, 10:59:44 UTC

This question is asked often (just asked in the Windows forum yesterday).

Leave BOINC alone and do it's job. It will figure things out and try and make sure that your work gets in, in time and will adjust the jobs it receives and necessary.

First in, first out is best, and will set a high priority to something it thinks it might miss.

Ageless made a good point, also. If all the short ones always ran, the longer ones would then have issues getting in on time.

____________

OzzFan
Volunteer tester
Avatar
Send message
Joined: 9 Apr 02
Posts: 13704
Credit: 31,710,881
RAC: 12,598
United States
Message 701237 - Posted: 18 Jan 2008, 20:19:15 UTC - in response to Message 701066.

Please change the execution order to select the WU with the shortest "Report Deadline" first. I see no reason to select WU using the order it was received.

PS. I know I can suspend WU so the other ones will execute, but I don't want to manually do this all the time.


There's no reason for this. BOINC's built in scheduler does a great job and has never missed a deadline on any of my systems in the last three or four years running it.

Bottom line is, don't micromanage BOINC. Don't try to do the work for BOINC by suspending workunits or otherwise modifying the work order. Let BOINC do its job and trust it. It will do fine. No human intervention required.
____________

walybare
Send message
Joined: 2 Mar 04
Posts: 5
Credit: 122,359
RAC: 0
United States
Message 701268 - Posted: 18 Jan 2008, 21:26:06 UTC - in response to Message 701237.

This question is asked often (just asked in the Windows forum yesterday).


Why do you think this question is asked so much?! Because, the current algorithm is NOT adequate for people who turn their computers off for ANY period of time.


Leave BOINC alone and do it's job. It will figure things out and try and make sure that your work gets in, in time and will adjust the jobs it receives and necessary.


I would like to leave it alone! That is what I am trying to get accomplished here. But, how is BOINC going to figure it out if the computer is off?! Duh! The point is - the algorithm used works fine if your computer runs 24/7 (or nearly 24/7) - mine, as well as many others, does not!


First in, first out is best, and will set a high priority to something it thinks it might miss.


FIFO is NOT the best! If I get a bunch of WU's with short "Report-Deadlines" and then shut my computer off for a few days, then the WU's with short "Report-Deadlines" will have been queued up and not executed in a timely manner. Then BOINC has to run in "high-priority" mode to complete them. Usually, the short "Report-Deadline" WU's could have completed or almost completed at the time I turned my computer off.


Ageless made a good point, also. If all the short ones always ran, the longer ones would then have issues getting in on time.


This is NOT a good point! That's a load of crap! Because if everyone is executing the WU's with shortest "Report-Date" first, the server will have exhausted the short turn-around WU's more quickly. Also, BOINC should easily be able to calculate the amount of time needed to complete WU with longer run times. And, it should not download shorter "Report-Deadline", if the existing WU cannot be completed by its "Report-Deadline". I don’t believe it is necessary for that calculation, because, as I stated, the server will exhaust the short “Report-Deadlines” quickly, however, it could be implemented as an additional check to ensure the timely execution. But it does beg the question, "Why is the server requiring such a short turn-around?" Is the world going to end if a particular WU with a 2-day turn-around doesn't get completed in 2 days? Why doesn't the server always provide a least 2-weeks, 3-weeks, 1-month, or something longer? You don't need to answer those last two questions - it probably political anyways. My goal here is to fix and to get BOINC to execute efficiently.


There's no reason for this. BOINC's built in scheduler does a great job and has never missed a deadline on any of my systems in the last three or four years running it.

OzzFan, that’s because you leave your computer(s) on 24/7! For you the current algorithm is fine.


Bottom line is, don't micromanage BOINC…


I’m not trying to micromanage, I trying to get BOINC to execute more efficiently for those who shut their computer down!

Another, (Additional Options) is that you can also add 2 new parameters in the preferences.
1). “Min Report Days” - The user and/or computer will only download WU's with a minimum of "x" days for the "Report-Deadline". So, if I know I need to go out of town in a couple of days and will have to shut my computer down for ... say 7 days, I can change this parameter and it would not download any WU's that may need to be completed before I get back. Again, the WU's with the shortest "Report-Deadline” must be selected to execute first before I shut my computer down.

2). “Num Days to Complete Before Deadline” - BOINC can use this parameter to calculate each WU’s estimated completion date, and check if each WU will complete by the number of days BEFORE the “Report-Deadline”. It should not download any other WU’s with a shorter “Report Deadline” than the WU’s that cannot be completed using this parameter (number of day before “Report Deadline”). This check will need to execute before each download of WU’s. This will ensure that WU’s are completed in a timely manner and provide a “buffer” and more “control” for those who turn their computers off for a period of time.

Parameter 1 should be easy to implement and give the user better control how to download appropriate WU’s. Parameter 2 might be a little more difficult, but it does provide a check to ensure that WU’s are completed in a timely manner. Though this parameter is probably not necessary because the short “Report-Deadline” will be executed from the server quickly. But, in any case BOINC still needs to select WU's with the shortest "Report-Deadline" first – ALWAYS!

Ageless and Pooh_Bear_27, have you ever tested this method? No? I didn’t think so. So why not “TEST” this method before you try to criticize. I see your both are “volunteer testers” – so test it! Do you two program BOINC? If not, then let one of the programmers respond to this message.

Send me the BOINC code (Linux version) and I’ll fix it. I have a degree in Computer Science and have over 25 years (yes, I'm a crotchety Olde Phart) as a computer analyst. So, I DO know what I'm talking about.

____________

OzzFan
Volunteer tester
Avatar
Send message
Joined: 9 Apr 02
Posts: 13704
Credit: 31,710,881
RAC: 12,598
United States
Message 701279 - Posted: 18 Jan 2008, 21:52:48 UTC - in response to Message 701268.

OzzFan, that’s because you leave your computer(s) on 24/7! For you the current algorithm is fine.


You may want to double check your facts before making assumptions like this. As a matter of fact, not all of my systems run 24/7. My laptop (that I am typing this on) actually only runs 5/5 (5 hours a day, 5 days a week).

A few of my other systems don't allow BOINC to run while user is active, thus lowering the amount of crunching time available (the same effect as shutting down your system).

None of my systems have missed a deadline.

There are additional stats/calculations in place that take all this into account to ensure BOINC doesn't miss any deadlines. The system works. I'll bet if you leave it alone, you'll see this in action.
____________

walybare
Send message
Joined: 2 Mar 04
Posts: 5
Credit: 122,359
RAC: 0
United States
Message 701296 - Posted: 18 Jan 2008, 22:32:22 UTC - in response to Message 701279.


Risks: If you never try anything new, you'll miss out on many of life's great disappointments.


My point is your computer are runing with minimal down time. Why not try something new?

____________

OzzFan
Volunteer tester
Avatar
Send message
Joined: 9 Apr 02
Posts: 13704
Credit: 31,710,881
RAC: 12,598
United States
Message 701301 - Posted: 18 Jan 2008, 22:44:48 UTC - in response to Message 701268.
Last modified: 19 Jan 2008, 0:56:16 UTC

Ageless and Pooh_Bear_27, have you ever tested this method? No? I didn’t think so. So why not “TEST” this method before you try to criticize. I see your both are “volunteer testers” – so test it! Do you two program BOINC? If not, then let one of the programmers respond to this message.

Send me the BOINC code (Linux version) and I’ll fix it. I have a degree in Computer Science and have over 25 years (yes, I'm a crotchety Olde Phart) as a computer analyst. So, I DO know what I'm talking about.


Since you obviously need a more detailed answer on the BOINC scheduler than any of us can provide you with, I've PM'd John McLeod VII, who wrote the BOINC scheduler himself (and has done a fine job at it if I do say so myself).

The bigger picture that you are missing here is that there's more to BOINC than simply seeing deadlines. There's a "% of time BOINC is allowed to run" stat and a "% of time BOINC client is running" stat that all get reduced the less your computer runs. There is also a "CPU efficiency" and "Task duration correction factor" used to signify one CPU's superior architecture over another. All these stats are used to calculate whether a computer runs 24/7 or not, and if not, how much that computer can handle. If BOINC calculates that a computer isn't powered on enough to finish a WU before it's deadline, it enters something called EDF (Earliest Deadline First) mode to process any workunits that may not complete in time. Most of the time, EDF isn't necessary, nor should the manner in which workunits get processed since most of the deadlines are long enough to compensate for this.

Bottom line is there is already something in place for what you are suggesting (computers that can't finish their work time because they don't run 24/7). This process has been in place for a while (and gets refined from time to time), but someone looking from the outside in (doesn't know the inner workings of BOINC very well) wouldn't know this easily.

No one is doubting your skills as a programmer. All we're saying is that you're not the only talented programmer out there and you're not the only one who doesn't run their computer 24/7, so this isn't an issue that the developers would have missed in the four or five years of BOINC's existence. There is plenty of experience built into BOINC that takes care of what you're asking for.
____________

OzzFan
Volunteer tester
Avatar
Send message
Joined: 9 Apr 02
Posts: 13704
Credit: 31,710,881
RAC: 12,598
United States
Message 701302 - Posted: 18 Jan 2008, 22:47:04 UTC - in response to Message 701296.


Risks: If you never try anything new, you'll miss out on many of life's great disappointments.


My point is your computer are runing with minimal down time. Why not try something new?


Why "try something new" when there's already something in place that takes care of what you're asking for? It's like air: just because you can't see it, doesn't mean it's not there.
____________

Profile Ageless
Avatar
Send message
Joined: 9 Jun 99
Posts: 12467
Credit: 2,690,735
RAC: 1,178
Netherlands
Message 701330 - Posted: 19 Jan 2008, 0:15:54 UTC - in response to Message 701268.


Ageless made a good point, also. If all the short ones always ran, the longer ones would then have issues getting in on time.
This is NOT a good point! That's a load of crap! Because if everyone is executing the WU's with shortest "Report-Date" first, the server will have exhausted the short turn-around WU's more quickly.

You're thinking that BOINC == Seti only.

But BOINC is a whole lot more. You can attach to 70+ projects. Now go figure if it's still crap what I said. :-)

Ageless and Pooh_Bear_27, have you ever tested this method? No? I didn’t think so. So why not “TEST” this method before you try to criticize. I see your both are “volunteer testers” – so test it!

I got that tag because I tested Seti Beta in the past. There's no way to get rid of that tag, even though I wouldn't mind it being dropped.

So before you criticize, first check what the tag might stand for.

Do you two program BOINC? If not, then let one of the programmers respond to this message.

I've compiled my own BOINC versions in the past. I'm Alpha testing the new BOINC. I can say I am very good at debugging the code and giving help to people with the strangest error messages, by going through the code and see what it does that it gives that error message. Heck, I even started the BOINC FAQ Service.

If that's not good enough for you, tough.

Send me the BOINC code (Linux version) and I’ll fix it.

http://boinc.berkeley.edu/trac/wiki/CompileClient
Go download it yourself. It's Open Source. If you think you can do better, then by all means go ahead and do better.

As for your conclusion that it is BOINC's fault for downloading way too much work... say I set my connect to to 5 days, plus an additional extra work of 5 days. That gives me just under 10 days worth of work.

Now I shut down my computer for 9 days. Is it BOINC's fault it gave me too much work to be able to do all that in the 1 day my computer is on? Or do you think it's better that You adjust your connect to/additional work settings, so it won't get so much work in the first place?

Go get around that analogy.
____________
Jord

Fighting for the correct use of the apostrophe, together with Weird Al Yankovic

John McLeod VII
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 15 Jul 99
Posts: 24806
Credit: 530,050
RAC: 324
United States
Message 712564 - Posted: 14 Feb 2008, 19:07:17 UTC

Try the following I would be interested to see if it works.
If you do not have a global_prefs_override.xml file in the BOINC directory, create one that looks like (this assumes that your host will be off for 5 days.):

<global_preferences>
<work_buf_min_days>5.000000</work_buf_min_days>
<work_buf_additional_days>-4.900000</work_buf_additional_days>
</global_preferences>

If you already have one, change the lines above to match.

The scheduler really can cope if you tell it what you are doing.

I just started trying this to see what happens.
____________


BOINC WIKI

BigJohn
Send message
Joined: 16 Mar 03
Posts: 9
Credit: 1,581,335
RAC: 0
United States
Message 715839 - Posted: 20 Feb 2008, 17:43:51 UTC - in response to Message 701066.

Is there a way to change the order in which the WU are selected for execution. WU should be selected for execution by the shortest "Report Deadline" - not by order received.

I get WU that are due two months down the line and then I get a bunch that are due in a week or a few days. But the WU that are due two months down the road (which were received first) are consistently selected before the shorter deadline WU's.

Since I don't leave my computer on all the time, some WU with short "Report Deadline" may not get finished in time.

Please change the execution order to select the WU with the shortest "Report Deadline" first. I see no reason to select WU using the order it was received.

PS. I know I can suspend WU so the other ones will execute, but I don't want to manually do this all the time.


____________

BigJohn
Send message
Joined: 16 Mar 03
Posts: 9
Credit: 1,581,335
RAC: 0
United States
Message 715840 - Posted: 20 Feb 2008, 17:48:17 UTC

I agree.

The order of processing should be by date.
The due date should determine order of processing.

BigJohn
____________

Profile Ageless
Avatar
Send message
Joined: 9 Jun 99
Posts: 12467
Credit: 2,690,735
RAC: 1,178
Netherlands
Message 715845 - Posted: 20 Feb 2008, 18:03:57 UTC - in response to Message 715840.

The due date should determine order of processing.

What then if I have a CPDN model with a deadline of somewhere in 2009? Is it only allowed to process somewhere directly before that deadline, thereby me missing my deadline because of the sheer size of the model?

If it is for Seti only, do you really think it is worth it that you run them by deadline, not on a first in/first out basis? Do you really think that a deadline will matter on the question of if the possible signal is in the tasks you run?

Or is it solely reasoned on the basis that you may get your credits earlier if BOINC sorts the tasks on deadline and queues them that way for calculation?

I'll repeat the question, what if you're also attached to a project that has very short deadlines, something like 3 days? If you are attached to a project like that as well, you're only running that project's tasks. Then what happens to your Seti tasks?

Remember please, Seti <> BOINC. In other words, BOINC isn't just a manager for Seti, it's a manager for some 70+ projects, all with their own deadline restrictions.
____________
Jord

Fighting for the correct use of the apostrophe, together with Weird Al Yankovic

Robert Smith
Avatar
Send message
Joined: 15 Jan 01
Posts: 266
Credit: 62,123
RAC: 25
United Kingdom
Message 715988 - Posted: 20 Feb 2008, 21:43:54 UTC - in response to Message 701268.

I would like to leave it alone! That is what I am trying to get accomplished here. But, how is BOINC going to figure it out if the computer is off?! Duh! The point is - the algorithm used works fine if your computer runs 24/7 (or nearly 24/7) - mine, as well as many others, does not!

OzzFan, that’s because you leave your computer(s) on 24/7! For you the current algorithm is fine.

I’m not trying to micromanage, I trying to get BOINC to execute more efficiently for those who shut their computer down!

Interesting...

My little Sempron rarely runs for more than a couple of hours a day, sometimes it's switched off for days at a time. I've never changed any of the default settings and I won't waste my valuable time second-guessing the scheduler... So do I miss deadlines? I'll give you a clue: My Mac hasn't been switched on yet this month - it won't miss its deadline either.

So how exactly is the scheduler broken?
Unneeded micromanaging + unnecessarily large work cache = missed deadlines

Alinator
Volunteer tester
Send message
Joined: 19 Apr 05
Posts: 4178
Credit: 4,647,982
RAC: 0
United States
Message 717416 - Posted: 23 Feb 2008, 16:08:08 UTC - in response to Message 712564.

Try the following I would be interested to see if it works.
If you do not have a global_prefs_override.xml file in the BOINC directory, create one that looks like (this assumes that your host will be off for 5 days.):

<global_preferences>
<work_buf_min_days>5.000000</work_buf_min_days>
<work_buf_additional_days>-4.900000</work_buf_additional_days>
</global_preferences>

If you already have one, change the lines above to match.

The scheduler really can cope if you tell it what you are doing.

I just started trying this to see what happens.


Hmmm...

Interesting, I had never considered what putting a negative value in for the cache override setting would do (I guess I just assumed it was illegal).

So what should it do in this case? My reasoning is the CC would fetch 1 task at a time five days in advance.

Alinator

John McLeod VII
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 15 Jul 99
Posts: 24806
Credit: 530,050
RAC: 324
United States
Message 718343 - Posted: 25 Feb 2008, 14:12:37 UTC - in response to Message 717416.

Try the following I would be interested to see if it works.
If you do not have a global_prefs_override.xml file in the BOINC directory, create one that looks like (this assumes that your host will be off for 5 days.):

<global_preferences>
<work_buf_min_days>5.000000</work_buf_min_days>
<work_buf_additional_days>-4.900000</work_buf_additional_days>
</global_preferences>

If you already have one, change the lines above to match.

The scheduler really can cope if you tell it what you are doing.

I just started trying this to see what happens.


Hmmm...

Interesting, I had never considered what putting a negative value in for the cache override setting would do (I guess I just assumed it was illegal).

So what should it do in this case? My reasoning is the CC would fetch 1 task at a time five days in advance.

Alinator

The settings will possibly keep a cache of 0.1 days, and attempt to ensure that work is completed and returned 5 days in advance of the report deadline. If that does not work, then it will cache 5 days of work and return the work 5 days in advance of the report deadline.
____________


BOINC WIKI

Questions and Answers : Wish list : Work Unit Execution order (Bug?).

Copyright © 2014 University of California