ausymark wrote:

I'm going to suggest what I think might be a simple system that may help matters.
...
...
So does any of the above have merit and is it simple enough to be easily implemented?

Cheers

Mark

Yes, it definitely has merit, but I don't think any changes relating to quota management are likely to be easily implemented.

The existing system attempts to do some of what you suggest. If you look at the "Applications details" for a given host (for instance, yours would be here), you'll already find fields for "Average turnaround time" and "Max tasks per day", for each application that can be sent to that host.

In general, I suspect your proposal would probably be more restrictive on new hosts, but perhaps overly generous for existing ones. For new hosts, the initial value for "Max tasks per day" is 33. I think that number is much too high as it is, notwithstanding the fact that the value in that field is somewhat misleading. As I recall, the project applies a multiplier to that value to calculate the true maximum, and my recollection is that the multiplier is 8. (If anyone can point to someplace that would verify or refute that number, please do.)

That initial setting is then incremented by 1 for every task that is completed and validated, so it can get to be quite a huge number for reliable hosts. Theoretically, the number is decreased for each time a task is marked as an Error or Invalid (the latter being somewhat questionable). It can never drop lower than 1, but with the multiplier, and with the scheduler having multiple applications to choose from for hosts running stock), that number is kind of meaningless.

Even for hosts with huge "Max tasks" numbers, there are already two limiting factors in play. One is the maximum of 100 CPU tasks per host and 100 tasks per GPU. The other is the "Store at least nn days of work" and "Store up to an additional nn days of work" combination, controllable through user/host Preference settings. Each of these has a maximum value of 10 days which, even in extreme cases, should limit a host to no more work than it can process in 20 days. If that value is less than the maximum allowed by the other limits, that's where it should be capped. Otherwise, the other limits should come into play and, in the case of the most productive hosts, should be far less than 20 days. So, given all that, why a host with a greater than 20-day average turnaround is able to download more than 1 or 2 tasks at a time is beyond me. :^)

the issue with using WU average turn around time is that it is something that happens after potentially lots of unneeded WU's are allocated to a host, especially true for slow hosts.

It takes some serious messing around with a system for it to get more work than it can handle. By default, the Manager is very conservative in it's requests for work until it has a good idea of what the actual turnaround times are- and that depends on the run time for the WUs, the number of projects the system is working on, as well as the amount of time the system is powered up & it is able to process work.

I dont know how/if seti uses turnaround time atm, if it does how is it used to allocate buffers and determine WU deadlines?

WU deadlines are set by the project. How large a buffer (cache) a system uses can be changed by the user- I don't know what the default values are.
The actual processing times of WUs are used to determine the Estimated run time. Once those values settle down (the original estimates are always very high) then the Manager is generally able to download the appropriate amount of work to keep the cache at it's set level- up to the server side maximum limits.

---

Grant
Darwin NT

Brent Norman wrote:

OK, I revisit to an old subject ... Turn resend lost tasks back on. I remember it was said it was causing excess database load which is why it is off, but forget what the bottle neck was - hardware or software. Would SSDs in the array help or is it the overall system that can't handle the throughput?

Same goes for the daily, monthly, yearly stats and other things. They just keep shutting things off as the database size increases. So what is needed to fix the real problems with the huge database?

I think the issue would likely be with the query that would necessarily run against the DB for every scheduler request just to determine if any lost tasks were present for the host making the request. The scheduler would have to be able to match up every task listed in the "<other_results>" section of the scheduler request with the tasks that are stored on the DB for that host, in order to determine if there are any "lost" ones that need to be resent. That probably wasn't a big deal back when an average host only carried a few dozen tasks (or less). But with so many hosts now carrying task limits for multiple GPUs, I shudder to think what the overhead would be every 5 minutes when my 4-GPU host sends in a scheduler request. Multiply that by all hosts out there carrying their task limits and I can understand why it might bring the DB and server to its knees.

However, by having that feature turned off, it allows ghosts to accumulate in the DB, though how many that might be is anybody's guess. I, for one, try to be responsible in retrieving ghosts if I know about them, and especially if I'm responsible for having created them in the first place.....which I usually am. :^(

On the other hand, I've got to think that there must be some middle ground, where the lost task resend process could be turned back on for short periods when database access is near its low point, just to see if a significant number of lost tasks can find their way home again. Alternatively, just comparing a count of the <other_results> in the scheduler request with what the server thinks should be there could be a low-impact preliminary test. Only if a discrepancy was found would a full comparison be necessary.

. Once those values settle down

They may never.
On one host of mine they never settle down. It runs 45% reserved E@H on at GTX1060 and all the CPU on E@H also. The rest of the time it runs S@H on the GPU. The CPU and GPU run times seem to averaged between the CPU and GPU per project. It does not take into account the app or the processor. The GPU can processes at the rate of 4 wk units per hr, the cpu a bit less than 6 hrs per wk unit. I see est run times for the GPU to vary between 1 and 2.5 hrs. When it reports a CPU task the est run time for the cached GPU work goes way up, Boinc often thinks my 1.5 day cache is full even though it is not.

---

I guess the simplest test for a need to check would be ... if(#InProgress/#Devices>100, CheckForResends, NormalRequest) that should definitely cut down server load for checking since those values should(??) already be used in the calculation for # to send.

betreger wrote:

. Once those values settle down

They may never.
On one host of mine they never settle down. It runs 45% reserved E@H on at GTX1060 and all the CPU on E@H also. The rest of the time it runs S@H on the GPU. The CPU and GPU run times seem to averaged between the CPU and GPU per project. It does not take into account the app or the processor. The GPU can processes at the rate of 4 wk units per hr, the cpu a bit less than 6 hrs per wk unit. I see est run times for the GPU to vary between 1 and 2.5 hrs. When it reports a CPU task the est run time for the cached GPU work goes way up, Boinc often thinks my 1.5 day cache is full even though it is not.

In addition to factoring in project share, the average turnaround time also depends on how many hours a day a system runs. If it's running 24/7, then the average should be close to actual. But if a machine only crunches, for instance, an average of 12 hours per day, then the average will be double the actual. In other words, if the average actual run time is 6 hours, the turnaround will average out to .5 days because the host is only returning 2 tasks per 24-hours, due to the 12-hour per day downtime. That sort of analogy probably also applies if you subtract the number of hours an alternate project is running from the base 24-hour day. It should, however, differentiate between CPU and GPU apps, just as it does on the Application Details page, but hey, some BOINC calculations are just inexplicable. That's all there is to it!

Brent Norman wrote:

I guess the simplest test for a need to check would be ... if(#InProgress/#Devices>100, CheckForResends, NormalRequest) that should definitely cut down server load for checking since those values should(??) already be used in the calculation for # to send.

I'm not sure. The Scheduler certainly looks to see if the host has reached its limit on tasks in progress, but I assume that's based on the tasks carried in the scheduler request. However, I believe the Scheduler primarily looks at the number of seconds of work requested and just decrements that figure as it pulls tasks off the feeder until either the request is satisfied or the task limit is reached. I don't know that it ever needs to see how many tasks are already recorded as "In Progress" in the DB for the host. Somebody like Richard could probably pin that down better than I can.

ausymark wrote:

I'm going to suggest what I think might be a simple system that may help matters.
...
...
So does any of the above have merit and is it simple enough to be easily implemented?

Cheers

Mark

Yes, it definitely has merit, but I don't think any changes relating to quota management are likely to be easily implemented.

The existing system attempts to do some of what you suggest. If you look at the "Applications details" for a given host (for instance, yours would be here), you'll already find fields for "Average turnaround time" and "Max tasks per day", for each application that can be sent to that host.

In general, I suspect your proposal would probably be more restrictive on new hosts, but perhaps overly generous for existing ones. For new hosts, the initial value for "Max tasks per day" is 33. I think that number is much too high as it is, notwithstanding the fact that the value in that field is somewhat misleading. As I recall, the project applies a multiplier to that value to calculate the true maximum, and my recollection is that the multiplier is 8. (If anyone can point to someplace that would verify or refute that number, please do.)

That initial setting is then incremented by 1 for every task that is completed and validated, so it can get to be quite a huge number for reliable hosts. Theoretically, the number is decreased for each time a task is marked as an Error or Invalid (the latter being somewhat questionable). It can never drop lower than 1, but with the multiplier, and with the scheduler having multiple applications to choose from for hosts running stock), that number is kind of meaningless.

Even for hosts with huge "Max tasks" numbers, there are already two limiting factors in play. One is the maximum of 100 CPU tasks per host and 100 tasks per GPU. The other is the "Store at least nn days of work" and "Store up to an additional nn days of work" combination, controllable through user/host Preference settings. Each of these has a maximum value of 10 days which, even in extreme cases, should limit a host to no more work than it can process in 20 days. If that value is less than the maximum allowed by the other limits, that's where it should be capped. Otherwise, the other limits should come into play and, in the case of the most productive hosts, should be far less than 20 days. So, given all that, why a host with a greater than 20-day average turnaround is able to download more than 1 or 2 tasks at a time is beyond me. :^)

Thanks for that Jeff

Knowing the above it might be a 'simple' matter of tweaking the existing system a little. I am thinking the 33 Max Tasks figure and the multiply by 8 figure are there to create a buffer in the first case and compensate for a few days extra storage - to buffer any initial user requirement to download 'xxx' days of extra work. These figures were probably created for when accessing seti was via dial-up modem so that users could disconnect and crunch away (I used to be one of those users ;) ).

By the sounds of it the 100 tasks per CPU and 100 tasks per GPU are there for two reasons:
1) the Max Tasks field isn't being calculated properly and
2) just in case there is a runaway of invalids for whatever reason

The Average Turnaround Time figure is obviously calculated over several validated work units. This figure should increase if more WU's are sent to the host, and decrease with fewer - assuming the computer is operating consistently

So the question really becomes, "how many WU should be allowed to be dispatched in total taking into account the users buffer options?".

BOINC knows the performance of the host so the initial WU dispatch should be whatever that host is calculated to be able process in a 24 hour period multiplied by how many extra hours/days the user is requesting for their buffer. This should be the initial "Max Tasks" number/limit. This also allows us to calculate an initial Average Turnaround Time. Most likely this initial distribution of WU's will be more than the host will process as most aren't running 24/7.

If this is the case then the Max Tasks total should be reduced until the Average Turn Around time is close to the original calculated Average Turn Around time. This must also allow for X days of buffer. So if a 3 day buffer then Average Turnaround Time might be 4 days + 3days of buffer which equals 7 days Average Turn Around Time.

So, for example, if the Average Turnaround Time is 30% longer than expected, then reduce the number of WU by 30%. Though this reduction may have to be done in small amounts over time, perhaps 2% per day, that way it will take 15 days to slowly correct itself. The reverse would occur if Average Turnaround Time was smaller than expected. (As I write that, and knowing how slowly seti reacts to changes, makes me wonder if this isnt how its currently done lol).


This however doesn't account for failed/invalid WU's that are processed very fast. My gut instinct is that any result that is at > 85% smaller variance to the Average Turnaround Time then that WU should be excluded from the calculations. If Host hardware has changed then BOINC should notify seti and the initial startup routine would be initiated again.

Deadline dates with the above should be Average Turnaround Time + Number of Buffer Days + x days that the database people think are fair, I am guessing somewhere between 7 and 10 days, or posibly a multiplication of Average Turnaround Time + Number of Buffer Days.

Anyway, Im just thinking aloud. These thoughts may be not be new, IDK, just adding some ideas into the mix.

Cheers

Mark

---

That should probably be ... if(#InProgress/#Devices>ServerLimit) then CheckForResends.

The Reached Limit message would be coming from the client saying,
I have 100 CPU tasks and want more for 10 days, and I have 90 GPU tasks and want more for 10 days.
Server says, none for CPU, here is 10 GPU tasks ..
or (as we often see)
You have reached your limit (for CPU[as defined server limit]) and Server has No Tasks Available (for your GPU) ... a message for each device.

It must be the client reporting how many tasks it has for the server to know how many more will reach the ServerLimit and takes that as face value without checking if you have 6000 InProgress.

Then of course there is the time limit constraint formula in that mix too.

We know the server currently has the ability to check for resends since that is how we recover ghosts - by forcing an upload Failure/TimeOut. That must set a server side flag for that ID - I know we're out of sync, so I better check ...

It might be interesting to track what Team the people with 6000 tasks are from to see if there is a link to a 'Group' using modified code ...

This is off topic from deadlines, but is also a major cause of bloating.

This is off topic from deadlines, but is also a major cause of bloating.

Which is why I added my 2c worth, running on the assumption that if the underlying dispatch system works better then improved deadlines would be a likely outcome ;)

Cheers

---

Brent Norman wrote:

That should probably be ... if(#InProgress/#Devices>ServerLimit) then CheckForResends.

The Reached Limit message would be coming from the client saying,
I have 100 CPU tasks and want more for 10 days, and I have 90 GPU tasks and want more for 10 days.
Server says, none for CPU, here is 10 GPU tasks ..
or (as we often see)
You have reached your limit (for CPU[as defined server limit]) and Server has No Tasks Available (for your GPU) ... a message for each device.

It must be the client reporting how many tasks it has for the server to know how many more will reach the ServerLimit and takes that as face value without checking if you have 6000 InProgress.

Then of course there is the time limit constraint formula in that mix too.

We know the server currently has the ability to check for resends since that is how we recover ghosts - by forcing an upload Failure/TimeOut. That must set a server side flag for that ID - I know we're out of sync, so I better check ...

It might be interesting to track what Team the people with 6000 tasks are from to see if there is a link to a 'Group' using modified code ...

This is off topic from deadlines, but is also a major cause of bloating.

Generally speaking, I think that's accurate. I believe, though, that the client only indirectly tells the server how many tasks it has on hand. It lets the server figure that out from the <other_result> sections in the scheduler request. Take a look at your sched_request_setiathome.berkeley.edu.xml file. I don't think there's a hard count in there anywhere, but each <other_result> section carries an app_version and, optionally, a plan_class. It appears to be up to the server to piece it all together, which it seems to do without reference to the "In Progress" count in the DB.

As far as those with thousands of tasks go, it's likely a mix of those who do it intentionally and those who are oblivious, such as the one in my earlier example. That one's just a host with a serious ghosting problem that the owner likely isn't even aware of.

As to whether "ghosts" are a major cause of bloating or not is also an open question. Without figures to back it up, the scale of the problem is difficult to judge. But we do know it exists and some hosts stand out more than others.

By the sounds of it the 100 tasks per CPU and 100 tasks per GPU are there for two reasons:
1) the Max Tasks field isn't being calculated properly and
2) just in case there is a runaway of invalids for whatever reason

Nope.
The only reason it is there is to reduce the size of the Master database.

---

Grant
Darwin NT

Which is why I added my 2c worth, running on the assumption that if the underlying dispatch system works better then improved deadlines would be a likely outcome ;)

The deadlines are set by the project, and Jeff's initial numbers show that even as things are, there's no need for such long deadlines.

---

Grant
Darwin NT

Which is why I added my 2c worth, running on the assumption that if the underlying dispatch system works better then improved deadlines would be a likely outcome ;)

The deadlines are set by the project, and Jeff's initial numbers show that even as things are, there's no need for such long deadlines.

Hi Grant

There maybe extreme cases where a long deadline is needed, I am thinking something like a laptop that has both its buffers set as high as possible, and which only connects to the net every few weeks.

In this day and age that would be pretty rare, but it wasn't rare 15 years ago, so the long deadlines could just be a throwback from earlier times to make allowances for this type of situation.

Cheers

---

There maybe extreme cases where a long deadline is needed, I am thinking something like a laptop that has both its buffers set as high as possible, and which only connects to the net every few weeks.

In this day and age that would be pretty rare, but it wasn't rare 15 years ago, so the long deadlines could just be a throwback from earlier times to make allowances for this type of situation.

Which means the longest deadline for such a system would be 3 weeks.
And their extremely long turnaround time and cache settings are already taken in to account when the Scheduler allocates work to that system.
The numbers that Jeff posted in the first post in this thread show that there isn't any need for such long deadlines.

---

Grant
Darwin NT

I still reckon that SETI still needs gets smarter in the way it uses peoples' PC's resources, especially when a GPU (or 2) is present (each GPU task needs to be given a full core support and not some stupid impractical and unrealistic fraction of a core), because the way it runs as stock ATM just over commits a persons PC's resources and until that is address we'll just wind up with more hit and run jobs, just checking the rig numbers will show how many new users (higher numbers are the most often involved) just give up because they can't use there PC's with the current stock setup. It's these hit and runs that have been my biggest timeout problem over the last few years now, but more so under MB8 and the SoG app (but even the old CUDA apps have a bigger impact now under MB V8 as I found out last winter here).

IMHO if we can stop these hit and runs by better use of their PC resources (instead of crippling them) it would likely ease the database load a lot more than shortening deadlines.

Cheers.

Since quota management seems to have become as much a part of this discussion as task deadlines, I thought it might be appropriate to use that host with 6,000+ tasks again to try to pose a basic question. Inasmuch as that host is generating nothing but timeout errors, the "Max tasks per day" for each active application has dropped to 1, as it should. According to the Application Details page for the host, there appear to be 5 currently active applications, so even with that strange 8x multiplier, the host should theoretically be limited to no more than 40 tasks per day. Yet it seems to be steadily downloading about 3 times that many.

My question is simply, how? Does anyone know what other factor is in play? Here are the details for the 5 active apps:

SETI@home v8 8.00 windows_intelx86
Number of tasks completed 	1923
Max tasks per day 	1
Number of tasks today 	15
Consecutive valid tasks 	0
Average processing rate 	11.24 GFLOPS
Average turnaround time 	24.99 days

SETI@home v8 8.00 windows_intelx86 (cuda42)
Number of tasks completed 	6144
Max tasks per day 	1
Number of tasks today 	42
Consecutive valid tasks 	0
Average processing rate 	78.79 GFLOPS
Average turnaround time 	43.39 days

SETI@home v8 8.00 windows_intelx86 (cuda50)
Number of tasks completed 	14954
Max tasks per day 	1
Number of tasks today 	49
Consecutive valid tasks 	0
Average processing rate 	96.93 GFLOPS
Average turnaround time 	52.35 days

SETI@home v8 8.05 windows_x86_64
Number of tasks completed 	69
Max tasks per day 	1
Number of tasks today 	16
Consecutive valid tasks 	0
Average processing rate 	10.99 GFLOPS
Average turnaround time 	27.12 days

SETI@home v8 8.08 windows_x86_64 (alt)
Number of tasks completed 	0
Max tasks per day 	1
Number of tasks today 	17
Consecutive valid tasks 	0
Average turnaround time 	31.19 days

Obviously, in every case, the number of tasks sent to that host each day bears no obvious relationship to the "Max tasks per day" figure. So, what the heck is the scheduler thinking???

Good question. The simple analysis is that it should get exactly 5 tasks, one each for each of the five apps. But it actually is pulling down 139 a day.

---

Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)

Is he using an old version of BOINC?
Is he suffering bad communications so piling in the ghosts (I had a rogue router that resulted in a couple of thousand ghosts on an inaccessible cruncher).
Is he trying to reschedule and failing?
Is he trying to bunker for some reason or other?

And probably a few more if I put my mind to it.

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

Is he using an old version of BOINC?
Is he suffering bad communications so piling in the ghosts (I had a rogue router that resulted in a couple of thousand ghosts on an inaccessible cruncher).
Is he trying to reschedule and failing?
Is he trying to bunker for some reason or other?

And probably a few more if I put my mind to it.

I think, from the earlier discussion, that it should already be clear that this host is doing nothing but downloading tasks that all eventually time out. That doesn't appear to represent a user who's actively doing anything. And he's on BOINC 7.8.3.

The question is why the scheduler is ignoring the "Max tasks per day" value, which has correctly been reduced to '1' for all the active applications.