It is now 3 months since I reported the problems of Average Processing Rate (APR) and DCF when attaching a computer. It was also noted that this problem would also affect all computers when a project introduces a new application.
Seti has a new application waiting in the wings, at SetiBeta.

This is the problem that has enforced the download limits.

So, to the BOINC watchers, has there been any news or noticeable progress on this problem.

It is now 3 months since I reported the problems of Average Processing Rate (APR) and DCF when attaching a computer. It was also noted that this problem would also affect all computers when a project introduces a new application.
Seti has a new application waiting in the wings, at SetiBeta.

This is the problem that has enforced the download limits.

So, to the BOINC watchers, has there been any news or noticeable progress on this problem.

Those of us who watch these things closely keep nagging the SETI staff (including DA in that category, for the purposes of this discussion) to continue and eventually complete the recovery from the temporary band-aid solution which is currently requiring erratic DCF values and quota limits. I know they were reminded on Monday this week, in the hope that the next step could be completed during yesterday's maintenance - but I guess the 'two billion' bug rather scuppered that.

On the wider APR issue, which is going to be important for the SETI v7 launch (the transfer from Beta that you mention): I launched a wider discussion on the general issues involved on the boinc_dev mailing list, also on Monday. It's gained some support, but it's a very big question, and I'm not surprised that no progress has been observed in response.

Is your email adress still the same as when we exchanged logs in 2007? If so, I'll send you a copy.

Yes e-mail still the same.
Thanks in advance.

Yes e-mail still the same.
Thanks in advance.

Copy sent.

Well, I've looked at what is happening in BOINC world. The outlook is not very hopeful, at least in the short term. (short term means months)

As you may already know the problems are about the subject called Credit New.

This seems to revolve around the calculation of APR, which is then used to calculate processing time, and therefore downloads, and also for credits.

The real problem is the first word in APR, Average.
Looking at the problem I reported, I now realise was actually a minor problem, That can be fixed by identifying "too much blanking" in AP tasks and -9 in MB tasks and not using these affected tasks in the calculation.

On other projects, at least part of the processing the tasks, is to find out how long they run. This apparently, on fairly powerful hardware, can vary from e few seconds to days. Any average for tasks like that, even if accurate for a few days, can suddenly be altered very quickly just by a few that run for the minimum or maximum time.

Also for new hosts, and re-attached host, and for new applications it needs the APR to be used for all tasks from the word go, and that means finding some way of assessing the hardware performance better than the present benchmarks. Waiting until 10 tasks have validated, especially if a project has more than one application, in my view will not work.

It also affects the credit calculations, and like here most think it is no better than a random number counter. I cannot see it getting any better, especially on projects that like Seti use more than one task to check the validation. So that credits granted rely on the wingmen and their performance, as much as anything else.

So there we have it, at least from my point of view. I think we are stuck with download limits and random credits for a few more months at least. Even if Credit New can be made to work for all projects I believe the random credits will be here until it is replaced.

While none of this seems to really affect me at all (not a power-cruncher), there have been a few situations over the years where I have suggested that instead of average, use median, as median takes a look at the actual values in your data set and picks the one in the middle. It can provide a much more stable result in quite a few situations.

Examples:

Average: 1,27,41,56,68,71,72,73,78,100,54245 = ~4984
Median: 1,27,41,56,68,71,72,73,78,100,54245 = 71

Average adds everything and divides by how many values were added. Median ejects the high and low outliers one at a time until there is only one or two left. If one is left, it is that value, if two are left, average those two together. It has a tendency to provide a more realistic "middle of the road" value than a simple average does.

Most everything these days (mysql, php, etc) has the capability of doing median just as easily as it does average, except there is one minor difference in what would be required.

See, rather than keeping a set of values for an average, all you need are two running numbers: total, and number of contributing values. For the above example, this would be something like: 54832,11. When you have another value to add, say.. 345 for example, your two numbers get updated to 55177,12. Then you can still have an average.

Median requires that you know what all the contributing values are, so if APR is supposed to be the last 10 tasks that you've done, you would need to keep the last 10 processing rates somewhere to get the median of it. This would end up using more DB space, but not really putting any more overhead onto the system overall.

Just my two cents/pence.

---

Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)

...
Median requires that you know what all the contributing values are, so if APR is supposed to be the last 10 tasks that you've done, you would need to keep the last 10 processing rates somewhere to get the median of it. This would end up using more DB space, but not really putting any more overhead onto the system overall.

Just my two cents/pence.

APR is based on the "et" exponential average. That kind of averaging doesn't require keeping any set of data. As each new value is received, its delta from the existing average is used to determine how much to adjust the average. For the "et" average, the delta is multiplied by 0.01 (IOW the maximum adjustment for one new sample is 1%). The adjustment is faster for the first 20 samples, and there's no seed value.

That 1% limit means that the half-life is just over 69 validated tasks. IOW, if a new CPU or GPU is installed, the shift in APR would be about half done after 69 tasks from the new hardware was validated.

                                                                  Joe

...
Median requires that you know what all the contributing values are, so if APR is supposed to be the last 10 tasks that you've done, you would need to keep the last 10 processing rates somewhere to get the median of it. This would end up using more DB space, but not really putting any more overhead onto the system overall.

Just my two cents/pence.

APR is based on the "et" exponential average. That kind of averaging doesn't require keeping any set of data. As each new value is received, its delta from the existing average is used to determine how much to adjust the average. For the "et" average, the delta is multiplied by 0.01 (IOW the maximum adjustment for one new sample is 1%). The adjustment is faster for the first 20 samples, and there's no seed value.

That 1% limit means that the half-life is just over 69 validated tasks. IOW, if a new CPU or GPU is installed, the shift in APR would be about half done after 69 tasks from the new hardware was validated.

                                                                  Joe

The phase, there's no seed value, is the possibly the main problem for new hosts or applications, under Credit New.

Some time ago, before AP, I connected a new computer that downloaded twenty tasks. Of these 17 were from the same tape, all suffered from the -9 overflow problem.

If that had been three months ago, it would probaly still be suffering from the effects of the incorrect APR setting.

My APR for AP, which was why I raised the subject initially, is now just over 46 after 140 validated tasks. This value, from my own rough calculation and your guide of 2 * APR for MB on CPU (18.817), is still about 10 too high.

Because on my comptuer the problem is with the AP task it hasn't been a great problem because there isn't a great surplus of AP tasks around. But if it had been on MB CUDA, you can imagine the results when the APR was ~150 instead of 35.

Yes, but a problem that affects few, shouldn't result in a fix that adversely affects many.

---

Grant
Darwin NT

Yes, but a problem that affects few, shouldn't result in a fix that adversely affects many.

The problem for Seti will not be the few when the next app is introduced it will all.

Yes, but a problem that affects few, shouldn't result in a fix that adversely affects many.

The problem for Seti will not be the few when the next app is introduced it will all.

Thanks Xenu it will finally be a democracy :-)

Which means we will all be in the same boat ?? . . . Titanic.

Damn the torpedoes, full speed ahead.

---

Dave

At least it will be entertaining. :D