Why is the 'new' credit system still erratic.

Message boards : Number crunching : Why is the 'new' credit system still erratic.
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · Next

AuthorMessage
W-K 666 Project Donor
Volunteer tester

Send message
Joined: 18 May 99
Posts: 19012
Credit: 40,757,560
RAC: 67
United Kingdom
Message 1059397 - Posted: 24 Dec 2010, 15:11:29 UTC

Regularly we are seeing posts questioning the credits granted. And the default answer is that after completing 10 tasks with the same application, processing estimates and credits will stabilise.

Well I've got some news for those that believe the stock answer. IT IS NOT TRUE.

On my PentM that only does AP tasks the estimate time for the 5 tasks that have not started varies by 1hr:42m (46 valid tasks), average crunch time is 25hrs.

And today my quad has crunched 12 tasks all AR=0.433756 from the same 'tape'. Nine have validated, the average granted credit is 110cr, but the variation is from 97.89 to 120.71.

You might to tempted to say "it depends on the wingman" well I've checked that also and not one of them has done less than 10 tasks with the application used. The wingman who did the lowest granted credit task has done 355 consecutive valid tasks.
ID: 1059397 · Report as offensive
Claggy
Volunteer tester

Send message
Joined: 5 Jul 99
Posts: 4654
Credit: 47,537,079
RAC: 4
United Kingdom
Message 1059443 - Posted: 24 Dec 2010, 18:07:18 UTC - in response to Message 1059397.  

You're going to have to read A new system for runtime estimation and credit.

From what i've read the Claimed Credit is eithier going to be:

C = F * cobblestone_scale (where cobblestone_scale is 200/86400e9. If replication is not used, this is the granted credit.)

If replication is used, we take the set of instances for which approx is false. If this set has one or more results, we grant the average of their claimed credit.

Otherwise:

if app.min_avg_pfc is defined
C = app.min_avg_pfc*wu.fpops_est*cobblestone_scale

if app.min_avg_pfc is not defined
C = wu.fpops_est * cobblestone_scale

Where F could be a number a values:


double claimed_pfc(J, double& f, bool& approx):
pfc = peak FLOP count(J)
approx = true;
if no app version info (relic of old scheduler)
F = wu.fpops_est
return

if pfc > wu.fpops_bound
host_app_version.scale_probation = true
host_app_version.error_rate = initial value
if app.min_avg_pfc is defined
F = app.min_avg_pfc * wu.fpops_est

or

if app.min_avg_pfc is not defined
F = wu.fpops_est

or

if job is anonymous platform
if app.min_avg_pfc is defined
if host_app_version.pfc_avg is above sample threshold
and not host_app_version.scale_probation
F *= app.min_avg_pfc / host_app_version.pfc_avg
approx = false

or

if host_app_version.pfc_avg is not above sample threshold
F = app.min_avg_pfc * wu.fpops_est

or

if app.min_avg_pfc is not defined
F = wu.fpops_est

or

if job is not anonymous platform
F = pfc;
host_scale = 0
if host_app_version.pfc_avg is above sample threshold
and app_version.pfc is above sample threshold
and not host_app_version.scale_probation
and host_app_version.pfc_avg <> 0
host_scale = app_version.pfc_avg / host_app_version.pfc_avg
if (host_scale) > 10 make host_scale = 10

if app_version.pfc_scale is defined
F *= app_version.pfc_scale
approx = false

if host_scale
F *= host_scale

if host_app_version.pfc_avg is not above sample threshold
and if app_version.pfc_scale is not defined
if host_scale
F *= host_scale

I'm not an expert in programming so i might have made some errors in logic, but this gives you a idea of the different ways NewCredit is worked out,

Claggy
ID: 1059443 · Report as offensive
Profile WHOSIT
Avatar

Send message
Joined: 12 Nov 10
Posts: 29
Credit: 162,958
RAC: 0
United States
Message 1059465 - Posted: 24 Dec 2010, 20:53:08 UTC
Last modified: 24 Dec 2010, 21:10:11 UTC

Greetings all,

I hope this is the correct posting area to place my questions concerning a small, strange problem I noticed with differences in my credit number reports. If not, I do apologise.

On my SETI@home screen saver, it states, that as of now, I have a Total credit of 6821.28 so far, where as in my signature, the old data (as is displayed) has remained static and unchanged for days.

{BTW, the data for the other three running programs are correct and do update.}

I have not changed any of my settings since I began running SETI and I believe this situation began after the latest "controlled outage". Anyone have an idea as to the actual cause of the non-updating signature numbers? Perhaps a cure for this unimportant nuisance problem?

Thanks in advance.
Wishing you continued science "fun" and at least, always valid results,
Laters,
Rick "WHOSIT" W.
Participating in: Einstein/MilkyWay/Rosetta/SETI
ID: 1059465 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1059468 - Posted: 24 Dec 2010, 21:09:57 UTC - in response to Message 1059465.  

Greetings all,

I hope this is the correct posting area to place my questions concerning a small, strange problem I noticed with differences in my credit number reports. If not, I do apologise.

On my SETI@home screen saver, it states, that as of now, I have a Total credit of 6821.28 so far, where as in my signature, the old data (as is displayed) has remained static and unchanged for days.

{BTW, the data for the other three running programs are correct and do update.}

I have not changed any of my settings since I began running SETI and I believe this situation began after the latest "controlled outage". Anyone have an idea as to the actual cause of the non-updating signature numbers? Perhaps a cure for this unimportant nuisance problem?

Thanks in advance.

Your signature graphic from an alternative stats site is more up-to-date. (But stats are never completely accurate, because the export only takes place once every 24 hours, and takes a significant length of time to process before appearing as a graphic)



That suggests that the export of data from the SETI@home project is working normally. You should perhaps address your question to the administrators of the (independent) stats site you chose to use for your signature.
ID: 1059468 · Report as offensive
Profile WHOSIT
Avatar

Send message
Joined: 12 Nov 10
Posts: 29
Credit: 162,958
RAC: 0
United States
Message 1059486 - Posted: 24 Dec 2010, 22:36:17 UTC - in response to Message 1059468.  
Last modified: 24 Dec 2010, 23:14:23 UTC


Your signature graphic from an alternative stats site is more up-to-date. (But stats are never completely accurate, because the export only takes place once every 24 hours, and takes a significant length of time to process before appearing as a graphic)



That suggests that the export of data from the SETI@home project is working normally. You should perhaps address your question to the administrators of the (independent) stats site you chose to use for your signature.



WOW! That was a FAST reply! :-)

Thank you very much Richard for your assistance!

Even though the matter is of little importance as compared to not getting any WUs, sending constant erred data back, or other such more serious problems, I shall undertake your suggested recommendation none the less, just to see what answer I may receive from them.
Perhaps one of their server's blades needs a good, swift kick. :-)

BTW, I noticed the numbers on my account, my screen saver, the graphic you included in your reply, and my malfunctioning signature graphic, are ALL different! HA!

Being that the work being and getting done IS the most important objective, to me anyway, rather than the displaying of my stats, I may place an all together different type of small graphic in my signature, just to include that added splash of color to it.
I can always just post the projects I'm running in the printed area of my signature and let the stat enthusiastic inclined go hunting for mine. He, he, he. ;-)

However, I still will be contacting that stat site, whatever I finally decide to do.

Again, Thank you Richard. Enjoy the holiday season my friend.

EDIT!!!!!

Richard! Oh my gosh! The SETI numbers in my sig's graphic just changed after my posting and match the ones in your reply posted graphic!

Now I'll just have to see if they keep updating and how long it takes to do so, to satisfy my curiosity. Geez! Impossible to make any changes now, being that this incident has occurred! I MUST SEEK THE ANSWERS TO THIS MINOR MYSTERY! :-)

Do you suppose, that your enquiry into my SETI stats, gave that faulty server blade the "kick" I mentioned it needed to get going again? Perhaps they discovered and fixed the problem at the same time? BOY! I hope it doesn't take DAYS to update the SETI stats!

AMAZING! I'll keep you posted on this matter.
Wishing you continued science "fun" and at least, always valid results,
Laters,
Rick "WHOSIT" W.
Participating in: Einstein/MilkyWay/Rosetta/SETI
ID: 1059486 · Report as offensive
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 1059499 - Posted: 25 Dec 2010, 0:45:55 UTC - in response to Message 1059397.  

Regularly we are seeing posts questioning the credits granted. And the default answer is that after completing 10 tasks with the same application, processing estimates and credits will stabilise.

Well I've got some news for those that believe the stock answer. IT IS NOT TRUE.
...

The new credit system is basically very similar to the original time*benchmarks approach. As Dr. Anderson terms that, it is "peak flops" based. The difference now is the "benchmark" is an average the servers maintain for each application on your host.

If you think back to credits under that original system, there was a fair amount of variation from WU to WU even in the claimed credit. Under the new system we don't see the actual claim, though the "pending" page continues to show the "work based" claim which is no longer used. But the variation is there, some WUs take longer than others for no obvious reason.

Then as Claggy has posted there are multiple conditions affecting how the granted credit is calculated. The "average" used when at least one of the results is from a stock application is not the arithmetic mean. Instead it uses a low_average() function which "(reduces the weight of large outliers)". The effect is that "claims" of 20 and 100 will generate a grant of 33.33... for example. But if all results are from anonymous platform hosts, the minimum claim is granted.

So there will be variations in granted credit, and the project difficulties have blurred the statistics enough I don't think it's yet practical to evaluate how well the new system is working.
                                                               Joe
ID: 1059499 · Report as offensive
Profile Frizz
Volunteer tester
Avatar

Send message
Joined: 17 May 99
Posts: 271
Credit: 5,852,934
RAC: 0
New Zealand
Message 1059508 - Posted: 25 Dec 2010, 1:58:40 UTC - in response to Message 1059499.  
Last modified: 25 Dec 2010, 2:01:16 UTC

But if all results are from anonymous platform hosts, the minimum claim is granted.


Isn't that a bit unfair? Anonymous platforms mostly use highly optimized apps. Why penalize those energy and resource efficient hosts?

It should be the other way round imho. To encourage developers to write even more efficient apps. And to encourage crunchers to use those apps.
ID: 1059508 · Report as offensive
Lane42

Send message
Joined: 17 May 99
Posts: 59
Credit: 227,150,556
RAC: 11
United States
Message 1059517 - Posted: 25 Dec 2010, 5:07:53 UTC - in response to Message 1059508.  

I got screwed on this one

http://setiathome.berkeley.edu/workunit.php?wuid=672119382
ID: 1059517 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1059519 - Posted: 25 Dec 2010, 5:24:04 UTC

I'm fine with more of these: http://setiathome.berkeley.edu/workunit.php?wuid=667609785 (it's an AP, got 2121.83 for it). I've noticed in the past day or two though, granted credit for APs is all over the place. Ranging from 480 to about 790, mostly in the 675-725 range though. Can we please just go back to the old system? It wasn't broken..
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1059519 · Report as offensive
Profile Geek@Play
Volunteer tester
Avatar

Send message
Joined: 31 Jul 01
Posts: 2467
Credit: 86,146,931
RAC: 0
United States
Message 1059520 - Posted: 25 Dec 2010, 5:24:30 UTC - in response to Message 1059517.  

I got screwed on this one

http://setiathome.berkeley.edu/workunit.php?wuid=672119382


You surely did but....................were in this for the science, right?

And the results were good science!
Boinc....Boinc....Boinc....Boinc....
ID: 1059520 · Report as offensive
-BeNt-
Avatar

Send message
Joined: 17 Oct 99
Posts: 1234
Credit: 10,116,112
RAC: 0
United States
Message 1059524 - Posted: 25 Dec 2010, 6:07:34 UTC - in response to Message 1059519.  

I'm fine with more of these: http://setiathome.berkeley.edu/workunit.php?wuid=667609785 (it's an AP, got 2121.83 for it). I've noticed in the past day or two though, granted credit for APs is all over the place. Ranging from 480 to about 790, mostly in the 675-725 range though. Can we please just go back to the old system? It wasn't broken..


Not to mention you got over 2,000 credit for those?
Traveling through space at ~67,000mph!
ID: 1059524 · Report as offensive
W-K 666 Project Donor
Volunteer tester

Send message
Joined: 18 May 99
Posts: 19012
Credit: 40,757,560
RAC: 67
United Kingdom
Message 1059526 - Posted: 25 Dec 2010, 6:09:49 UTC

Thanks for the responses, my conclusion from this is that, if this method is to be continued then someone ought to study statistics a bit more.
ID: 1059526 · Report as offensive
-BeNt-
Avatar

Send message
Joined: 17 Oct 99
Posts: 1234
Credit: 10,116,112
RAC: 0
United States
Message 1059528 - Posted: 25 Dec 2010, 6:12:01 UTC - in response to Message 1059526.  

Thanks for the responses, my conclusion from this is that, if this method is to be continued then someone ought to study statistics a bit more.


If you have a theory on a better way to handle it, have you thought about writing out what you are thinking and forwarding it to the people responsible? It may be worth writing because there is always room for improvement.
Traveling through space at ~67,000mph!
ID: 1059528 · Report as offensive
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 1059532 - Posted: 25 Dec 2010, 6:50:01 UTC - in response to Message 1059508.  

But if all results are from anonymous platform hosts, the minimum claim is granted.


Isn't that a bit unfair? Anonymous platforms mostly use highly optimized apps. Why penalize those energy and resource efficient hosts?

It should be the other way round imho. To encourage developers to write even more efficient apps. And to encourage crunchers to use those apps.

Yes, it's slightly unfair to those who use anonymous platform ethically. But as was thoroughly proven by S@H Classic there's a minority who get too competitive for credits and will cheat if possible. So BOINC always mistrusts anything a user can change.

Even with the best of intentions, there are things users may do which make the credit calculations deviate quite a bit. One example is shifting from doing one WU at a time on a Fermi GPU to doing more. Shifting from 1 to 3, for instance, might extend the elapsed times by about 2.5 times which would increase the claimed credit by the same factor until the server-side averages adapted.
                                                                Joe
ID: 1059532 · Report as offensive
-BeNt-
Avatar

Send message
Joined: 17 Oct 99
Posts: 1234
Credit: 10,116,112
RAC: 0
United States
Message 1059535 - Posted: 25 Dec 2010, 7:09:06 UTC - in response to Message 1059532.  


Yes, it's slightly unfair to those who use anonymous platform ethically. But as was thoroughly proven by S@H Classic there's a minority who get too competitive for credits and will cheat if possible. So BOINC always mistrusts anything a user can change.

Even with the best of intentions, there are things users may do which make the credit calculations deviate quite a bit. One example is shifting from doing one WU at a time on a Fermi GPU to doing more. Shifting from 1 to 3, for instance, might extend the elapsed times by about 2.5 times which would increase the claimed credit by the same factor until the server-side averages adapted.
                                                                Joe


Wait so doing more work is 'cheating'? So exactly what would you call using the anonymous platform ethically? I run two WU's at a time on my GTX 480 because it can do it not because it's cheating the system. Even with both units running it is still only using about 40% of the cards power. Which tells me with properly optimized apps this card could crunch 4 WU's at a time. With lower times it means you could place MORE returns into the system still giving you a higher credit count than the others.

So I would like to know, how, learning and putting in to place a more efficient way for your machine to work is unethical? In the end your theory on causing higher elapsed times you have disproven yourself. The cheat will only work once, until the calculations average themselves. This is coming from a 'cheater' though so I guess it isn't fair looking from the other side.

Oh and btw, running 2 units at a time actually causes the average of work time drop. With optimized apps it would drop even further. One WU on my Fermi takes 9 minutes, running 2 takes 14, meaning an average of 7 minutes per, so I would love to see your logic here.

Traveling through space at ~67,000mph!
ID: 1059535 · Report as offensive
Profile soft^spirit
Avatar

Send message
Joined: 18 May 99
Posts: 6497
Credit: 34,134,168
RAC: 0
United States
Message 1059537 - Posted: 25 Dec 2010, 7:33:00 UTC - in response to Message 1059535.  

Bent, they did not say using optimized apps was cheating. There is a history of some people "faking" results to show complete back in classic days.

It really got ugly, which is one of the reasons we now must get the unit "validated" and checked against others first.

In a perfect world, each FLOP used would match, and that would be the fair indicator. I guess that is hard to program.
Janice
ID: 1059537 · Report as offensive
-BeNt-
Avatar

Send message
Joined: 17 Oct 99
Posts: 1234
Credit: 10,116,112
RAC: 0
United States
Message 1059564 - Posted: 25 Dec 2010, 9:06:28 UTC - in response to Message 1059537.  

Bent, they did not say using optimized apps was cheating. There is a history of some people "faking" results to show complete back in classic days.

It really got ugly, which is one of the reasons we now must get the unit "validated" and checked against others first.

In a perfect world, each FLOP used would match, and that would be the fair indicator. I guess that is hard to program.


I know it isn't cheating. The way I read his post he is saying that using an optimized app and running 2 WU's at a time is somehow cheating the credit system by making it take longer for your WU to complete hence causing more credit to be granted.

I participated back in Classic but never heard about the faking of WU's, but I also didn't participate in the forums then either. However what's the point of even worrying about the credits? I like to see where I lay on the ranks and RAC etc. But does it really matter if I'm #2000 or 8000 in the rankings? Not to me. Just seems people take this whole thing entirely too seriously at times.

Guess some people do it for the science behind it, others for bragging rights, and the rest just because they want to help in a cause. Suppose pick who you are and roll lol.
Traveling through space at ~67,000mph!
ID: 1059564 · Report as offensive
Profile Frizz
Volunteer tester
Avatar

Send message
Joined: 17 May 99
Posts: 271
Credit: 5,852,934
RAC: 0
New Zealand
Message 1059567 - Posted: 25 Dec 2010, 9:15:30 UTC - in response to Message 1059532.  
Last modified: 25 Dec 2010, 9:19:07 UTC

Even with the best of intentions, there are things users may do which make the credit calculations deviate quite a bit. One example is shifting from doing one WU at a time on a Fermi GPU to doing more.


I am running 5 Astropulse tasks at a time on my HD6970 because running them in parallel is faster then running them serially. This simply utilizes the GPU better.

Plus it mitigates the effects of memory latency and blanking. So, in a way, I could say it's the Seti gods fault that I have to run several tasks at a time - because of this nonsensical blanking approach :)

I assume there already have been lots of discussions regarding the new credit system. Anyway ... for Astropulse it would be really simple (imho) :

credits = 500 + 10 * percent_blanked

P.S.
Yes, I like things to be simple :)
ID: 1059567 · Report as offensive
Profile soft^spirit
Avatar

Send message
Joined: 18 May 99
Posts: 6497
Credit: 34,134,168
RAC: 0
United States
Message 1059568 - Posted: 25 Dec 2010, 9:16:27 UTC - in response to Message 1059564.  

Bent, they did not say using optimized apps was cheating. There is a history of some people "faking" results to show complete back in classic days.

It really got ugly, which is one of the reasons we now must get the unit "validated" and checked against others first.

In a perfect world, each FLOP used would match, and that would be the fair indicator. I guess that is hard to program.


I know it isn't cheating. The way I read his post he is saying that using an optimized app and running 2 WU's at a time is somehow cheating the credit system by making it take longer for your WU to complete hence causing more credit to be granted.

I participated back in Classic but never heard about the faking of WU's, but I also didn't participate in the forums then either. However what's the point of even worrying about the credits? I like to see where I lay on the ranks and RAC etc. But does it really matter if I'm #2000 or 8000 in the rankings? Not to me. Just seems people take this whole thing entirely too seriously at times.

Guess some people do it for the science behind it, others for bragging rights, and the rest just because they want to help in a cause. Suppose pick who you are and roll lol.


In the case of the classic incident(which still leaves its mark today) it was one person that made their own script that would almost instantly "turn in" the results, got others to download it(It may well have contained malware) and refused to share what was in it. When asked if the math he used was up to SETI's own standards, his response was "it doesn't matter, there isn't anything there anyway".

A lot of damage to the entire process of sending work out to the public suffered. Safety checks were implimented. And the person perpetuating this finally slipped into oblivion.

lunatics op apps is pretty well recognized by the SETI staff, they seem comfortable with it, and my understanding the source is readily available to them. If the lab was uncomfortable with it, I would uninstall it in a heartbeat.

As far as credits, I am happy as long as there is a consistent system that I can guage how well my systems are set up for it.

Well.. there is some history as seen from the cheap seats.
Janice
ID: 1059568 · Report as offensive
-BeNt-
Avatar

Send message
Joined: 17 Oct 99
Posts: 1234
Credit: 10,116,112
RAC: 0
United States
Message 1059570 - Posted: 25 Dec 2010, 9:18:45 UTC - in response to Message 1059567.  
Last modified: 25 Dec 2010, 9:19:38 UTC

Even with the best of intentions, there are things users may do which make the credit calculations deviate quite a bit. One example is shifting from doing one WU at a time on a Fermi GPU to doing more.


I am running 5 Astropulse tasks at a time on my HD6970 because running them in parallel is faster then running them serially. This utilizes the GPU better.

Plus it mitigates the effects of memory latency and blanking. So, in a way, I could say it's the Seti gods fault that I have to run several tasks at a time - because of this nonsensical blanking approach:)

I assume there already have been lots of discussions regarding the new credit system. Anyway ... for Astropulse it would be really simple (imho) :

credits = 500 + 10 * percent_blanked

P.S.
Yes, I like things to be simple :)


If you like it simple why don't you just put a flat value on everything. Then it comes down to the amount of work you are finishing.

AP = 500
MB (cpu) = 250
MB (cuda) = 350

High astropulse credits since it's soo much work, higher count for MB on cuda to encourage people to use faster machines. More faster machines gets more work finshed quicker. Now that would be simple.

I know the whole not every WU is the same size etc. But honestly this isn't a race or a competition when it comes down to it.
Traveling through space at ~67,000mph!
ID: 1059570 · Report as offensive
1 · 2 · 3 · Next

Message boards : Number crunching : Why is the 'new' credit system still erratic.


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.