Observation of CreditNew Impact (3)

Message boards : Number crunching : Observation of CreditNew Impact (3)
Message board moderation

To post messages, you must log in.

Previous · 1 · 2

AuthorMessage
tbret
Volunteer tester
Avatar

Send message
Joined: 28 May 99
Posts: 3380
Credit: 296,162,071
RAC: 40
United States
Message 1416171 - Posted: 15 Sep 2013, 23:36:10 UTC - in response to Message 1416070.  



And there's the rub, the only problem I have with Credit systems normalising as they do (using whatever method). It sadly removes one of few 'real' motivators to improve the stock codebase. There are other less immediate motivators, like efficiency & striving for perfection, though somehow it's hard to find those satisfying when you double the performance of an application and everyone's credit drops by half.


Jason, please don't let that bother you. The advantage you have over most of us is that you know what you did and see the results in your testing.

Unfortunately, with "credits" being the only way most of us have to tell what's going-on, it takes a while for us to adjust our perceptions to the new reality. In the meantime it "feels bad" and we can't tell if that tweak we made helped or hurt.

And then complicate that with a "mix" of work units (AP, MB, vlars, vhars, vvars) and it takes a long, long time for us to adjust.

But don't let our whining out here in the dark take away from what you know you have accomplished.
ID: 1416171 · Report as offensive
Lionel

Send message
Joined: 25 Mar 00
Posts: 680
Credit: 563,640,304
RAC: 597
Australia
Message 1416174 - Posted: 15 Sep 2013, 23:46:46 UTC - in response to Message 1415990.  

SETI is one of a few, or perhaps the only, project that uses CreditNew (I call it CreditFew).

Subscribers to the boinc_projects mailing list will know that Eric Korpela is fully aware of this situation, and emailed the list last week in an attempt to verify this assertion. From the limited number of replies he received, it is clear that SETI is not the only project running CreditNew - I refer you (as I also referred Eric) to http://lhcathomeclassic.cern.ch/sixtrack/forum_thread.php?id=3754&postid=25782: Igor Zacharov states categorically that

All credit assignments as per boinc library settings and it is the NewCredit system.

In the past, we did experiment ...

There are other cases, as well.

Now I'll happily agree that there may not be any other (production) projects relying on CreditNew for GPU application credit - the jury's still out on that one. But I think we ought to be careful about making strong assertions without citing evidence or sources.



What evidence do you need Richard? All you have to do is look at the numbers. They tell the story quite clearly.

By the way, just after posting the comment above, I also posted a comment in Eric's staff blogg under "What's new about SETI@home v7" to let him know that my comment was there.

Overall, what amazes me is the lack of scientific rigor that is being applied to determining what the root cause(s) is/are with the rating system and this behaviour is from the "lead" scientists of the project. Tweaking a knob isn't the solution.














ID: 1416174 · Report as offensive
Lionel

Send message
Joined: 25 Mar 00
Posts: 680
Credit: 563,640,304
RAC: 597
Australia
Message 1416201 - Posted: 16 Sep 2013, 1:45:59 UTC - in response to Message 1415925.  

Over this same period have you noticed a drop in the average cobblestones awarded for AP task now?

Pre-MB7 AP's averaged around 785, but now that has dropped to around 710.

So it seems that some sort of leveling is happening.

Cheers.


Wiggo, I don't believe that I am noticing a drop in average credit awarded for an AP work unit. The credits bounce around quite a bit and in looking at my list of valids, they appear to be similar to before. If we could get a good run of APs for about 10 days straight we should be able to see if there is an indicator towards a change but with the oscillation occurring quite frequently it's a bit difficult to get a sense of it.

Lionel

ID: 1416201 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51477
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1416351 - Posted: 16 Sep 2013, 12:36:06 UTC - in response to Message 1416201.  

Over this same period have you noticed a drop in the average cobblestones awarded for AP task now?

Pre-MB7 AP's averaged around 785, but now that has dropped to around 710.

So it seems that some sort of leveling is happening.

Cheers.


Wiggo, I don't believe that I am noticing a drop in average credit awarded for an AP work unit. The credits bounce around quite a bit and in looking at my list of valids, they appear to be similar to before. If we could get a good run of APs for about 10 days straight we should be able to see if there is an indicator towards a change but with the oscillation occurring quite frequently it's a bit difficult to get a sense of it.

Lionel

A 10 day AP run???
LOL...you are dreaming, but the kitties would love to see that happen.

"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1416351 · Report as offensive
alan
Avatar

Send message
Joined: 18 Feb 00
Posts: 131
Credit: 401,606
RAC: 0
United Kingdom
Message 1416362 - Posted: 16 Sep 2013, 13:23:23 UTC

I have been running the optimised CPU applications for MB and AP since 2011 on the same machine and keeping records (by copying them from the fast-deleted workunit sheets) of credits and cpu-seconds. This allows me to graph CPU-seconds per credit (where a lower figure is better "payback").

I don't have a usable GPU in this laptop.

I have found that in the past, "shorties" have paid back badly compared to "regular" MB tasks, and that AP's paid back slightly better.

Since the introduction of Version 7 MB, the payback for optimised MB apps has roughly halved for me, as can be seen in the graphs, and the payback for AP has improved slightly.

The result is quite clear, that there is an imbalance in payback in my case that was not the case before V7. However the "shorties" now seem to pay back at about the same rate as regular MB's.

To me, the amount of cpu work done per credit granted is a useful and sensible measure of the value of each credit, where RAC is not, having too many other variables.

The only explanation I have seen that makes any sense of this is that the credit granted is "normalised" against the standard, non-optimised, application. In other words, the V7 "standard" app is now performing much better than the V6 one, and is closer to the optimised app than before.

What this also seems to mean is that no attempt is made to compare the new credit to the old when it comes to this normalisation, and that every new release of a standard application will reset the value of a "standard" credit. Or, to put it another way, for SETI@home CPU-based MB, V7 credits are devalued with respect to V6 credits as they are based on more efficient applications. Using the optimised apps, you have to work harder to get the same number of credits as before, which is unexpected. It would be interesting to see the same figures for someone running the standard apps, but this can only be done if someone has been saving their run-time figures like I have - the historic data is not publicly available (and may not be saved at all).

My graph is therefore wrong to show a continuous line linking V6 and V7 results, and there should be a discontinuity. It is true that the same factors affect everyone on the SETI@home project. The diminished payback compared to AP is now very significant and will result in AP's being grabbed and processed preferentially - which is no bad thing as they are producing real, fully processed results and as far as I know attract funding. This may be why they are now so rewarding.



ID: 1416362 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14676
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1416377 - Posted: 16 Sep 2013, 13:58:58 UTC - in response to Message 1416362.  

To me, the amount of cpu work done per credit granted is a useful and sensible measure of the value of each credit, where RAC is not, having too many other variables.

The only explanation I have seen that makes any sense of this is that the credit granted is "normalised" against the standard, non-optimised, application. In other words, the V7 "standard" app is now performing much better than the V6 one, and is closer to the optimised app than before.

I'd agree with the methodology and the general conclusion: RAC is hopeless for detailed examination like this.

One other comment on your final explanation: we had to withdraw the AK_v8 range of optimised CPU applications because of licencing complications. The replacement v7 optimised app, built using GPL-compliant open-source tools, is less tightly optimised than its predecessor. That will account for at least as much of the convergence between the apps as the improved optimisation in the stock app does.
ID: 1416377 · Report as offensive
Previous · 1 · 2

Message boards : Number crunching : Observation of CreditNew Impact (3)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.