Eric's Out of Adjectives Post #12: Credit pandemonium

It's been a while again. One of my new year's resolutions is to post more frequently. A lot has happened since the last time I posted. The winter funding drive, working on the version 6.0 SETI@home application (which is in beta with few problems), continued work on adding the database structures to support the NTPCKR, etc. I went to a depressing American Astronomical Society conference in Austin (depressing because of the NASA and NSF funding situations) where I picked up a nasty flu which had me laid out for a week, and from which my lungs are still recovering. And the information cascade failure we call the U.S. presidential primary system has begun. To top it all off, last weekend one of our cats died. But, by request, I'm going to talk about something else entirely. This is also going to be long-winded. There's a lot of back-story and more issues than can really be fully explored, but I want to get it all down somewhere accessible.

There was a bit of a uproar in the last couple weeks about BOINC credit system and possible changes to it. It's not the first time, and it won't be the last time it comes up. And every time it comes up, there will be people who get angry because they feel the change will be bad for them.

The BOINC credit system is a sort of artificial economic system. I'm not an economist, and I've never taken a class on economics, so I know very little about economic systems. So I may get some details of this wrong.

In the BOINC credit system, credits take the place of money. Projects pay you for your CPU time in credits, and with your credits, you buy bragging rights or ranking in a list. It seems somewhat odd to me, but there are people who run BOINC only for the purpose of earning credits and who will change projects to maximize their earnings.

In an ideal world, a BOINC credit would be equal to a fixed amount of work actually done. This would be the equivalent of the Gold standard, where the supply of money is fixed by the amount of gold. In this case the amount of BOINC credit is set by the amount of work that the machines running BOINC do. But of course the world isn't ideal. It's actually very difficult to measure the actual amount of work done by an application program. It's also difficult to define exactly what work is. On the other hand it's fairly easy for the BOINC application to approximate the maximum amount of work that a processor is capable of doing.

So when BOINC started, the general means of granting credit was to multiply the measurement of maximum work capability time the CPU time used. This has several problems... Different versions of BOINC give different estimates of maximum work capability and processors have different efficiencies. That means that different machines would claim different amounts of credit for the same amount of work. That's the equivalent of using gold coins with an unknown percentage of gold in them. The other problem is that some people use applications optimized for their specific processor, which can be much faster than a generalized application. These machines would ask for less credit than they were owed. It led to a lot of complaints about people being granted too much or too little credit.

There was also a problem when we released updated, more highly optimized applications. Run times decreased and therefore granted credits decreased for the same amount of work. For people running the stock application, this was hardly noticed, since they got roughly the same amount of credit per day. People running optimized applications which had been 8X as fast as the generic stock application were now only running applications that were 2X or 3X as fast as the stock application. Their rate of credit accumulation went down because of it and some of them got angry and left the project. Let that sink in... Some people left the project because the project had become more efficient. In economic terms, optimizing the stock application when granting credit based upon CPU time is deflationary. Less credit is granted for the same amount of work. Deflation is considered a "bad thing" for multiple reasons.

So we modified the application to grant a specific amount of credit for each floating point operation in the application, essentially moving back toward the gold standard. To keep the rate of granted credit roughly constant for the same machine, we instituted a multiplier that would set the number of credits each floating point operation generated. That was released without much problem, aside from some complaints from people whose rates went down by 10 to 20%. There were as many people whose rates went up by 10 to 20%, but they didn't complain.

But more optimizations were incorporated into the application. At first, in the beta project, we didn't change the credit multiplier. That made the rate at which credits were being granted go up. I started to see complaints from other projects, and users that ran multiple projects that the SETI beta was granting more credit per day than other projects on the same machine. When granting constant credit per unit work, optimizing the application is inflationary. If something wasn't done, other projects would start increasing the amount of credit they granted to match.

An even worse possibility is that projects would start increasing the amount of credit they grant as a means of attracting crunchers. In this case the "money supply" is not fixed by any mechanism. There is nothing that physically prevents a project from granting 10X, 100X, or a million times as much credit per CPU second as SETI@home does. To combat this potential for hyper-inflation, there is an unwritten agreement among BOINC projects that we will strive to achieve credit parity. That is to say, a specific machine should generate the same number of credits per day whether it runs SETI@home, Einstein@home, or Rosetta@home. That allows volunteers to base their decisions of which projects to support based upon the goals of the project, rather than the amount of credit they grant.

Of course this isn't perfect either. Every time we release a new version of SETI@home, we need to adjust the credit multiplier. That takes quite a bit of work, and even then it isn't perfect. Right now SETI@home is granting 10 to 15% more credit than it should. The credit granted per CPU time also varies depending upon the details of the work unit. Neither of these problems will be solved before SETI@home version 6 is released. Maybe not even then. It would be nice if there were some sort of automation for determining the credit multiplier on the fly.

A couple weeks ago, David posted a message to the boinc_projects mailing list requesting that projects normalize the amount of credit they grant to SETI@home. Some of the projects were granting 75% of the credit SETI@home does. Others were granting up to 3 times as much credit as SETI@home does. Since new BOINC projects come along all of the time, the managers of these projects may not have been aware of the informal agreement to maintain credit parity. It appears that most, if not all projects, will do so. I think all of the project managers understand the potential for credit inflation, and understand that it would be damaging to the projects and to BOINC overall. Some people, especially those that had migrated to high credit projects in order to maximize the number of credits they earn, felt that David's request was an unwarranted ultimatum intended to coerce projects into compliance. While I understand that David could have used more diplomatic language, I don't think that the message was an ultimatum, because David really doesn't hold enough power to drive compliance.

I ask those that have complained that previously high credit projects will reduce their credit multipliers to understand that BOINC credits have value because they represent an amount of work that can be compared. Credit inflation destroys that value. If I were to raise the SETI@home credit multiplier by a factor of 10, the credits you have earned so far will lose 90% of their value. Someone could earn the same number of credits in a tenth of the time. Other BOINC projects would probably be forced to raise their credit multipliers, in order to retain the "I crunch for credit" crowd. Of course some projects will attempt to retaliate against SETI@home by increasing theirs by a factor of 20. Very soon after that, BOINC credits would lose all value and meaning. Cross project comparisons become impossible.

Shortly after his first message, David released a credit proposal that would address some of these issues. Like all proposals, it has good features and it has bad features. I doubt that it will be implemented in anything close to its present form. The essence of the proposal is "In any project, the average machine should receive 100 credits per day if it participates full time."

There are some benefits to this proposal. First, a project can determine its credit multiplier without referencing other projects. All that needs to be done is to figure out how many credits per day the average machine gets and adjust the multiplier higher or lower to achieve 100. This could even be automated in the BOINC server software. That's less work for me to do manually, so I'm all for that.

The proposed scheme contains incentive to work on non-competing projects as they become available. For example an average machine could earn 100 credits per day running a CPU intensive project like SETI@home, while simultaneously earning 100 credits per day running a storage project, and 100 credits per day running a network intensive project.

There are also some fairly major drawbacks, and that is why I think it won't be implemented as is. First, it's immediately deflationary. The average host that connected to SETI@home today earns 292 credits per day, which is significantly more than 100. There would be strong (perhaps irresistible) demand for projects to keep their current credit scheme for as long as possible.

Second, it's deflationary over time. If Moore's law holds, the average host a year from now will earn 470 credits per day. So under the proposed credit scheme a host that earns 100 credits per day today would earn about 62 credits per day a year from now. It violates the idea that was implicit in earlier credit schemes, equal credit for equal work.

But in order to have an better scheme implemented, its incumbent upon the projects, and therefore me, to propose something that has the same benefits without the drawbacks. I'm thinking about it now. Ideas are welcomed.

EDIT: I forgot to include number of processors in my average credit per day calculation. The average host currently earns 535 credits per day not 292. The average host has 1.65 processors.
____________

The average host has 1.65 processors.

The average host has 1.65 processors.

... There was a bit of a uproar in the last couple weeks about BOINC credit system and possible changes to it. It's not the first time, and it won't be the last time it comes up. ...

The BOINC credit system is a sort of artificial economic system. ...

In an ideal world, a BOINC credit would be equal to a fixed amount of work actually done. This would be the equivalent of the Gold standard, where the supply of money is fixed by the amount of gold. ...

The proposed scheme contains incentive to work on non-competing projects as they become available. For example an average machine could earn 100 credits per day running a CPU intensive project like SETI@home, while simultaneously earning 100 credits per day running a storage project, and 100 credits per day running a network intensive project.

There are also some fairly major drawbacks, and that is why I think it won't be implemented as is. First, it's immediately deflationary. The average host that connected to SETI@home today earns 292 credits per day, which is significantly more than 100. There would be strong (perhaps irresistible) demand for projects to keep their current credit scheme for as long as possible.

Second, it's deflationary over time. If Moore's law holds, the average host a year from now will earn 470 credits per day. So under the proposed credit scheme a host that earns 100 credits per day today would earn about 62 credits per day a year from now. It violates the idea that was implicit in earlier credit schemes, equal credit for equal work.

But in order to have an better scheme implemented, its incumbent upon the projects, and therefore me, to propose something that has the same benefits without the drawbacks. I'm thinking about it now. Ideas are welcomed. ...

I'm not angry because I feel the change would be "bad" for me. I'm not even really "angry", per se. I reacted strongly to the wording of David's message. I find it very troubling that he would even consider asking the factual data to be manipulated at the stats sites. That, at least to me, seems to go against the foundation of the Scientific Method.

Anyway, the notion of equivalent credit per cpu second is a good notion. However it takes more work than just stating that Project X is the baseline and you must rig your credits to be 0.8X - 1.2X the value of Project X. All you're doing is rigging credits. You're not addressing the performance of a processor on various projects, like how some projects appear to use more L2 cache than others.

Another interesting aspect is that since the science application here is Open Source, people have been free to work on optimizations, including processor architecture optimizations. Other projects do not have open source applications nor do they have resources to put into optimization.

As such, one likely outcome of mandating SETI as the standard will mean that those who you "fear" that run to projects on the amount of credits granted would realize that cr/sec would be higher than the "1.0" baseline if one were to use the optimized application.

As such, a mandate by the Director of this project that all projects must conform to this project is at best a conflict of interest.

Finally, you'll note that I have stopped participation in doing tasks for this project. At this point I do not know if I will return. I had already started feeling that this project has less immediate scientific merit than the projects I am currently working on (Einstein, LHC, and Cosmology). This noise, and in particular David's behavior, turned me away...

[S]omeone crunching for credits is better off going to an open source project where they can find an optimized application. That makes it a benefit for an application to be open source, and I personally think that's a plus, although projects with closed source apps might find it to be a minus. On the minus side, being open source will also drive some people crunching for credits to find the project with the most poorly optimized stock application, since that's where they would get the most credit. (IOW, an optimized app can run 10X the speed of a poorly optimized stock app and generate 10X the credits, whereas if the stock app is well optimized, a processor specific app might only be able to go 2X as fast and generate 2X the credits.) I really don't know how to fix this problem, and that's part of the reason why we're having this discussion.

I hope you will continue to focus on the scientific merit of projects when choosing what to crunch. It's the way I choose.

BTW, my condolences to all in your household for the departed furry companion.

Just because people perceive they have value, doesn't mean they have;

Again IANAE,

but isn't the lesson of economics that things only have value because people perceive that they have value? A transaction is where people exchange something they value to get something they value more. In the case of BOINC projects, people give up their excess computing power in exchange for a combination of advancing the science, earning credits, and being part of a community. Each person will place a different value on each of those three items, but it's important that the projects recognize that all three are of value to some of their participants.

@OzzFan
Shame on you. Google....
IANAE
____________
Questions? Answers are in the "Unofficial" BOINC Wiki.

Boinc V7.0.27
Win7 i5 3.33G 4GB, GTX470

Again IANAE,

What does IANAE mean?