Really busy day for me, but not much on the public facing side of things. Jeff and I are revamping our current backend data pipeline in light of continual hardware and I/O headaches. I'm pulling a bunch of stuff out of the database for Josh so he can do some more "find the known pulsar and see if it looks like RFI" game in Astropulse. I enacted the "no redundancy" policy on beta - we're curious to see how well it works in practice, mostly for the sake of general BOINC testing. I had some updating/programming to do regarding our donations database - stuff that campus requested.

Still no signs of the air conditioner being fixed (though it is running cooler in the closet than earlier in the week). And we haven't yet replaced the bad drive on bambi (though we have a spare sitting on the shelf).

- Matt

---

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Thanks for the update...'ave a good night, Matt...

---

Clk2HlpSetiCty:::PayIt4ward

Great news that you are testing the no-redundancy option. Let us know what the test parameters are and success criteria, when you get a chance and have dancing fingers. Thx.

...
Still no signs of the air conditioner being fixed (though it is running cooler in the closet than earlier in the week). And we haven't yet replaced the bad drive on bambi (though we have a spare sitting on the shelf).

Dicing with the devil....

Can the array stand another drive going down?

---

mic.

Really busy day for me...

Some days, it's just a bunch of little things. Sounds like they are still important though. I like having a lot of check marks on my task list. :)

Great news that you are testing the no-redundancy option. Let us know what the test parameters are and success criteria, when you get a chance and have dancing fingers. Thx.

So, how can you test for a false negative?

(That is, there's a signal there but not reported... Could be THE signal.)

Happy crunchin',
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)

. . . Thanks to Each of You @ Berkeley for All the Work bein' done - iT is appreciated

and Matt - as usual - Thank You for the Updates . . .

---

BOINC Wiki . . .

Science Status Page . . .

Great news that you are testing the no-redundancy option. Let us know what the test parameters are and success criteria, when you get a chance and have dancing fingers. Thx.

So, how can you test for a false negative?

(That is, there's a signal there but not reported... Could be THE signal.)

Happy crunchin',
Martin

If that question is for me, there has been a long thread generated elsewhere about this recently where you might find my answer. Basically, for each positive signal coming in we have a high confidence (>95%) of detecting it with a single client. This confidence level is limited primarily by the reliability of the typical client and the ability (assumption?) of our science software of doing its job. Hence, the no-redundancy option becomes one of a few percentage points more in our confidence level, or of processing the (essentially infinite) sky twice as fast. I vote for the latter because if there is one signal there will be others, and the statistics indicate we are wasting our computing time otherwise.

But my comment here was a request to understand how the no-redundancy test was going to proceed; what the success criteria are; and so forth. This is independent of whether we should have redundancy or not. But it may shed some light on how CentralCommand goes about their work.

I've got a bad feeling about going to zero redundancy based upon local experience. I had a computer that was sickening. The first symptom was that some SETI units would process OK, some would error out, and -- here's the worrisome part -- some would get rejected because they didn't match up with their wingmen's results. The machine was making occasional numerical mistakes.

I couldn't find the sickness and it slowly worsened until the machine was spontaneously rebooting frequently. I finally had to replace the machine.

With zero redundancy, my machine would have put bad results into the science database. *shudder*

---

I've got a bad feeling about going to zero redundancy based upon local experience. I had a computer that was sickening. The first symptom was that some SETI units would process OK, some would error out, and -- here's the worrisome part -- some would get rejected because they didn't match up with their wingmen's results. The machine was making occasional numerical mistakes.

I couldn't find the sickness and it slowly worsened until the machine was spontaneously rebooting frequently. I finally had to replace the machine.

With zero redundancy, my machine would have put bad results into the science database. *shudder*

I think the rebuttal is that if you reported a false negative, then that's life. The next signal from ET will be caught by a more reliable client with a 95% probability. (This argument obviously breaks down if there is only one ET signal every generated or ever will be generated. Assuming there is any signal-producing ET out there, then such a unitary-signal would be unlikely to the extreme.)

If you reported a false postive, the system would repeat your calculation with a high degree of redundancy to be sure. That would happen for any positive result. So, again, your errant client will not be a problem. (This argument breaks down if we all had errant clients; in which case we would need a quorum of n>>1 to be 'sure' we had a valid result.)

What we get for the n=1 risk is the ability to scan the heavens about 2x faster, or the ability to run more complex analysis (like AP) with the corresponding time savings. This means a lot to us who don't want to die before ET is found, or, more practically, who want to get grants to do more science and further their career. Plus, at the risk of speaking out of turn, I'm reasonably sure that Matt doesn't want to retire doing seti IT administration in its current state. A discovery of any sort would be really neat! Too bad that we are temporary beings, or we could afford to take our time.

Matt Lebofsky said:
... I enacted the "no redundancy" policy on beta - we're curious to see how well it works in practice, mostly for the sake of general BOINC testing....

I see numerous similarities between distributed computing and complex systems. Complex systems is the focus of my graduate research right now. One distinctive property of complex systems is some built-in redundancy that causes a degree of robustness and tolerance of error. As much as it is tempting to only have 1 task per result (to save space and time), it would be a mistake.

I am curious to see how well it works on Beta, but we have seen cases where things worked fine in Beta but did not scale as expected on the main project.

PhonAcq said:
If you reported a false postive, the system would repeat your calculation with a high degree of redundancy to be sure.

One question: How do you detect a false positive?
Answer: You can't; it's an error. You'll have to resample the data, hence the 2 task per WU. The SETI executable MAY detect system problems, but without validation (sample comparison) at the next level up, you'll never catch some errors. The sample principle applies in statistics. You cannot reject the null hypothesis with only 1 sample.

If there is a positive, can't the workunit be given to two others to verify it? It seems like that would be easy. It would be nice to know the actual error rate. Is it 1%? 2%? 0.1%? Erroneous negatives would be the same as just throwing out the unit without testing it. If the error rate is low, that's not so bad.

---

If that question is for me, there has been a long thread generated elsewhere about this recently where you might find my answer.

The conclusion was?

Basically, for each positive signal coming in we have a high confidence (>95%) of detecting it with a single client. This confidence level is limited primarily by the reliability of the typical client and the ability (assumption?)...

What are your assumptions for your "95%" and for what?...


Do you know of the examples for s@h-classic?

Keep searchin' or Happy crunchin'?
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)

If that question is for me, there has been a long thread generated elsewhere about this recently where you might find my answer.

The conclusion was?

Basically, for each positive signal coming in we have a high confidence (>95%) of detecting it with a single client. This confidence level is limited primarily by the reliability of the typical client and the ability (assumption?)...

What are your assumptions for your "95%" and for what?...


Do you know of the examples for s@h-classic?

Keep searchin' or Happy crunchin'?
Martin

DJStarfox:
CCPhillips understands how to find false positives, and my general point, very well and only needed 3 lines. You take a positive and check and double check, just like any scientific result. You can't prove anything, but you can begin to believe it when results repeat enough times.

The analogy to complex systems may be satisfying, but proof by analogy is no proof. (Sorry, I probably just got all the Santa Fe institute simulators all upset with that PGIO (penetrating glimpse into the obvious)

ML1: does anyone ever conclude anything in a thread? That would be nice. I tried to summarize my conclusions in my previous reply, however.

95% is a number floating around here for quite some time; it's not mine. I interpret it to reflect the probability that a random wu will only need to be analyzed by the first two independent clients because they return matching results. Someone in CentralCommand should be able to run the numbers for us for better estimate. I seem to remember a number like 99% from Matt as part of the justification taking us from n=3 to n=2 a long while back. But if it is 99% or 89%, the argument remains the same.

Great news that you are testing the no-redundancy option. Let us know what the test parameters are and success criteria, when you get a chance and have dancing fingers. Thx.

So, how can you test for a false negative?

(That is, there's a signal there but not reported... Could be THE signal.)

Happy crunchin',
Martin

I think Matt may have touched on the answer you are looking for here, at least that will force the client to return some result to be double checked either server-side or via a "light-weight" form of a redundant WU. But is your question here to see if a WU contains "better" signals than those reported that were (inadvertantly or otherwise) omitted? I think that may be where the random full validation would have to cover things.

Basically, for each positive signal coming in we have a high confidence (>95%) of detecting it with a single client. This confidence level is limited primarily by the reliability of the typical client and the ability (assumption?)...

DJStarfox:
CCPhillips understands how to find false positives, and my general point, very well and only needed 3 lines. You take a positive and check and double check, just like any scientific result. You can't prove anything, but you can begin to believe it when results repeat enough times.

The analogy to complex systems may be satisfying, but proof by analogy is no proof. (Sorry, I probably just got all the Santa Fe institute simulators all upset with that PGIO (penetrating glimpse into the obvious)

My point was, if detecting a signal in a WU was a deterministic process/algorithm, then you'd only need one "success" returned (quota 1) to get a valid answer. Then, this discussion would be over quickly.

If the process is stochastic, then you'll need at least 2 answers, preferably 3 or more. By only having quota 2, SETI is balancing statistical error with efficiency of only processing each WU twice (instead of 3 times).

If CCPhillips understands and can provide extra error checking for false positives, that is still good either way. I would support such an addition to the code in principle.

... false positives, ...

Annoying but easily (if expensively) verified away. More of a problem is that of forever losing signals due to false negatives.

The analogy to complex systems may be satisfying, but ...

Redundancy is usually good for adding an aspect of robustness. It's all down to how 'clever' you can be about it.

ML1: does anyone ever conclude anything in a thread?

Sometimes, yes. Otherwise, you just meander onwards...

... 95% is a number floating around...

So does that mean that about 5% of the returned information is polluting junk?...

Also note that if cheaters see any aspect that is not fully checked, they will use that to scam credits, even if that may well sabotage the science database.

There are some very 'strange' people out there... There are also many "happily ignorant" people out there...

Happy crunchin',
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)

I think Matt may have touched on the answer you are looking for here, at least that will force the client to return some result to be double checked either server-side or via a "light-weight" form of a redundant WU. But is your question here to see if a WU contains "better" signals than those reported that were (inadvertantly or otherwise) omitted? I think that may be where the random full validation would have to cover things.

I was thinking along the lines of how reliable the n=1 system is, measured using some sort of metrics. Those that come to mind would be 1) can the ready to send queue keep up; 2) can the database keep up; 3) how much is peak and average storage requirements affected; 4) some sort of network metric if possible; 5) any increase in reported 'positives' versus the n=2 production system. I'm sure there are more.

False positives: if we found a positive today, seti would double and triple check it to be sure. A problem with the n=1 solution is if the positive return rate were to change 'appreciably' from what we have today, incurring more work. 'Appreciably' would need to be decided upon, of course.

95% today, if that is the number, means that 5% of the wu's need be validated with a third client (at least). I'm not talking about those wu's with n>2 because of bad downloads or time-out conditions. I think that today, with n=2, there is no poluting junk because the database doesn't get an entry that hasn't come from 2 clients. Of course, both of those clients could be wrong I suppose.

I wonder if this cheater thing is really an issue, because I believe that seti has the power to remove credit or ban users. We are part of a science project and not a numbers racket, after all! At least for the false positive case, or any wu with incredible credit claims, the cheaters should be easy to sort out. A few cheaters reporting inflated garbage may be an issue unless some sort of sampling is introduced (see the other thread). Like the US-IRS mechanism, people behave only because there is a 1% or less probability that they will be audited each year. I envision the best solution would be a 1<=n<2 type of process, where the average n drops below 2 at the administrator's perogative. We could get some performance boost and maintain a reasonable check this way.

False positives: ...

Are not the problem other than from innocent random junk from bad machines. It's a mess to clean up after the event, but that's a cost that can be balanced against the overall speedup benefit.

The killer problem is that of false negatives.

The only work-around I can see for that is for s@h to inject a test signal into every WU... But then again, a cheater can cheat as soon as the test signal is found and 'forge' the rest of the results in less than a second... The s@h server-side checks would need to be more sophisticated and perhaps test for and check for a random signal at a random strength... That gets the server-side more complicated and more CPU costly...

95% today, if that is the number, means that 5% of the wu's need be validated...

The problem there is to accurately identify the "5%" or whatever number you might have...

I wonder if this cheater thing is really an issue...

At the moment, we do not know that it exists other than for a small case of inflated credits.

Cheaters nearly brought down s@h-classic. Some of the lessons learned there are incorporated into Boinc. However, Boinc is just another set of rules that can be twisted for cheating for 'whatever'.

Note that cheaters may not even understand the science and likely care nothing for it. Boinc can be seen purely as a game to gain "credits". A recent example is that of a certain 'sneak' whom hijacked a large number of Boinc groups to then **** to then claim the credits... Unfortunately, with no WU redundancy or conditional redundancy, I can easily dream up ways to gain cobblestones for greatly reduced processing time. The easiest ways mean that we suffer false negatives...


If we have gone to the trouble to collect data, we should analyse that data reliably. Doing an unreliable 'quick scan' is futile even with twice the throughput if you've got a good chance of missing what you're looking for!

Keep searchin',
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)