Statistics of Valid/Invalid/Inconclusive

Message boards : Number crunching : Statistics of Valid/Invalid/Inconclusive
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Cloak & Dagger
Avatar

Send message
Joined: 14 May 11
Posts: 2
Credit: 66,735
RAC: 0
United States
Message 1531461 - Posted: 24 Jun 2014, 4:16:37 UTC

I've looked around a bit and can't find this type of thing addressed elsewhere (it's probably somewhere already, so please direct me if that is the case), but...

What are the general statistics on how often an 'invalid' or 'inconclusive' come about following a process? I've got 2 machines that I run and one seems to give an inconclusive 1/4 times, and the other 1/10. Is that just stochastic fun or does it mean one machine does a better job? Anyone?
ID: 1531461 · Report as offensive
Profile Zalster Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 27 May 99
Posts: 5517
Credit: 528,817,460
RAC: 242
United States
Message 1531475 - Posted: 24 Jun 2014, 4:43:00 UTC - in response to Message 1531461.  
Last modified: 24 Jun 2014, 4:44:14 UTC

I don't know if they actually track that sort of thing. Maybe they do but maybe more important is the distinction between the 2 terms you mentioned. First, each work unit is sent to 2 computers. They are given a certain amount of time to finish analyzing the data. Once they are done, the results are sent back and await the result from the other computer. Once both computers have reported, they are compared to each other. If they match, then the results are considered valid and credit is given. If the results don't match close enough then they are considered inconclusive, and the work unit is sent to a 3rd computer to analyze and that result will then be compared against the first 2 results. This can go on for a maximum of 5 computers. Depending on which results match, those computers are given a valid status and the results that doesn't match are considered invalid. If none of the 5 computers match then an error result is given and no computer is given credit or maybe a partial credit. Now, how often will you get a inconclusive, depends on how close your "wingman" is or isn't to your results. Since you typically will have different wingmen for each work unit (you may be pair with someone for a couple of work units but never for all of your work units) you can't really tell how often you will get an inconclusive. Invalids however, may be something altogether different. An occasional invalid is par for the course. But if you start to see large number of invalids say like 20-30 or more then you have to consider the possibility that there is a problem somewhere in your computer processing (be it the chip, graphic card, temps, drivers, etc). Hope this helps.


Zalster
ID: 1531475 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1531483 - Posted: 24 Jun 2014, 5:31:15 UTC

Your figures will vary based on the amount and type of computing you do. For me my stats are: Validation pending (449) · Validation inconclusive (45) · Valid (427) across about 30 machines.
I don't pay any attention to "inconclusive" results unless I have any invalid ones. Generally after the third result is sent in you will find all three the results are marked valid. Because they match "close enough"
I have 1 machine that coughs up an invalid CPU task about once a week and then an old NVIDIA GPU that goes on a killing spree every so often. I have also seen an invalid come up after a machine hard locks. Otherwise an invalid is rare for me.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1531483 · Report as offensive
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 1531497 - Posted: 24 Jun 2014, 6:21:00 UTC

In addition to what Zalster posted, there are some other details to consider.

Once a valid or invalid judgement has been made, the records are usually purged from the database and no longer visible 24 hours later. An inconclusive remains visible while waiting for another host to resolve the uncertainty, and of course for 24 hours after that resolution. So the inconclusive count is for a considerably longer period than the valid and invalid counts.

Inconclusives are cases where one or more signals in the comparison of two uploaded result files do not match. They may transition to either valid or invalid, and may be given the "valid" designation and granted credit based on a "weakly similar" comparison which means at least half the signals match the result which is declared canonical and assimilated to the science database.

If you have a task in inconclusive state, it is good to check which host's result was declared canonical when the final judgement has been made. If your result is canonical, you know that all its signals matched. Or if your result was the 3rd or 4th in and the validator was then able to choose a canonical result from among earlier results, that also indicates all signals matched the canonical one.

There are some flaky hosts which produce a large number of inconclusives but often get credit on the "weakly similar" basis so look to be quite productive based on credits. In fact, they are simply wasting electricity since an additional host must crunch the same workunit.

One of your inconclusives now, WU 1527763732 is a case where the counts of reported signals match. Most commonly that indicates a very small calculation difference which just happens to affect what's in the uploaded results enough to be doubtful. In addition to the counted signals, result files contain some "best" signals which the validator also checks and the difference may be there.

The other inconclusive now, WU 1526636643 is a case where your processing ran full length and the wingmate's host "found" a bunch of Spikes very quickly so quit with a result_overflow. That is almost certainly an indication of a problem with the wingmate's processing.

Based on the "Consecutive valid" counts for each of the app versions on each of your hosts, I don't think either host has a problem. OTOH, there's not enough data yet to start considering real statistical analysis.
                                                                  Joe
ID: 1531497 · Report as offensive
Profile Donald L. Johnson
Avatar

Send message
Joined: 5 Aug 02
Posts: 8240
Credit: 14,654,533
RAC: 20
United States
Message 1531503 - Posted: 24 Jun 2014, 6:34:57 UTC
Last modified: 24 Jun 2014, 6:39:34 UTC

Talking a little more about inconclusive results, I see one of your computers has a GPU, the other does not. I run only stock CPU apps. As best I can remember, when my boxes get inconclusives, it is one of two situations:
1) My 1st wingman is a GPU (once in a great while a CPU) that is malfunctioning, generating nothing but -9 overflow/ 30 signals per Work Unit.

2) My wingman is a GPU that, because of the differences inherent between CPU and GPU processing, finds 1 more or 1 less signal than me above the reportable threshold.

In case 1) the 2nd wingman usually matches my result, so he and I are validated, and the malfuntioning box's result is ruled Invalid. I have heard of situations where the 2nd wingman is also malfunctioning, and validates the -9 overflow with the correct result ruled invalid, but it has never happened to me.

In case 2), the 2nd wingman may match either of the original results, and those two results will validate. But, if the non-exact result matches at least 50% of the signals in the valid result, it, too, will validate and get credit.

So it's pretty much the luck of the draw as to your wingman. Inconclusives happen, but usually resolve themselves to valid results and credit gramted. Worry if you are getting a lot of invalids or a large number of -9 overflow results, which indicates a malfunction in your box.

edit} and while I was composing, Joe poppped in with a more complete answer.
Donald
Infernal Optimist / Submariner, retired
ID: 1531503 · Report as offensive
Rasputin42
Volunteer tester

Send message
Joined: 25 Jul 08
Posts: 412
Credit: 5,834,661
RAC: 0
United States
Message 1531594 - Posted: 24 Jun 2014, 13:11:09 UTC

Maybe, it is a good idea to shut down the computer once a week or so.
There may be cumulative errors, if the computer runs for weeks on end, without ever being powered down.
ID: 1531594 · Report as offensive
Profile Cloak & Dagger
Avatar

Send message
Joined: 14 May 11
Posts: 2
Credit: 66,735
RAC: 0
United States
Message 1531737 - Posted: 25 Jun 2014, 2:58:23 UTC

Thank you to all for your responses. They are most helpful and I appreciate the time you have all taken in replying to my initial post. Thank you Josef for your specific detail, in particular.

My main goal was making sure that I was contributing in a fulfilling way to SETI@home and not simply burning processing time at the loss of both my machines and SETI@home's results suite. I don't come here for competition, just for contribution, so knowing that what my machines are doing is within (so far) reasonable results is what I'm after.

Rasputin42: Both machines are shut off daily. The only odd thing I've ever found is something that occurs not with SETI@home, but Einstein@home (which I no longer run). My one machine is set up to only run when plugged in, and whenever the battery was full I'd unplug (I guess I'm classically trained to do so now). Anyway, Einstein@home apps would reset every single time this happened rather than pause progress. This problem is why I no longer run it. I've never seen this for SETI@home, and that is really the BOINC I came for anyway. Einstein was a bonus for me.
ID: 1531737 · Report as offensive

Message boards : Number crunching : Statistics of Valid/Invalid/Inconclusive


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.