validation testing

Message boards : Number crunching : validation testing
Message board moderation

To post messages, you must log in.

AuthorMessage
PhonAcq

Send message
Joined: 14 Apr 01
Posts: 1656
Credit: 30,616,848
RAC: 81
United States
Message 1918315 - Posted: 11 Feb 2018, 20:13:30 UTC

Does anybody know if SETI puts a fake-signal into the occasional work unit to validate that it would be detected by the distributed system? There are all sorts of objective things that could be tested, of course. They could clandestinely give the same file to numerous (lots) of rigs to see if there is any variance. Etc. And, frankly it wouldn't cost much time away from the real signal, even it meant one work unit once a year per host and application. I doubt that the beta-seti project would be a valid test, given what ever differences exist on beta versus the standard project.

We never hear of this sort of activity, but it would be appalling if they weren't doing some sort of full process validation from time to time. I just haven't seen such a step posted or described anywhere.
ID: 1918315 · Report as offensive
rob smith Special Project $250 donor
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 16158
Credit: 313,596,769
RAC: 253,879
United Kingdom
Message 1918348 - Posted: 11 Feb 2018, 21:36:09 UTC

SETI@Home uses correlation techniques to sort potential signals from the mountains of signals and noise. No single detection of a potential signal is counted as "valid", there must be at least two, and preferably a few more, spread over a period of time (months or years of observations).
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1918348 · Report as offensive
Profile Wiggo "Socialist"
Avatar

Send message
Joined: 24 Jan 00
Posts: 14269
Credit: 185,677,438
RAC: 82,380
Australia
Message 1918352 - Posted: 11 Feb 2018, 21:43:19 UTC

"Fake signals" are injected into SETI's back end, Nebula, for correctness.

Cheers.
ID: 1918352 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11992
Credit: 118,824,940
RAC: 40,208
United Kingdom
Message 1918358 - Posted: 11 Feb 2018, 22:02:57 UTC

Or you could look for something like "birdie signals" from way back when.
ID: 1918358 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 4504
Credit: 281,561,511
RAC: 629,149
United States
Message 1918361 - Posted: 11 Feb 2018, 22:18:06 UTC - in response to Message 1918358.  

Thanks for the 'blast from the past' from Matt. Was wonderful having his technical updates posted regularly. Wish that frequency of project news happened still.
Seti@Home classic workunits:20,676 CPU time:74,226 hours
ID: 1918361 · Report as offensive
PhonAcq

Send message
Joined: 14 Apr 01
Posts: 1656
Credit: 30,616,848
RAC: 81
United States
Message 1918481 - Posted: 12 Feb 2018, 13:14:27 UTC - in response to Message 1918348.  

Thanks Rob. I don't think that the validation you mention is what I'm asking about. In my mind that even when sending out redundant tasks it is possible that the algorithm(s) might produce two matching results on two different hosts but still be unable to detect a true signal at some threshold level.
ID: 1918481 · Report as offensive
PhonAcq

Send message
Joined: 14 Apr 01
Posts: 1656
Credit: 30,616,848
RAC: 81
United States
Message 1918482 - Posted: 12 Feb 2018, 13:21:48 UTC - in response to Message 1918352.  

Wiggo, I haven't tracked Nebula. But my question may be different. I wonder about how much we can trust the algorithm used to process the original data, which by implication also tests the splitters etc upstream, I would guess. This venerable project is so old and apparently complex that we shouldn't be over confident-- scientific skepticism is a healthy trait.
ID: 1918482 · Report as offensive
Profile mr.mac52 Project Donor
Avatar

Send message
Joined: 18 Mar 03
Posts: 62
Credit: 215,600,508
RAC: 83,911
United States
Message 1918553 - Posted: 12 Feb 2018, 20:55:56 UTC - in response to Message 1918358.  

Ah Richard, you've kicked my brain drip pan with your 'birdie' comment.

In WWII when radar was being first used, the operators did not know if the radar was actually working when they had no targets. The solution was to use a crystal radio, no power needed, and listen for the RF pulse to go out. The outgoing RF pulse in the crystal radio sounded like a bird chirp. When I worked on a radar in the late '70s at TI, there was a signal called 'chirp' and one called 'chirp gate' that triggered the outbound pulse of RF.

Another such story is that of telephony where the wires names, at least in the USA, were referred to as 'tip', 'ring', and 'sleeve'. 'Tip' and 'ring' were the talking pair and 'sleeve' was a control ground. On the old so called cord boards, the incoming circuit was a number of jacks with cords attached and a field of sockets connecting to a end user. The operator would touch the 'tip' of the jack to the socket which was the 'sleeve' connection and if the end user was not using his phone, there would be no so called busy tone. The so called busy tone was, again in the USA, a 60Hz tone produced by the unbalanced line, 'tip' and 'ring', when one side was grounded. In the mid '70s, even the newest computerised telephone switches I worked on used 'tip', 'ring', and 'sleeve' or 'sleeve ground'

And then there was Tektronix schematics that would have somebody riding a bicycle down a circuit here and there.

Thanks for triggering such fond memoires Richard,

John
ID: 1918553 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 11992
Credit: 118,824,940
RAC: 40,208
United Kingdom
Message 1918768 - Posted: 14 Feb 2018, 12:10:25 UTC - in response to Message 1918553.  

Tip, ring, and sleeve must surely be familiar to anyone who works with stereo audio equipment? Presumably the audio industry found and adopted ready-made telephony connectors when they wanted to switch to stereo recording and playback.
ID: 1918768 · Report as offensive

Message boards : Number crunching : validation testing


 
©2018 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.