Joined: 13 Feb 99
(The title refers to a group of one-act plays by David Ives.)
A while back, Eric and I had the idea of "observation pruning": removing spikes whose duration (a function of their FFT length) is too short for the observation in which they occur.
For example, suppose a 13-second spike S (the longest spike, hence the narrowest bandwidth) occurs at a time when the telescope beam is moving at a speed where it crosses a point in the sky in 1 second. Then it's unlikely that the source of S is a cosmic signal; it's more likely RFI or noise. And if S is from a cosmic signal, that signal will probably be detected with greater power at a shorter FFT length.
Anyway, a few weeks ago I wrote some code to do observation pruning. But - surprise! - I didn't get it right the first time.
I decided to do observation pruning in 'nebula_score', the program that finds and scores multiplets, because that's the only place where we do things on a per-pixel basis, and fetching a pixel's observation history is expensive.
For a given pixel P, nebula_score reads and processes detections from a sky "disc" that includes parts of the 8 adjacent pixels. In my first attempt, I was comparing the intervals of these detections with the observation intervals of P (for the beam in which the detection occurs). This removed too many detections - not surprisingly, since the adjacent pixels have different observation intervals.
In my second attempt, I merged (for each beam) the observation intervals of all 9 pixels and compared against that. But this didn't help at all! I had a panicky feeling that there was some insidious problem in how I was computing the intervals, or in the data itself. I started working on tools for studying this.
But then I looked at the code for a while, and I realized the problem was that, after concatenating the 9 lists of observation intervals, I wasn't time-sorting the result, as required by the pruning logic.
A few seconds later, I realized that merging the lists was the wrong idea to begin with! I should just compare each detection against the observation intervals for the pixel that it's in. D'oh!
It wasn't immediately clear how to implement this efficiently, but eventually I figured out a way that, although not quite optimal, required changing only a few lines of code. Sweet.
I tested this on a few pixels and it seems to work; it removes about 10% of spikes, which is about what I expected. I did a scoring run using the new version of nebula_score; it's online now.
This experience highlights a big problem in Nebula: we're not doing design review or code review. The only programmers are Eric and me. Each of us has other projects - Eric has ICON, I have BOINC - as well as our Nebula work. Neither of us has quite enough time to look carefully at what the other one is doing. We pay a price for this: we make mistakes, and it costs us time. Eric has been mired in birdie_gen for months now, while I lurch along with RFI and multiplet stuff. And, or course, the real fear is that we're making undetected mistakes.
In case you, Dear Reader, are in a position to help, the Nebula source code repo is here. Warning: the learning curve is huge.
Joined: 25 Nov 01
Thanks for some good insight into some real science in action.
From my own experiences, I sympathize greatly for the problem of juggling multiple development tasks distracting from all the tasks.
Is there scope for sponsorship for an undistracted 3rd pair of programmer's eyes to focus on the code and tools?
Meanwhile also, thanks for good detail and insight for analyzing the s@h results!
See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)
Joined: 16 Jun 01
Thanks for sharing.
<div style="width:640px">(The title refers to <a href=https://en.wikipedia.org/wiki/All_in_the_Timing>a group of one-act plays by David Ives</a>.)
Too long, perhaps, not too short. At least your example implies this.
Regarding BOINC and your work on it: could you please evaluate current implementation of max_concurrent tag for apps in app_config.xml and quite important design problems it opens up.
Discussion with examples on BOINC forum.
the Nebula source code repo is <a href=https://sourceforge.net/p/seti-science/code/ci/master/tree/>here</a>.
Thanks for the link. Can you list papers that describe current Nebula design if any (if not - would be worth to write)?
SETI apps news
We're not gonna fight them. We're gonna transcend them.
Joined: 29 Feb 16
I will be very willing to help to do science, at home and without becoming a scientist.
©2022 University of California
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.