All in the Timing IV

Message boards : Nebula : All in the Timing IV
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile David Anderson
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 13 Feb 99
Posts: 164
Credit: 502,653
RAC: 0
Message 2070907 - Posted: 17 Mar 2021, 8:39:28 UTC

(The title refers to a group of one-act plays by David Ives.)

A while back, Eric and I had the idea of "observation pruning": removing spikes whose duration (a function of their FFT length) is too short for the observation in which they occur.

For example, suppose a 13-second spike S (the longest spike, hence the narrowest bandwidth) occurs at a time when the telescope beam is moving at a speed where it crosses a point in the sky in 1 second. Then it's unlikely that the source of S is a cosmic signal; it's more likely RFI or noise. And if S is from a cosmic signal, that signal will probably be detected with greater power at a shorter FFT length.

Anyway, a few weeks ago I wrote some code to do observation pruning. But - surprise! - I didn't get it right the first time.

I decided to do observation pruning in 'nebula_score', the program that finds and scores multiplets, because that's the only place where we do things on a per-pixel basis, and fetching a pixel's observation history is expensive.

For a given pixel P, nebula_score reads and processes detections from a sky "disc" that includes parts of the 8 adjacent pixels. In my first attempt, I was comparing the intervals of these detections with the observation intervals of P (for the beam in which the detection occurs). This removed too many detections - not surprisingly, since the adjacent pixels have different observation intervals.

In my second attempt, I merged (for each beam) the observation intervals of all 9 pixels and compared against that. But this didn't help at all! I had a panicky feeling that there was some insidious problem in how I was computing the intervals, or in the data itself. I started working on tools for studying this.

But then I looked at the code for a while, and I realized the problem was that, after concatenating the 9 lists of observation intervals, I wasn't time-sorting the result, as required by the pruning logic.

A few seconds later, I realized that merging the lists was the wrong idea to begin with! I should just compare each detection against the observation intervals for the pixel that it's in. D'oh!

It wasn't immediately clear how to implement this efficiently, but eventually I figured out a way that, although not quite optimal, required changing only a few lines of code. Sweet.

I tested this on a few pixels and it seems to work; it removes about 10% of spikes, which is about what I expected. I did a scoring run using the new version of nebula_score; it's online now.

This experience highlights a big problem in Nebula: we're not doing design review or code review. The only programmers are Eric and me. Each of us has other projects - Eric has ICON, I have BOINC - as well as our Nebula work. Neither of us has quite enough time to look carefully at what the other one is doing. We pay a price for this: we make mistakes, and it costs us time. Eric has been mired in birdie_gen for months now, while I lurch along with RFI and multiplet stuff. And, or course, the real fear is that we're making undetected mistakes.

In case you, Dear Reader, are in a position to help, the Nebula source code repo is here. Warning: the learning curve is huge.
ID: 2070907 · Report as offensive     Reply Quote
Profile ML1
Volunteer moderator
Volunteer tester

Send message
Joined: 25 Nov 01
Posts: 13626
Credit: 7,508,002
RAC: 20
United Kingdom
Message 2071083 - Posted: 18 Mar 2021, 18:21:25 UTC

Thanks for some good insight into some real science in action.

From my own experiences, I sympathize greatly for the problem of juggling multiple development tasks distracting from all the tasks.

Is there scope for sponsorship for an undistracted 3rd pair of programmer's eyes to focus on the code and tools?


Meanwhile also, thanks for good detail and insight for analyzing the s@h results!


Keep searchin'!

Best regards,
Martin
See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)
ID: 2071083 · Report as offensive     Reply Quote
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6320
Credit: 106,370,077
RAC: 121
Russia
Message 2071642 - Posted: 23 Mar 2021, 21:41:25 UTC - in response to Message 2070907.  
Last modified: 23 Mar 2021, 21:42:58 UTC

Thanks for sharing.

<div style="width:640px">(The title refers to <a href=https://en.wikipedia.org/wiki/All_in_the_Timing>a group of one-act plays by David Ives</a>.)

A while back, Eric and I had the idea of "observation pruning": removing spikes whose duration (a function of their FFT length) is too short for the observation in which they occur.

For example, suppose a 13-second spike S (the longest spike, hence the narrowest bandwidth) occurs at a time when the telescope beam is moving at a speed where it crosses a point in the sky in 1 second. Then it's unlikely that the source of S is a cosmic signal; it's more likely RFI or noise. And if S is from a cosmic signal, that signal will probably be detected with greater power at a shorter FFT length.

Too long, perhaps, not too short. At least your example implies this.


This experience highlights a big problem in Nebula: we're not doing design review or code review. The only programmers are Eric and me. Each of us has other projects - Eric has ICON, I have BOINC - as well as our Nebula work.

Regarding BOINC and your work on it: could you please evaluate current implementation of max_concurrent tag for apps in app_config.xml and quite important design problems it opens up.
Discussion with examples on BOINC forum.

the Nebula source code repo is <a href=https://sourceforge.net/p/seti-science/code/ci/master/tree/>here</a>.

Thanks for the link. Can you list papers that describe current Nebula design if any (if not - would be worth to write)?
SETI apps news
We're not gonna fight them. We're gonna transcend them.
ID: 2071642 · Report as offensive     Reply Quote
Sesson

Send message
Joined: 29 Feb 16
Posts: 19
Credit: 1,353,463
RAC: 3
Message 2072260 - Posted: 31 Mar 2021, 15:46:37 UTC

I will be very willing to help to do science, at home and without becoming a scientist.
ID: 2072260 · Report as offensive     Reply Quote

Message boards : Nebula : All in the Timing IV


 
©2021 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.