There, I said it: after 25 years working on SETI@home, and 5 years working on Nebula, we're finally on the home stretch. The finish line (producing a final candidate list, and writing a paper) is in view, maybe a few months off. It's been a slog, and I'm eager to have it behind me.

A full-sky Nebula run

Over the last several years (as chronicled here) we've been developing and refining algorithms for detecting RFI, generating birdies, finding signal candidates (multiplets) and scoring these candidates. For these purposes it wasn't necessary to look for multiplets in all 15 million pixels; I generally looked in only 256K per cycle, sometimes 1M. Finding multiplets uses lots of computing and produces lots of files (2 per pixel). I didn't want to overstay my welcome at the Atlas computing cluster.

But a few weeks ago we decided that these algorithms had reached the point of being good enough, and I did, for the first time, a Nebula run that scored all 15 million pixels. This went faster than I thought it would. I processed batches of 1.25M pixels at a time, and each batch took only a few hours, running on about 1000 cluster nodes.

It turned out that the "multiplet uniqueness" step - ensuring that multiplets in adjacent pixels are disjoint - would take pretty long, like a week. So I did a minor rewrite of this program to increase its efficiency; now it's down to about a day.

Also, copying the 30M or so files from Atlas (Germany) to Centurion (Berkeley) took a couple of days. It seemed to confuse rsync; I had to do a couple of tries to get everything copied.

Keeping birdies separate

Until now, we've processed and scored birdie detections and real detections together. I decided to change things so that we do two separate Nebula runs: a) using only real (non-birdie) detections, and scoring all pixels; b) using real and birdie detections, and scoring only birdie pixels. There are two reasons for this:

• A birdie could mask (i.e. prevent us from detecting) an actual ET signal in the same pixel. The odds of this are small - with 3,000 birdies, only .02% of pixels have a birdie - but still.
  
• We may want to change the number or parameters of our birdies, perhaps repeatedly. It would be good to be able to do this without rescoring all pixels (and transferring all the files).
  

It took me a while to figure out a clean way to keep the two runs separate, in terms of files. I was already keeping all score-related files in a subdirectory, score/. What I settled on is:

• On Atlas, everything stays the same. When we do a run, either birdie or non-birdie, the results go in score/.
  
• On Centurion, we have a new directory score_birdie/. After a birdie run, we copy score/ on Atlas to score_birdie/ on Centurion.
  

This meant changing the scripts and PHP pages that run on Centurion to look for birdie-related data in score_birdie/, rather than in specially-named files in score/. This wasn't hard to do, and actually makes things a bit simpler.

Human rating of multiplets

The final output of Nebula is a bunch of score-ranked lists of multiplets. We already know that some fraction of these will be RFI of the sort that's apparent to a human observer, but hard to identify algorithmically. That's OK. The goal our RFI algorithms is not to remove 100% of RFI; doing so would probably remove ET signals too. The goal is to remove enough RFI that a good fraction (say, at least half) of the top-ranking multiplets are not obvious RFI.
The final (post-Nebula) stages of SETI@home are

• Manually examine the top few thousand multiplets, in all the various categories and score variants, and remove the ones that are obvious RFI. At least initially we'll do this as a group, on Zoom, to make sure we agree on what constitutes obvious RFI.
  
• Make a list of the multiplets that remain; re-observe those spots in the sky (hopefully using FAST), analyze the resulting data (probably using the SETI@home client running on a cluster) and see if we find detections consistent with the multiplets. If we do, maybe that's ET.
  

So I extended our existing "bookmark" system to let you rate multiplets. When you bookmark a multiplet you can now give it a 0-10 rating, as well as a comment. The web page for each multiplet shows the ratings that have been reported so far. This mechanism is intended for use by our group (me, Eric, Dan, Jeff) but any SETI@home user can browse and rate multiplets. Feel free to do so!