Evolution of the SETI@home back end

The original back end

At the outset of SETI@home we decided to base the back end on a SQL database; we used Informix, a commercial DB system. We designed a database schema that included the following tables:

These tables form a hierarchy; each row contains the ID of a row in the table above it.

Eric Korpela and Jeff Cobb developed the back-end software using C++ and IDL. All data access was through the database server. For example, the scoring program issued SQL queries to get signals from the database, and it stored the multiplets in another database table.

Problems with the original back end

The main problem with the original back end is that was slow. This slowness was due primarily to two factors:

In addition, parts of the back end were never implemented:

The underlying problem was that the SETI@home project has never had adequate funding, and the people who could have completed and improved the back end (like Jeff and Eric) were often involved in "fire-fighting" - dealing with hardware and sysadmin problems to keep the front end working.

History

In March 2003, when the signal DB was still relatively small, we completed one cycle of back-end processing: we removed RFI, found the highest-scoring pixels, and re-observed them (no persistent signals were found). See an article about this.

As the signal DB increased in size, it became apparent that we couldn't repeat this: it would take years to do RFI removal and scoring. So in 2007 Jeff Cobb began working on a new approach called the Near-Time Persistency Checker, or NTPCkr. The idea was to keep track of the set of pixels for which new signals had arrived recently, and to do RFI removal and scoring for these pixels. Eventually we realized that even this was too slow to keep up with the arrival of signals, and it was shelved.

So for a number of years - 2003 to 2015 - SETI@home has continued to accumulate signals, with no viable strategy for back-end processing.

For most of this period I wasn't working much on SETI@home. In 2002 I got NSF funding to work on BOINC, and my involvement in SETI@home was reduced; mostly I worked on maintaining the web site, and developing BOINC features needed by SETI@home.

In 2015 my BOINC funding ended and I was looking for work. I talked with Dan Werthimer. SETI@home had some money from the Templeton Foundation, and I proposed working for SETI@home with the goal of making the back end functional. It was clear that we needed to moving things off our own computers, onto a scalable platform such as a commercial "cloud". So I called this new project Nebula.

Nebula is partly new code, partly a reworking of the original back-end code developed by by Jeff and Eric. In the course of developing Nebula I've worked closely with Eric, who worked out the mathematics behind the algorithms.

Next: Nebula goals.




 
©2018 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.