Recall that the SETI@home database uses a hierarchy of tables. In processing detections there are several places where we need information from higher-level tables.

• Redundant flag (tape): Sometimes the data from a tape is re-processed using a later version of the client. In this case we create a new tape record (and a new set of workunits and results linked to it) and set a flag in the old tape record. Detections linked to the old tape must be ignored.
• Beam (tape): which of the seven beams the data is from.
• Angle range (workunit group): the angular range in sky position over the duration of the workunit. This is an indicator of whether the telescope was tracking; it's used in multi-beam RFI removal.

Nebula "flattens" the table hierarchy to create arrays, indexed by result ID, of these items. Given a detection (which contains a result ID) we can immediately look up the info in these arrays, without accessing other tables.

Flattening the table hierarchy

The needed files are created by "pushing down" information, starting from the tape table and working down. Memory-mapped files are used at each step. The steps are as follows:

• Unload the tape, workunit group, and workunit tables to DB dump files.
• Create binary files mapping tape ID to redundant flag and beam. This is done by memory-mapping the files, scanning the tape dump file, and writing the redundant and beam values to the arrays.
• Create binary files mapping WU group ID to redundant flag, beam, and angle range. To do this: map these files, and the files created in step 2. Scan the WU group dump file. Each WU group has a tape ID. Copy the redundant flag and beam from the tape arrays to the appropriate locations in the WU group arrays. Write the angle range in the array.
• Create binary files mapping WU ID to redundant flag, beam, and angle range. To do this: map these files and the files created in step 3. Scan the WU table dump file, copying items from the WU group arrays to the WU arrays.
• Same thing for the result table.

The resulting files are 12.1 GB (angle range, 32-bit float), 3.1 GB (beam number, 8-bit integer) and 375 MB (redundant flag, 1 bit).

---

Next: The Nebula pipeline.