After the last pipeline run I found two major anomalies:

1) Lots of birdie signals (which in general should not get flagged as RFI) were in fact being flagged as drifting RFI. The drifting algorithm, described here, is kind of tricky. I isolated an example, and spent a lot of time adding printf()s to the code and staring at output files.

This didn't really help; I needed to visualize what was going on. So I wrote some code to generate a plot of the "drift rectangle". Here's the result:



The birdie signals are the vertical band on the upper left. In the two triangles with statistical excess, they're colored red. It doesn't look like the corresponding lower triangles have a statistical excess of signals, but on closer examination what look like individual signals are in fact "clusters" of nearly identical signals, and one of the triangles passes the excess test.

I identified this problem a long time and ago and fixed it by collapsing clusters into a single signal. But changes to the RFI code had broken this. Once I realized what was going on it was easy to fix.

2) Way too many pulses and triplets (like 90-95%) were being flagged as multi-beam RFI. This algorithm, described here, involves associating signals with rectangles in time/freq space, and using these to decide when 2 signals have the same source. We had been scaling these rectangles by large factors (5 and 11) based on long-ago data analysis. I removed this scaling and it improved the situation, although the % of pulses flagged as MB RFI - 42% - is still too high. Eric pointed out that we can add a period component to the similarity test, which will bring this down much further. That'll be in the next run.

Anyway, the results of these fixes are online. Please inspect them and flag anomalies.

- -David