Fun with 618,813 pulses...

Hi again everyone. Looks like I've broken my promise of weekly blog entries by 41 days or so. Oh well, nobody's perfect (except Angela, of course). Anyway, you know those times when you get so involved in your work that you do things like forget to eat or go home? It's been like that. (That and, unfortunately, I can't really bring myself to believe that posting to a web site is really part of what I get paid to do.)

This week I've been busy thinking about how to remove radio frequency interference (RFI) from the Astropulse database. As you may recall, Astropulse has a big problem with radar. There's a radar blanking system at the observatory, but that only tracks the FAA radar. There are about 6 other radar systems that we see in our data, and the radar blanking system does nothing about them. Matt's working on a way to remove the rest of them, but until then we need to figure out a way to work around them.

This is a waterfall plot of 618,813 pulses we've gotten back from Astropulse version 5.03 (as of yesterday). Rather than the usual spectrogram type waterfall which would be a plot of frequency versus time, this is a plot of dispersion measure (DM) versus time. DM in this case is how much sooner the high frequencies arrived than the low frequencies did. A dispersion measure of 15000 means that the 1.42125GHz radiation arrived 15000 samples (6 milliseconds) before the 1.41875GHz radiation did. I've compressed the time axis to get rid of the spaces between different observations.



The regions that are radar contaminated are pretty obvious. Anyplace that doesn't look smooth and random is radar. The easiest way is just to throw out any workunit where Astropulse says it detects multiple pulses that saturate the amplifiers. That's shown in the next plot.



That does a pretty good job of removing the worst times. But it leave behind some vertical RFI features, one at a DM of -14,200, another at a DM of 5,600, another at a DM of 11,000, etc. But we have other ways to remove RFI. RFI tends to show up at specific dispersion measures. One way is if the signal stays around when you are looking at another part of the sky. So I threw together a program (written in IDL) that grabs 10,000 pulses from around the same time, but from different parts of the sky from a signal I want to check, and then calculates the probability than that many signals would occur at that dispersion measure due to random noise. If the probability is less than 1 in a thousand, I consider it to be RFI. If there were no RFI, it would still remove one out of every thousand signals, but that's an acceptable loss. The next image is the signals that this algorithm determined to be RFI (25% of the total) the first time it was run.



And here's what it didn't think was RFI...



It's got some problems left in the radar region, but it got rid of a few of the leftovers that the previous method left. Best of all, I can run it multiple times on what didn't get flagged the first time. The second time I ran it, it removed 4% more of the signals. I'm running it a third time right now, but I haven't made images of those runs.

If we use both methods, we've got a pretty good system. Here's what's left over if I combine the first run of the RFI rejecter with rejecting the signals from workunits where there were multiple amplifier saturation events.



Now all that's left in the short term is to port the algorithm to C++ and have it write the results back into the database. I'd also like to know what that splat is at an index of 70 and a DM of 300 is. But that's a story for another time...

I'll try to make my next post in less than a week to make up for my missing time.
____________

Thank you very much for so interesting and illustrative post.
If you could continue the story later it would be wonderful!

Oh well, nobody's perfect (except Angela, of course).

i have a question about seti mybee it is dumb but, one of the things you always need is hard drives. why does seti not use peoples spare hd space the same as they use spare prossecor time? or do they? there is alot of spare hd space. i have two 20 gig drives in my machine i do not even use. i know 20gig is a drop in the bucket but 20 gig times all your users?

i have a question about seti mybee it is dumb but, one of the things you always need is hard drives. why does seti not use peoples spare hd space the same as they use spare prossecor time? or do they? there is alot of spare hd space. i have two 20 gig drives in my machine i do not even use. i know 20gig is a drop in the bucket but 20 gig times all your users?

I personally, would NEVER allow ANYONE access to my hard drives for ANY reason. You would be seriously asking for trouble. What a ridiculous idea! ...

i have a question about seti mybee it is dumb but, one of the things you always need is hard drives. why does seti not use peoples spare hd space the same as they use spare prossecor time? or do they? there is alot of spare hd space. i have two 20 gig drives in my machine i do not even use. i know 20gig is a drop in the bucket but 20 gig times all your users?

Ever read the "EULA"s?

(My understanding is that Microsoft users agree to Microsoft doing anything it wishes with your system and your files/data and all with utter impunity. It is well reported that Microsoft takes details of all the software on your system, and likely other details...)