Fun with 618,813 pulses...


log in

Advanced search

Message boards : SETI@home Staff Blog : Fun with 618,813 pulses...

1 · 2 · Next
Author Message
Eric KorpelaProject donor
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 3 Apr 99
Posts: 1092
Credit: 9,548,323
RAC: 36,735
United States
Message 877279 - Posted: 19 Mar 2009, 23:22:54 UTC
Last modified: 19 Mar 2009, 23:24:33 UTC

Hi again everyone. Looks like I've broken my promise of weekly blog entries by 41 days or so. Oh well, nobody's perfect (except Angela, of course). Anyway, you know those times when you get so involved in your work that you do things like forget to eat or go home? It's been like that. (That and, unfortunately, I can't really bring myself to believe that posting to a web site is really part of what I get paid to do.)

This week I've been busy thinking about how to remove radio frequency interference (RFI) from the Astropulse database. As you may recall, Astropulse has a big problem with radar. There's a radar blanking system at the observatory, but that only tracks the FAA radar. There are about 6 other radar systems that we see in our data, and the radar blanking system does nothing about them. Matt's working on a way to remove the rest of them, but until then we need to figure out a way to work around them.

This is a waterfall plot of 618,813 pulses we've gotten back from Astropulse version 5.03 (as of yesterday). Rather than the usual spectrogram type waterfall which would be a plot of frequency versus time, this is a plot of dispersion measure (DM) versus time. DM in this case is how much sooner the high frequencies arrived than the low frequencies did. A dispersion measure of 15000 means that the 1.42125GHz radiation arrived 15000 samples (6 milliseconds) before the 1.41875GHz radiation did. I've compressed the time axis to get rid of the spaces between different observations.



The regions that are radar contaminated are pretty obvious. Anyplace that doesn't look smooth and random is radar. The easiest way is just to throw out any workunit where Astropulse says it detects multiple pulses that saturate the amplifiers. That's shown in the next plot.



That does a pretty good job of removing the worst times. But it leave behind some vertical RFI features, one at a DM of -14,200, another at a DM of 5,600, another at a DM of 11,000, etc. But we have other ways to remove RFI. RFI tends to show up at specific dispersion measures. One way is if the signal stays around when you are looking at another part of the sky. So I threw together a program (written in IDL) that grabs 10,000 pulses from around the same time, but from different parts of the sky from a signal I want to check, and then calculates the probability than that many signals would occur at that dispersion measure due to random noise. If the probability is less than 1 in a thousand, I consider it to be RFI. If there were no RFI, it would still remove one out of every thousand signals, but that's an acceptable loss. The next image is the signals that this algorithm determined to be RFI (25% of the total) the first time it was run.



And here's what it didn't think was RFI...



It's got some problems left in the radar region, but it got rid of a few of the leftovers that the previous method left. Best of all, I can run it multiple times on what didn't get flagged the first time. The second time I ran it, it removed 4% more of the signals. I'm running it a third time right now, but I haven't made images of those runs.

If we use both methods, we've got a pretty good system. Here's what's left over if I combine the first run of the RFI rejecter with rejecting the signals from workunits where there were multiple amplifier saturation events.



Now all that's left in the short term is to port the algorithm to C++ and have it write the results back into the database. I'd also like to know what that splat is at an index of 70 and a DM of 300 is. But that's a story for another time...

I'll try to make my next post in less than a week to make up for my missing time.
____________

Profile Raistmer
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 16 Jun 01
Posts: 3486
Credit: 47,504,143
RAC: 35,717
Russia
Message 877763 - Posted: 21 Mar 2009, 6:00:33 UTC - in response to Message 877279.

Thank you very much for so interesting and illustrative post.
If you could continue the story later it would be wonderful!

Profile Dr. C.E.T.I.
Avatar
Send message
Joined: 29 Feb 00
Posts: 15993
Credit: 690,597
RAC: 0
United States
Message 878204 - Posted: 22 Mar 2009, 4:47:11 UTC






. . . Thanks for the Post Eric and, as well the Beautiful Artworks Sir!

> note: 'reminds' me of when another Scientist 'used-to' Post 'ere [reference to Kevin Douglas PhD w/ GALFA . . .]


____________
BOINC Wiki . . .

Science Status Page . . .

Profile Bruno Moretti IK2WQA
Avatar
Send message
Joined: 15 May 99
Posts: 284
Credit: 49,167
RAC: 0
Italy
Message 878289 - Posted: 22 Mar 2009, 15:14:06 UTC
Last modified: 22 Mar 2009, 15:14:37 UTC

Very interesting post, Eric.
Thanks you.

73 & clear skies from Bruno IK2WQA - brmoret_at_libero.it
Messier 13 Astronomical Observatory, Tradate (Italy)
Founder SETI@home Berkeley's Staff Friends Club
Founder SETI ITALIA Team G. Cocconi

Profile AngelaProject donor
Volunteer tester
Avatar
Send message
Joined: 16 Oct 07
Posts: 9098
Credit: 3,520,169
RAC: 6,425
United States
Message 878794 - Posted: 24 Mar 2009, 5:42:54 UTC

Oh well, nobody's perfect (except Angela, of course).


"True-er" words were never said...

Profile Virtual Boss*
Volunteer tester
Avatar
Send message
Joined: 4 May 08
Posts: 417
Credit: 6,197,469
RAC: 385
Australia
Message 878809 - Posted: 24 Mar 2009, 7:21:44 UTC - in response to Message 878802.
Last modified: 24 Mar 2009, 7:27:47 UTC

I'd say that kenzieB's comment of "bias it against signal strength. That is, any
signal above x dB has to be terrestrial and thus, can be ignored" was
a rather astute observation. Slightly modified version of "throw out any
workunit where Astropulse says it detects multiple pulses that saturate the amplifiers".



If only it was that simple......

From my radio transmitter experience and the small amount of knowledge I have of Aricebo's setup, I offer the following scenario:-

I woul guess that the interefering signal, although very strong, is probably ousite of the frequency band of interest and hence cannot be directly detected by Seti in the data. Instead what would be present in the data would be intermodulation products (signals) caused be the Radar (or other) signals saturating (overloading the inputs of) the amplifiers. These products could appear anywhere within the search frequency band and at any amplitude from below noise level to max recordable values, and would vary with any change in frequency or amplitude of the interfering signal.

I hope this gives an insight into the massive headache this interference causes for Seti.

Bruce
EDIT: Oh.. I forgot to mention there could be hundreds, or even thousands of intermod products generated simultaneously.

Profile Virtual Boss*
Volunteer tester
Avatar
Send message
Joined: 4 May 08
Posts: 417
Credit: 6,197,469
RAC: 385
Australia
Message 878818 - Posted: 24 Mar 2009, 9:50:50 UTC - in response to Message 878814.

Chill out Boss!


But I am chilled out. :)

My post was only to help those who have little or no understanding of the problem gain a little insight into how complex it really is, and how difficult to eliminate.


virtual = not real


Ouch ... I've been exposed. :(

Profile Dr. C.E.T.I.
Avatar
Send message
Joined: 29 Feb 00
Posts: 15993
Credit: 690,597
RAC: 0
United States
Message 878824 - Posted: 24 Mar 2009, 11:10:05 UTC - in response to Message 878794.




Oh well, nobody's perfect (except Angela, of course).


"True-er" words were never said...



. . . so, what you were both sayin' [changes name BAcK to nobody] ;)))

. . . watChes as Eric steps in and 'clarifies' and then 're-clarifies' everything



'. . . there's light @ the end of this tunnel' the Berkeley Feed exclaims


:>]
____________
BOINC Wiki . . .

Science Status Page . . .

Profile vistaga
Send message
Joined: 20 Sep 04
Posts: 7
Credit: 846,906
RAC: 0
United States
Message 879046 - Posted: 25 Mar 2009, 8:43:14 UTC

i have a question about seti mybee it is dumb but, one of the things you always need is hard drives. why does seti not use peoples spare hd space the same as they use spare prossecor time? or do they? there is alot of spare hd space. i have two 20 gig drives in my machine i do not even use. i know 20gig is a drop in the bucket but 20 gig times all your users?
____________

Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 8489
Credit: 4,188,631
RAC: 1,733
United Kingdom
Message 879124 - Posted: 25 Mar 2009, 16:47:36 UTC - in response to Message 879046.

i have a question about seti mybee it is dumb but, one of the things you always need is hard drives. why does seti not use peoples spare hd space the same as they use spare prossecor time? or do they? there is alot of spare hd space. i have two 20 gig drives in my machine i do not even use. i know 20gig is a drop in the bucket but 20 gig times all your users?

That is an application of Boinc that has been talked about a few times.

I guess it is rather low priority compared to other development focus.

Then again... If anyone can put some code together... With checks/security for data integrity and so on...

Welcome to the forums. Try a look in on "Number Crunching".

Keep searchin,
Martin


____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

Profile Sorceress
Volunteer tester
Avatar
Send message
Joined: 18 Nov 06
Posts: 6
Credit: 84,813
RAC: 0
United States
Message 879186 - Posted: 25 Mar 2009, 20:33:42 UTC - in response to Message 879124.
Last modified: 25 Mar 2009, 20:34:29 UTC

i have a question about seti mybee it is dumb but, one of the things you always need is hard drives. why does seti not use peoples spare hd space the same as they use spare prossecor time? or do they? there is alot of spare hd space. i have two 20 gig drives in my machine i do not even use. i know 20gig is a drop in the bucket but 20 gig times all your users?


I personally, would NEVER allow ANYONE access to my hard drives for ANY reason. You would be seriously asking for trouble. What a ridiculous idea! It's obvious why it would be a 'low priority'.
____________




Profile Dr. C.E.T.I.
Avatar
Send message
Joined: 29 Feb 00
Posts: 15993
Credit: 690,597
RAC: 0
United States
Message 879189 - Posted: 25 Mar 2009, 20:39:41 UTC - in response to Message 879186.


. . . Welcome to Posting Sorceress - How's Rico doin' btw

i have a question about seti mybee it is dumb but, one of the things you always need is hard drives. why does seti not use peoples spare hd space the same as they use spare prossecor time? or do they? there is alot of spare hd space. i have two 20 gig drives in my machine i do not even use. i know 20gig is a drop in the bucket but 20 gig times all your users?


I personally, would NEVER allow ANYONE access to my hard drives for ANY reason. You would be seriously asking for trouble. What a ridiculous idea! It's obvious why it would be a 'low priority'.


____________
BOINC Wiki . . .

Science Status Page . . .

Profile RandyC
Avatar
Send message
Joined: 20 Oct 99
Posts: 714
Credit: 1,704,345
RAC: 0
United States
Message 879354 - Posted: 26 Mar 2009, 9:42:24 UTC - in response to Message 879046.

i have a question about seti mybee it is dumb but, one of the things you always need is hard drives. why does seti not use peoples spare hd space the same as they use spare prossecor time? or do they? there is alot of spare hd space. i have two 20 gig drives in my machine i do not even use. i know 20gig is a drop in the bucket but 20 gig times all your users?


Just to throw my 2 cents in with the other replies...it would take a significant amount of bandwidth to transfer the files to users' systems plus the possibility of data loss (user quits, system dies, etc.) makes it just not worth the attempt.

Profile vistaga
Send message
Joined: 20 Sep 04
Posts: 7
Credit: 846,906
RAC: 0
United States
Message 879361 - Posted: 26 Mar 2009, 10:46:03 UTC - in response to Message 879124.

You already are alowing them acsess to your hard drive to run there software. what do you think you download and upload. do you think that just goes into thin air? think b4 you talk
____________

Profile vistaga
Send message
Joined: 20 Sep 04
Posts: 7
Credit: 846,906
RAC: 0
United States
Message 879362 - Posted: 26 Mar 2009, 10:50:21 UTC - in response to Message 879189.

you could use double or triple redundant saves. and require a connection of atleast 5mps conection speed
____________

Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 8489
Credit: 4,188,631
RAC: 1,733
United Kingdom
Message 879387 - Posted: 26 Mar 2009, 14:20:39 UTC - in response to Message 879186.

I personally, would NEVER allow ANYONE access to my hard drives for ANY reason. You would be seriously asking for trouble. What a ridiculous idea! ...

Do you run on a Windows OS & use proprietary software?

Ever read the "EULA"s?

(My understanding is that Microsoft users agree to Microsoft doing anything it wishes with your system and your files/data and all with utter impunity. It is well reported that Microsoft takes details of all the software on your system, and likely other details...)

Good first post!

Welcome to the forums,

Keep searchin',
Martin

____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

Josef W. SegurProject donor
Volunteer developer
Volunteer tester
Send message
Joined: 30 Oct 99
Posts: 4299
Credit: 1,068,158
RAC: 1,012
United States
Message 879441 - Posted: 26 Mar 2009, 16:04:33 UTC - in response to Message 879046.

i have a question about seti mybee it is dumb but, one of the things you always need is hard drives. why does seti not use peoples spare hd space the same as they use spare prossecor time? or do they? there is alot of spare hd space. i have two 20 gig drives in my machine i do not even use. i know 20gig is a drop in the bucket but 20 gig times all your users?

User's hard drives wouldn't be suitable for most of the project's needs simply because there are delays involved with internet transfers. It would be impossible to have the active databases distributed to users, for instance. The difference between lab computers interconnected with gigabit ethernet and what can be done over the internet is huge.

As offsite storage of backup records, user's disk space could be a good idea. It would be another BOINC application, of the non compute intensive kind. Data could be delivered to hosts like a 10 MB or larger workunit, the application would simply report at intervals by trickle-up that it still had the data and be given a small amount of credit. From time to time the project might tell the host it no longer needed that data, and send some newer data to replace it. Then if the project ever needed that backup data it would be sent back and some more credit given. The same data would be given to several hosts, maybe with geographic diversity ensured by checking the host offset from GMT time. That kind of distributed offsite backup should be very robust. Still, it would be an additional load on the limited internet bandwidth the project has available, additional work for the project servers, and significant work for the staff to develop.
Joe

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 885896 - Posted: 16 Apr 2009, 20:25:33 UTC - in response to Message 879387.


Ever read the "EULA"s?

(My understanding is that Microsoft users agree to Microsoft doing anything it wishes with your system and your files/data and all with utter impunity. It is well reported that Microsoft takes details of all the software on your system, and likely other details...)

Mostly as a pre-emptive strike against any claims that Microsoft evilly and maliciously changed something on a users' system or accessed something proprietary.

Of course, once they grant themselves that right, it certainly removes one barrier to evilly and maliciously changing something on a users' system.

____________

Profile Xen
Avatar
Send message
Joined: 22 Jul 00
Posts: 86
Credit: 2,113,984
RAC: 869
United States
Message 889415 - Posted: 29 Apr 2009, 11:20:05 UTC - in response to Message 879354.

i have a question about seti mybee it is dumb but, one of the things you always need is hard drives. why does seti not use peoples spare hd space the same as they use spare prossecor time? or do they? there is alot of spare hd space. i have two 20 gig drives in my machine i do not even use. i know 20gig is a drop in the bucket but 20 gig times all your users?


Just to throw my 2 cents in with the other replies...it would take a significant amount of bandwidth to transfer the files to users' systems plus the possibility of data loss (user quits, system dies, etc.) makes it just not worth the attempt.

I'm w/Randy on this one, could cause more trouble than it's worth. Not to mention people getting upset over rac scores if one of my gerbils died and did'nt validate someone else's work.
____________
Nobody is nobody. Everyone has something to offer

Nicolas
Avatar
Send message
Joined: 30 Mar 05
Posts: 160
Credit: 10,335
RAC: 0
Argentina
Message 898512 - Posted: 23 May 2009, 2:13:49 UTC - in response to Message 879186.

I personally, would NEVER allow ANYONE access to my hard drives for ANY reason. You would be seriously asking for trouble. What a ridiculous idea! It's obvious why it would be a 'low priority'.]

You already gave "access" to your hard drive. Or where do you think SETI WUs and apps are stored while you process them? Where else other than the hard disk could that be?
____________

Contribute to the Wiki!

1 · 2 · Next

Message boards : SETI@home Staff Blog : Fun with 618,813 pulses...

Copyright © 2014 University of California