Shadow (Feb 11 2009)

Message boards : Technical News : Shadow (Feb 11 2009)
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 864445 - Posted: 11 Feb 2009, 23:01:41 UTC

Before releasing the astropulse application Eric had to add a couple fields to the result tables in the science database that are now necessary. These are large fields, and it's taking informix forever to update the table. The job was started 24 hours ago and is still chugging along. I guess it doesn't help that the assimilator queue is still rather large (though it is draining). So the release is delayed until this job finishes.

The radar blanking stuff I was whining about the other day has nothing to do with the astropulse release, in case there was some confusion about that. Josh and I are working on two completely separate and different forms of radar mitigation. Mine is to better clean up data before any splitting/analysis, Josh's is to deal with radar that squeaked through the first pass and made it all the way to the client. The good news is that I made significant progress on mine today.

- Matt

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 864445 · Report as offensive
Profile perryjay
Volunteer tester
Avatar

Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 20,676,751
RAC: 0
United States
Message 864457 - Posted: 11 Feb 2009, 23:29:49 UTC - in response to Message 864445.  

Thanks for the update Matt. Gives the optimizing crew a little more time to get ready for the new APs. :)


PROUD MEMBER OF Team Starfire World BOINC
ID: 864457 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 864612 - Posted: 12 Feb 2009, 10:55:35 UTC - in response to Message 864445.  
Last modified: 12 Feb 2009, 10:56:06 UTC

Before releasing the astropulse application Eric had to add a couple fields to the result tables in the science database that are now necessary. These are large fields, and it's taking informix forever to update the table. The job was started 24 hours ago and is still chugging along. I guess it doesn't help that the assimilator queue is still rather large (though it is draining). So the release is delayed until this job finishes.

The radar blanking stuff I was whining about the other day has nothing to do with the astropulse release, in case there was some confusion about that. Josh and I are working on two completely separate and different forms of radar mitigation. Mine is to better clean up data before any splitting/analysis, Josh's is to deal with radar that squeaked through the first pass and made it all the way to the client. The good news is that I made significant progress on mine today.

- Matt


Do you use any data substitution on server-side data cleanup (I mean, do you insert something in data array instead of removed radar pulses ? )
If yes, is it possible to mark these "blanked" areas in task header so client could know what areas are "true" ones and what contain some substituted stub data? Some list of indexes of blanked regions for example...
ID: 864612 · Report as offensive
Profile Dr. C.E.T.I.
Avatar

Send message
Joined: 29 Feb 00
Posts: 16019
Credit: 794,685
RAC: 0
United States
Message 864615 - Posted: 12 Feb 2009, 11:02:07 UTC


. . . Congratulations Matt on the 'significant progress' - Good to hear

< Accolades to Each of You @ Berkeley - Your Hard Work is Noted . . .


BOINC Wiki . . .

Science Status Page . . .
ID: 864615 · Report as offensive
Profile speedimic
Volunteer tester
Avatar

Send message
Joined: 28 Sep 02
Posts: 362
Credit: 16,590,653
RAC: 0
Germany
Message 864634 - Posted: 12 Feb 2009, 12:52:19 UTC - in response to Message 864612.  
Last modified: 12 Feb 2009, 12:52:46 UTC


Do you use any data substitution on server-side data cleanup (I mean, do you insert something in data array instead of removed radar pulses ? )
If yes, is it possible to mark these "blanked" areas in task header so client could know what areas are "true" ones and what contain some substituted stub data? Some list of indexes of blanked regions for example...


...so the client could skip those?
mic.


ID: 864634 · Report as offensive
Profile ML1
Volunteer moderator
Volunteer tester

Send message
Joined: 25 Nov 01
Posts: 20084
Credit: 7,508,002
RAC: 20
United Kingdom
Message 864647 - Posted: 12 Feb 2009, 14:29:59 UTC - in response to Message 864612.  
Last modified: 12 Feb 2009, 14:34:31 UTC

The radar blanking stuff...


Do you use any data substitution on server-side data cleanup (I mean, do you insert something in data array instead of removed radar pulses ? )...

What angle are you going for?

My scraps of understanding from previous posts are that there is a radar detector on the telescope at Arecibo. The output from that is recorded on a spare data recorder channel in parallel to the 14 streams of radio data. Matt's code then substitutes psuedo-random random noise into the data stream to replace the radar mess whenever the radar has been detected. The psuedo-random noise then goes innocently through the rest of the data processing as though Arecibo hasn't heard anything other than the usual background noise.

The radar source is a nearby military radar. The radar is kindly blanked for the arc sweeping Arecibo. However, that doesn't stop reflections that bounce off aircraft (or other local structures?) splattering Arecibo...

Perhaps:

Also set aside an air exclusion zone around Arecibo?

Change the radar frequency to something much higher than the Arecibo receivers? (Or will the radar ERP saturate anything more sensitive than an AC power rectifier diode?!)


An interesting question is whether Matt has had to add any cleverness to avoid a signal spike being detected by the FFT for the transition to/from the psuedo-random noise section...? I would expect the FFT to "see" the transition in the noise "texture".


Happy crunchin',
Martin
See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)
ID: 864647 · Report as offensive
PhonAcq

Send message
Joined: 14 Apr 01
Posts: 1656
Credit: 30,658,217
RAC: 1
United States
Message 864672 - Posted: 12 Feb 2009, 16:18:38 UTC

Can anyone comment on the mathematical correctness of adding pseudorandom noise?

It seems to me that doing so would add pseudo-energy to the broadband power spectrum. This has the effect of reducing S/N a bit. For example, if 50% of the time slice had radar noise and it was replaced with random noise, then 50% of the power spectrum, across frequencies, would be artificial. That just doesn't sound right.

Wouldn't just replacing the time with a constant, or if processed after the fact replace the radar blank period with the signal average for the time slice, then this would only add noise at the DC level and then would be ignored.

I'm sure Matt and others have thought about this, so I'm just looking to understand the analysis a bit.
ID: 864672 · Report as offensive
Profile Kurt R. Radtke
Avatar

Send message
Joined: 30 Mar 00
Posts: 4
Credit: 51,481
RAC: 0
United States
Message 864753 - Posted: 12 Feb 2009, 19:46:42 UTC

Matt,

Thanks for the postings here. It's nice to know there is a human or two somewhere in the mix watching over, updating and maintaining things, and it's also nice to have someone crack the door so we can peek in.

Thank you.
Kurt R. Radtke
SETI@Home; Founder of Team Macintosh
ID: 864753 · Report as offensive
Profile Paul D. Buck
Volunteer tester

Send message
Joined: 19 Jul 00
Posts: 3898
Credit: 1,158,042
RAC: 0
United States
Message 864844 - Posted: 13 Feb 2009, 0:28:24 UTC - in response to Message 864647.  

Change the radar frequency to something much higher than the Arecibo receivers? (Or will the radar ERP saturate anything more sensitive than an AC power rectifier diode?!)


If it is a ground based radar it likely will saturate even the power diode.

Heck, when I worked on OTH-B ... when we radiated on a frequency no one in the world could use that slot ... the good news is that it was low frequency so most people could not care less ... but it was funny working with the Radar Control guys ... we would suggest a frequency band and they would find a clear spot and then "park" the radar there with short scans before changing the whole system over ... they kinda played nice by reserving the space before stepping on anyone that had had an idea of coming up on that spot ...
ID: 864844 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 864996 - Posted: 13 Feb 2009, 11:27:46 UTC - in response to Message 864844.  

Well, short question is:
Is it possible to include info what areas of data was changed by blanking algorithm into WU header ?
ID: 864996 · Report as offensive
PhonAcq

Send message
Joined: 14 Apr 01
Posts: 1656
Credit: 30,658,217
RAC: 1
United States
Message 865017 - Posted: 13 Feb 2009, 13:25:00 UTC - in response to Message 864996.  

I think they archive all raw data. If so the software blanking could in principle be modified in the future and the data re-processed.
ID: 865017 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 865088 - Posted: 13 Feb 2009, 17:04:45 UTC - in response to Message 865017.  

I think they archive all raw data. If so the software blanking could in principle be modified in the future and the data re-processed.

Too much unneeded work to rely on this archived datas. In general, if there is some way not to do smth, better use it as soon as possible ;)
ID: 865088 · Report as offensive

Message boards : Technical News : Shadow (Feb 11 2009)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.