Shadow (Feb 11 2009)


log in

Advanced search

Message boards : Technical News : Shadow (Feb 11 2009)

Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 864445 - Posted: 11 Feb 2009, 23:01:41 UTC

Before releasing the astropulse application Eric had to add a couple fields to the result tables in the science database that are now necessary. These are large fields, and it's taking informix forever to update the table. The job was started 24 hours ago and is still chugging along. I guess it doesn't help that the assimilator queue is still rather large (though it is draining). So the release is delayed until this job finishes.

The radar blanking stuff I was whining about the other day has nothing to do with the astropulse release, in case there was some confusion about that. Josh and I are working on two completely separate and different forms of radar mitigation. Mine is to better clean up data before any splitting/analysis, Josh's is to deal with radar that squeaked through the first pass and made it all the way to the client. The good news is that I made significant progress on mine today.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Profile perryjay
Volunteer tester
Avatar
Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 14,947,780
RAC: 12,227
United States
Message 864457 - Posted: 11 Feb 2009, 23:29:49 UTC - in response to Message 864445.

Thanks for the update Matt. Gives the optimizing crew a little more time to get ready for the new APs. :)
____________


PROUD MEMBER OF Team Starfire World BOINC

Profile Raistmer
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 16 Jun 01
Posts: 3369
Credit: 46,113,391
RAC: 19,198
Russia
Message 864612 - Posted: 12 Feb 2009, 10:55:35 UTC - in response to Message 864445.
Last modified: 12 Feb 2009, 10:56:06 UTC

Before releasing the astropulse application Eric had to add a couple fields to the result tables in the science database that are now necessary. These are large fields, and it's taking informix forever to update the table. The job was started 24 hours ago and is still chugging along. I guess it doesn't help that the assimilator queue is still rather large (though it is draining). So the release is delayed until this job finishes.

The radar blanking stuff I was whining about the other day has nothing to do with the astropulse release, in case there was some confusion about that. Josh and I are working on two completely separate and different forms of radar mitigation. Mine is to better clean up data before any splitting/analysis, Josh's is to deal with radar that squeaked through the first pass and made it all the way to the client. The good news is that I made significant progress on mine today.

- Matt


Do you use any data substitution on server-side data cleanup (I mean, do you insert something in data array instead of removed radar pulses ? )
If yes, is it possible to mark these "blanked" areas in task header so client could know what areas are "true" ones and what contain some substituted stub data? Some list of indexes of blanked regions for example...

Profile Dr. C.E.T.I.
Avatar
Send message
Joined: 29 Feb 00
Posts: 15993
Credit: 690,597
RAC: 6
United States
Message 864615 - Posted: 12 Feb 2009, 11:02:07 UTC


. . . Congratulations Matt on the 'significant progress' - Good to hear

< Accolades to Each of You @ Berkeley - Your Hard Work is Noted . . .


____________
BOINC Wiki . . .

Science Status Page . . .

Profile speedimic
Volunteer tester
Avatar
Send message
Joined: 28 Sep 02
Posts: 362
Credit: 16,590,653
RAC: 0
Germany
Message 864634 - Posted: 12 Feb 2009, 12:52:19 UTC - in response to Message 864612.
Last modified: 12 Feb 2009, 12:52:46 UTC


Do you use any data substitution on server-side data cleanup (I mean, do you insert something in data array instead of removed radar pulses ? )
If yes, is it possible to mark these "blanked" areas in task header so client could know what areas are "true" ones and what contain some substituted stub data? Some list of indexes of blanked regions for example...


...so the client could skip those?
____________
mic.


Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 8273
Credit: 4,072,596
RAC: 347
United Kingdom
Message 864647 - Posted: 12 Feb 2009, 14:29:59 UTC - in response to Message 864612.
Last modified: 12 Feb 2009, 14:34:31 UTC

The radar blanking stuff...


Do you use any data substitution on server-side data cleanup (I mean, do you insert something in data array instead of removed radar pulses ? )...

What angle are you going for?

My scraps of understanding from previous posts are that there is a radar detector on the telescope at Arecibo. The output from that is recorded on a spare data recorder channel in parallel to the 14 streams of radio data. Matt's code then substitutes psuedo-random random noise into the data stream to replace the radar mess whenever the radar has been detected. The psuedo-random noise then goes innocently through the rest of the data processing as though Arecibo hasn't heard anything other than the usual background noise.

The radar source is a nearby military radar. The radar is kindly blanked for the arc sweeping Arecibo. However, that doesn't stop reflections that bounce off aircraft (or other local structures?) splattering Arecibo...

Perhaps:

Also set aside an air exclusion zone around Arecibo?

Change the radar frequency to something much higher than the Arecibo receivers? (Or will the radar ERP saturate anything more sensitive than an AC power rectifier diode?!)


An interesting question is whether Matt has had to add any cleverness to avoid a signal spike being detected by the FFT for the transition to/from the psuedo-random noise section...? I would expect the FFT to "see" the transition in the noise "texture".


Happy crunchin',
Martin
____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

PhonAcq
Send message
Joined: 14 Apr 01
Posts: 1622
Credit: 21,978,897
RAC: 4,052
United States
Message 864672 - Posted: 12 Feb 2009, 16:18:38 UTC

Can anyone comment on the mathematical correctness of adding pseudorandom noise?

It seems to me that doing so would add pseudo-energy to the broadband power spectrum. This has the effect of reducing S/N a bit. For example, if 50% of the time slice had radar noise and it was replaced with random noise, then 50% of the power spectrum, across frequencies, would be artificial. That just doesn't sound right.

Wouldn't just replacing the time with a constant, or if processed after the fact replace the radar blank period with the signal average for the time slice, then this would only add noise at the DC level and then would be ignored.

I'm sure Matt and others have thought about this, so I'm just looking to understand the analysis a bit.

Profile Kurt R. Radtke
Avatar
Send message
Joined: 30 Mar 00
Posts: 4
Credit: 51,481
RAC: 0
United States
Message 864753 - Posted: 12 Feb 2009, 19:46:42 UTC

Matt,

Thanks for the postings here. It's nice to know there is a human or two somewhere in the mix watching over, updating and maintaining things, and it's also nice to have someone crack the door so we can peek in.

Thank you.
____________
Kurt R. Radtke
SETI@Home; Founder of Team Macintosh

Profile Paul D. Buck
Volunteer tester
Send message
Joined: 19 Jul 00
Posts: 3898
Credit: 1,158,042
RAC: 0
United States
Message 864844 - Posted: 13 Feb 2009, 0:28:24 UTC - in response to Message 864647.

Change the radar frequency to something much higher than the Arecibo receivers? (Or will the radar ERP saturate anything more sensitive than an AC power rectifier diode?!)


If it is a ground based radar it likely will saturate even the power diode.

Heck, when I worked on OTH-B ... when we radiated on a frequency no one in the world could use that slot ... the good news is that it was low frequency so most people could not care less ... but it was funny working with the Radar Control guys ... we would suggest a frequency band and they would find a clear spot and then "park" the radar there with short scans before changing the whole system over ... they kinda played nice by reserving the space before stepping on anyone that had had an idea of coming up on that spot ...

Profile Raistmer
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 16 Jun 01
Posts: 3369
Credit: 46,113,391
RAC: 19,198
Russia
Message 864996 - Posted: 13 Feb 2009, 11:27:46 UTC - in response to Message 864844.

Well, short question is:
Is it possible to include info what areas of data was changed by blanking algorithm into WU header ?

PhonAcq
Send message
Joined: 14 Apr 01
Posts: 1622
Credit: 21,978,897
RAC: 4,052
United States
Message 865017 - Posted: 13 Feb 2009, 13:25:00 UTC - in response to Message 864996.

I think they archive all raw data. If so the software blanking could in principle be modified in the future and the data re-processed.

Profile Raistmer
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 16 Jun 01
Posts: 3369
Credit: 46,113,391
RAC: 19,198
Russia
Message 865088 - Posted: 13 Feb 2009, 17:04:45 UTC - in response to Message 865017.

I think they archive all raw data. If so the software blanking could in principle be modified in the future and the data re-processed.

Too much unneeded work to rely on this archived datas. In general, if there is some way not to do smth, better use it as soon as possible ;)

Message boards : Technical News : Shadow (Feb 11 2009)

Copyright © 2014 University of California