Shadow (Feb 11 2009)

Message boards : Technical News : Shadow (Feb 11 2009)

Author	Message
Matt Lebofsky Volunteer moderator Project administrator Project developer Project scientist Send message Joined: 1 Mar 99 Posts: 1375 Credit: 74,079 RAC: 0	Message 864445 - Posted: 11 Feb 2009, 23:01:41 UTC
	Before releasing the astropulse application Eric had to add a couple fields to the result tables in the science database that are now necessary. These are large fields, and it's taking informix forever to update the table. The job was started 24 hours ago and is still chugging along. I guess it doesn't help that the assimilator queue is still rather large (though it is draining). So the release is delayed until this job finishes. The radar blanking stuff I was whining about the other day has nothing to do with the astropulse release, in case there was some confusion about that. Josh and I are working on two completely separate and different forms of radar mitigation. Mine is to better clean up data before any splitting/analysis, Josh's is to deal with radar that squeaked through the first pass and made it all the way to the client. The good news is that I made significant progress on mine today. - Matt ____________ -- BOINC/SETI@home network/web/science/development person -- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
	ID: 864445 ·

perryjay Send message Joined: 20 Aug 02 Posts: 3359 Credit: 9,679,546 RAC: 10,526	Message 864457 - Posted: 11 Feb 2009, 23:29:49 UTC - in response to Message 864445.
	Thanks for the update Matt. Gives the optimizing crew a little more time to get ready for the new APs. :) ____________ PROUD MEMBER OF Team Starfire World BOINC
	ID: 864457 ·

Raistmer Volunteer developer Volunteer tester Send message Joined: 16 Jun 01 Posts: 2547 Credit: 25,756,633 RAC: 38,731	Message 864612 - Posted: 12 Feb 2009, 10:55:35 UTC - in response to Message 864445. Last modified: 12 Feb 2009, 10:56:06 UTC
	Before releasing the astropulse application Eric had to add a couple fields to the result tables in the science database that are now necessary. These are large fields, and it's taking informix forever to update the table. The job was started 24 hours ago and is still chugging along. I guess it doesn't help that the assimilator queue is still rather large (though it is draining). So the release is delayed until this job finishes. The radar blanking stuff I was whining about the other day has nothing to do with the astropulse release, in case there was some confusion about that. Josh and I are working on two completely separate and different forms of radar mitigation. Mine is to better clean up data before any splitting/analysis, Josh's is to deal with radar that squeaked through the first pass and made it all the way to the client. The good news is that I made significant progress on mine today. - Matt Do you use any data substitution on server-side data cleanup (I mean, do you insert something in data array instead of removed radar pulses ? ) If yes, is it possible to mark these "blanked" areas in task header so client could know what areas are "true" ones and what contain some substituted stub data? Some list of indexes of blanked regions for example...
	ID: 864612 ·

Dr. C.E.T.I. Send message Joined: 29 Feb 00 Posts: 15969 Credit: 625,669 RAC: 239	Message 864615 - Posted: 12 Feb 2009, 11:02:07 UTC
	. . . Congratulations Matt on the 'significant progress' - Good to hear < Accolades to Each of You @ Berkeley - Your Hard Work is Noted . . . ____________ BOINC Wiki . . . Science Status Page . . .
	ID: 864615 ·

speedimic Volunteer tester Send message Joined: 28 Sep 02 Posts: 362 Credit: 16,590,653 RAC: 0	Message 864634 - Posted: 12 Feb 2009, 12:52:19 UTC - in response to Message 864612. Last modified: 12 Feb 2009, 12:52:46 UTC
	Do you use any data substitution on server-side data cleanup (I mean, do you insert something in data array instead of removed radar pulses ? ) If yes, is it possible to mark these "blanked" areas in task header so client could know what areas are "true" ones and what contain some substituted stub data? Some list of indexes of blanked regions for example... ...so the client could skip those? ____________ mic.
	ID: 864634 ·

ML1 Volunteer tester Send message Joined: 25 Nov 01 Posts: 7141 Credit: 3,687,205 RAC: 958	Message 864647 - Posted: 12 Feb 2009, 14:29:59 UTC - in response to Message 864612. Last modified: 12 Feb 2009, 14:34:31 UTC
	The radar blanking stuff... Do you use any data substitution on server-side data cleanup (I mean, do you insert something in data array instead of removed radar pulses ? )... What angle are you going for? My scraps of understanding from previous posts are that there is a radar detector on the telescope at Arecibo. The output from that is recorded on a spare data recorder channel in parallel to the 14 streams of radio data. Matt's code then substitutes psuedo-random random noise into the data stream to replace the radar mess whenever the radar has been detected. The psuedo-random noise then goes innocently through the rest of the data processing as though Arecibo hasn't heard anything other than the usual background noise. The radar source is a nearby military radar. The radar is kindly blanked for the arc sweeping Arecibo. However, that doesn't stop reflections that bounce off aircraft (or other local structures?) splattering Arecibo... Perhaps: Also set aside an air exclusion zone around Arecibo? Change the radar frequency to something much higher than the Arecibo receivers? (Or will the radar ERP saturate anything more sensitive than an AC power rectifier diode?!) An interesting question is whether Matt has had to add any cleverness to avoid a signal spike being detected by the FFT for the transition to/from the psuedo-random noise section...? I would expect the FFT to "see" the transition in the noise "texture". Happy crunchin', Martin ____________ Mandriva Linux A user friendly OS! See new freedom Mageia2 The Future is what We make IT (GPLv3)
	ID: 864647 ·

PhonAcq Send message Joined: 14 Apr 01 Posts: 1620 Credit: 20,887,200 RAC: 4,798	Message 864672 - Posted: 12 Feb 2009, 16:18:38 UTC
	Can anyone comment on the mathematical correctness of adding pseudorandom noise? It seems to me that doing so would add pseudo-energy to the broadband power spectrum. This has the effect of reducing S/N a bit. For example, if 50% of the time slice had radar noise and it was replaced with random noise, then 50% of the power spectrum, across frequencies, would be artificial. That just doesn't sound right. Wouldn't just replacing the time with a constant, or if processed after the fact replace the radar blank period with the signal average for the time slice, then this would only add noise at the DC level and then would be ignored. I'm sure Matt and others have thought about this, so I'm just looking to understand the analysis a bit.
	ID: 864672 ·

Kurt R. Radtke Send message Joined: 30 Mar 00 Posts: 4 Credit: 51,481 RAC: 0	Message 864753 - Posted: 12 Feb 2009, 19:46:42 UTC
	Matt, Thanks for the postings here. It's nice to know there is a human or two somewhere in the mix watching over, updating and maintaining things, and it's also nice to have someone crack the door so we can peek in. Thank you. ____________ Kurt R. Radtke SETI@Home; Founder of Team Macintosh
	ID: 864753 ·

Paul D. Buck Volunteer tester Send message Joined: 19 Jul 00 Posts: 3898 Credit: 1,158,042 RAC: 0	Message 864844 - Posted: 13 Feb 2009, 0:28:24 UTC - in response to Message 864647.
	Change the radar frequency to something much higher than the Arecibo receivers? (Or will the radar ERP saturate anything more sensitive than an AC power rectifier diode?!) If it is a ground based radar it likely will saturate even the power diode. Heck, when I worked on OTH-B ... when we radiated on a frequency no one in the world could use that slot ... the good news is that it was low frequency so most people could not care less ... but it was funny working with the Radar Control guys ... we would suggest a frequency band and they would find a clear spot and then "park" the radar there with short scans before changing the whole system over ... they kinda played nice by reserving the space before stepping on anyone that had had an idea of coming up on that spot ...
	ID: 864844 ·

Raistmer Volunteer developer Volunteer tester Send message Joined: 16 Jun 01 Posts: 2547 Credit: 25,756,633 RAC: 38,731	Message 864996 - Posted: 13 Feb 2009, 11:27:46 UTC - in response to Message 864844.
	Well, short question is: Is it possible to include info what areas of data was changed by blanking algorithm into WU header ?
	ID: 864996 ·

PhonAcq Send message Joined: 14 Apr 01 Posts: 1620 Credit: 20,887,200 RAC: 4,798	Message 865017 - Posted: 13 Feb 2009, 13:25:00 UTC - in response to Message 864996.
	I think they archive all raw data. If so the software blanking could in principle be modified in the future and the data re-processed.
	ID: 865017 ·

Raistmer Volunteer developer Volunteer tester Send message Joined: 16 Jun 01 Posts: 2547 Credit: 25,756,633 RAC: 38,731	Message 865088 - Posted: 13 Feb 2009, 17:04:45 UTC - in response to Message 865017.
	I think they archive all raw data. If so the software blanking could in principle be modified in the future and the data re-processed. Too much unneeded work to rely on this archived datas. In general, if there is some way not to do smth, better use it as soon as possible ;)
	ID: 865088 ·

Message boards : Technical News : Shadow (Feb 11 2009)