Fallen Leaf (Feb 12 2009)

Looks like "Astropulse V5" was finally released yesterday night. As far as I know so far, so good - work is going out, results are being validated. However, it seems like jocelyn (the master mysql database server) had a long period of mysterious pain over night, and recovered on its own this morning. This happens from time to time on our mysql servers, perhaps due to its own nebulous data scrubbing, or perhaps due to lack of memory which is becoming more a problem as the database continues to grow and less of it fits in RAM. Unless anybody out there has a couple Sun-qualified 2GB DIMMs that work in Sun v40z's kicking around, we're going to purchase a few. Currently the system has 28GB of RAM - 12 slots with 2GB DIMMs, the remaining 4 with 1GB. We hope to at least upgrade those four to 2GB. It is unclear whether or not our version of the v40z can take 4GB DIMMs (and go over 32GB total).

As for radar blanking, let me clear up the general picture.

Now that we are using the ALFA receiver (since 2006) we are susceptible to military radar, which causes many overflows in our SETI@home/astropulse analysis. The transmitter is aimed right at us approximately every 12 seconds, and then echoes bounce all over the mountains surrounding the telescope the rest of the time. Even the echoes cause us to overflow. The radar is fairly unpredictable - the military isn't very forthcoming about their transmission patterns, and when they are going to change to another pattern. Nevertheless, it is predictable enough: there are about 6 known "patterns" us civilians can lock on to.

Luckily, Arecibo solved this problem for us. They have a hardware device that broadcasts a bit letting all projects at the observatory know when it thinks the radar is on (1 for on, 0 for off). This we call the "hardware blanker" - and we inject this bit into an unused channel in our raw data. This has been quite helpful: when the bit is "1" we'd randomize the data, thus squashing the overflows. At least in theory - there were still three problems.

Problem 1: We only got the hardware blanker working sometime in 2007, so there is no such blanking information in the previous years' worth of data, thus rendering it fairly useless.

Problem 2: The hardware blanker sometimes isn't on like it should be, or even worse is mis-locked onto a wrong pattern and going out of phase with the actual radar, which also renders data quite useless.

Here's where my code comes in: The "software radar blanker." Actually, this is code/logic written by a summer student, Luke, and then I cleaned it up and (apparently so far) got it working. In short, the software radar blanker does a statistical analysis of the raw data - basically looking to see when we're blasted by radar and then trying to lock on to known patterns, and extrapolate from there. Luckily there's another free bit available in the raw data, so the ultimate plan is for raw data to come up here, go through the software radar blanker, and then process. The splitter will use the software and hardware radar blanker bits (exactly how is still up for discussion) to randomize the data. This brings us to...

Problem 3: The randomization shouldn't be totally random. Initially we were injecting white noise into the data when we were blanking. Turns out this causes edge effects and other artifacts during the client analysis. This noise was eventually shaped to fall in line with noise we'd expect to see from a quiet Arecibo. The exact mathematical details of this are left to others who aren't me. I was out of this loop.

All the above was taking too long, so Josh actually implemented code in the astropulse client to reduce some of these radar problems until they are completely solved. He isn't radar "blanking" (which happens during workunit creation) as much as having the clients find stuff that is probably radar and treating it accordingly. For what it's worth, one of the CASPER guys, Andrew, has been having the same exact military radar problems with the pulsar data they've been collecting at the ATA, so he's been simultaneously working on his own radar mitigation techniques. Man, the earth is noisy.

In any case, I figure it'll be about a month of testing/tweaking before we're actually using the software blanker.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Ok, I see.
I asked this in your prev post thread but rephrase question here:

Is it possible to change splitter that way ?:
When it encounter hardware or software blanking bit ON and does some data substitution (no matter will it be white noise or smth more clever) it put info about changed region of data into WU header too.

Why it's worth to do:

It will retain possibility of some different processing for these parts by alternative client while will give no difficulties to standart client.
So, if some way to treat altered areas more effective will be found alternative client could use info about altered regions for its work. And by default altered areas can be treated as other data regions (as now).

Sun hardware site says v40z can be populated to 64 GB

at link
http://www.sun.com/servers/entry/v40z/specs.xml#anchor2

It is unclear whether or not our version of the v40z can take 4GB DIMMs (and go over 32GB total).

When it encounter hardware or software blanking bit ON and does some data substitution (no matter will it be white noise or smth more clever) it put info about changed region of data into WU header too.

When it encounter hardware or software blanking bit ON and does some data substitution (no matter will it be white noise or smth more clever) it put info about changed region of data into WU header too.

Good idea, but I can think of one reason *not* to do this. Because of the nature of the radar, the bounces, the echoes, the periodicity, etc. we are toggling the blanking bit every 6000 samples, roughly. That's once every .002 seconds, and it's very random in nature. So to accurately depict the radar blanking info within the headers of a ~107 second workunit, we need to inflate each workunit by 200K, which will increase our bandwidth by 33%. These are rough numbers but you get the idea.

- Matt

I keep getting "project has no jobs available" when requesting work and then a http error. Server status is showing 102,953 results ready to send.

Seems something is amiss somewhere, will keep being patient.
____________

See entry on home page. Server problem.
____________

Again me being too fast at typing, I saw that after I hit send.

Thank you though for replying.
____________

For S@H Enhanced the 1024*1024 two-bit samples take 256K, sending one-bit blanking reduced to the same sample rate and similarly packed would take 128K. However, the highest time resolution used for analysis is 1/8 the sample rate so the blanking info could be further reduced by a factor of 8 and only need 16K packed. Rewriting the application to make use of that info wouldn't be trivial, though.

Quick Question: Is the AstroPulse_v5 (5.03) application's work hooked up to the preferences 'Receive Work for AstroPulse' tickbox ? I don't seem to get any except for in blobs of cuda multibeam work when enabling 'Receive work for other applications'. [Incidentally, swamped with cuda work, and CPUs going cold]

Jason
____________
"It is not the strongest of the species that survives, nor the most intelligent that survives. It is the one that is the most adaptable to change."
Charles Darwin

Quick Question: Is the AstroPulse_v5 (5.03) application's work hooked up to the preferences 'Receive Work for AstroPulse' tickbox ? I don't seem to get any except for in blobs of cuda multibeam work when enabling 'Receive work for other applications'. [Incidentally, swamped with cuda work, and CPUs going cold]

Jason

Since AstroPulse v5.03 (astropulse_v5) was released as a new application, and since AP v5.03 WUs are incompatible with the older APP, anyone running an optimized application specifying AP WUs for v5 will only get work for the older AP - of which there is only reissues at this point.

In order to get new AP v5.03 work, you have to either use the stock app or wait for the official release of a AP v5.03 compatible optimized app with the appropriate app_info.xml file.
____________

...In order to get new AP v5.03 work, you have to either use the stock app or wait for the official release of a AP v5.03 compatible optimized app with the appropriate app_info.xml file.

Forgive my half-understanding here, as the nitty-gritty math details elude me (they keeps getting paged out for a zillion other details). But (a) injecting anything by the right shaped noise during radar blanking periods causes hard-to-identify "edge effects," (b) if we don't insert noise we'll be obliterated with radar in every workunit, and (c) if we simply "skip" the radar periods, we can't do any gaussian searching (which requires contiguous time domain), maybe not even pulse searching given the nature of fast folding. I cannot stress how much data is being blitzed by radar - we're randomizing about 10-15% of the data, once every 0.002 seconds is a blanking period lasting about 0.00025 seconds. This all said, other people around here could give you a clearer idea/argument why we're using whatever methods we're using to deal with radar - I'm just working on finding the stuff and telling everybody where it is - the hard part being that radar quite often doesn't pass any obvious threshold, so I have to lock on to known patterns when they do pass threshold and then interpolate.

- Matt

In addition to the downloading problem, there now seems to be a problem uploading completed WU's to Berkeley... getting:

2/16/2009 8:57:52 AM|SETI@home|Temporarily failed upload of 16ja09af.28021.72.11.8.100_1_0: system connect

When I try...

[add]
also give Beta a kick, as a similar problem has been over there for a day or two...
[/add]

[2nd add]
re-started working at ~0920 Berkeley time...
[/2nd add]
____________
.

In addition to the downloading problem, there now seems to be a problem uploading completed WU's to Berkeley... getting:

2/16/2009 8:57:52 AM|SETI@home|Temporarily failed upload of 16ja09af.28021.72.11.8.100_1_0: system connect

When I try...

Someone must have kicked something as the uploads are all working fine on this end.

Haven't had a problem since 9am Berkeley time...then started working again soon after.
____________