Looks like "Astropulse V5" was finally released yesterday night. As far as I know so far, so good - work is going out, results are being validated. However, it seems like jocelyn (the master mysql database server) had a long period of mysterious pain over night, and recovered on its own this morning. This happens from time to time on our mysql servers, perhaps due to its own nebulous data scrubbing, or perhaps due to lack of memory which is becoming more a problem as the database continues to grow and less of it fits in RAM. Unless anybody out there has a couple Sun-qualified 2GB DIMMs that work in Sun v40z's kicking around, we're going to purchase a few. Currently the system has 28GB of RAM - 12 slots with 2GB DIMMs, the remaining 4 with 1GB. We hope to at least upgrade those four to 2GB. It is unclear whether or not our version of the v40z can take 4GB DIMMs (and go over 32GB total).

As for radar blanking, let me clear up the general picture.

Now that we are using the ALFA receiver (since 2006) we are susceptible to military radar, which causes many overflows in our SETI@home/astropulse analysis. The transmitter is aimed right at us approximately every 12 seconds, and then echoes bounce all over the mountains surrounding the telescope the rest of the time. Even the echoes cause us to overflow. The radar is fairly unpredictable - the military isn't very forthcoming about their transmission patterns, and when they are going to change to another pattern. Nevertheless, it is predictable enough: there are about 6 known "patterns" us civilians can lock on to.

Luckily, Arecibo solved this problem for us. They have a hardware device that broadcasts a bit letting all projects at the observatory know when it thinks the radar is on (1 for on, 0 for off). This we call the "hardware blanker" - and we inject this bit into an unused channel in our raw data. This has been quite helpful: when the bit is "1" we'd randomize the data, thus squashing the overflows. At least in theory - there were still three problems.

Problem 1: We only got the hardware blanker working sometime in 2007, so there is no such blanking information in the previous years' worth of data, thus rendering it fairly useless.

Problem 2: The hardware blanker sometimes isn't on like it should be, or even worse is mis-locked onto a wrong pattern and going out of phase with the actual radar, which also renders data quite useless.

Here's where my code comes in: The "software radar blanker." Actually, this is code/logic written by a summer student, Luke, and then I cleaned it up and (apparently so far) got it working. In short, the software radar blanker does a statistical analysis of the raw data - basically looking to see when we're blasted by radar and then trying to lock on to known patterns, and extrapolate from there. Luckily there's another free bit available in the raw data, so the ultimate plan is for raw data to come up here, go through the software radar blanker, and then process. The splitter will use the software and hardware radar blanker bits (exactly how is still up for discussion) to randomize the data. This brings us to...

Problem 3: The randomization shouldn't be totally random. Initially we were injecting white noise into the data when we were blanking. Turns out this causes edge effects and other artifacts during the client analysis. This noise was eventually shaped to fall in line with noise we'd expect to see from a quiet Arecibo. The exact mathematical details of this are left to others who aren't me. I was out of this loop.

All the above was taking too long, so Josh actually implemented code in the astropulse client to reduce some of these radar problems until they are completely solved. He isn't radar "blanking" (which happens during workunit creation) as much as having the clients find stuff that is probably radar and treating it accordingly. For what it's worth, one of the CASPER guys, Andrew, has been having the same exact military radar problems with the pulsar data they've been collecting at the ATA, so he's been simultaneously working on his own radar mitigation techniques. Man, the earth is noisy.

In any case, I figure it'll be about a month of testing/tweaking before we're actually using the software blanker.

- Matt

---

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Ok, I see.
I asked this in your prev post thread but rephrase question here:

Is it possible to change splitter that way ?:
When it encounter hardware or software blanking bit ON and does some data substitution (no matter will it be white noise or smth more clever) it put info about changed region of data into WU header too.

Why it's worth to do:

It will retain possibility of some different processing for these parts by alternative client while will give no difficulties to standart client.
So, if some way to treat altered areas more effective will be found alternative client could use info about altered regions for its work. And by default altered areas can be treated as other data regions (as now).

Hey,

what kind of RAM is now in your V40z.

Sun / X9251A / 1GB (2 x 512MB DDR333 DIMM) -- Sun Parts
Sun / X9252A / 2GB (2 x 1GB DDR333 DIMM) -- Sun Parts
Sun / X9266A / 4GB (2 x 2GB DDR333 DIMM) -- Sun Parts
Sun / X9295A / 540-6427 / 1GB (2 x 512MB DDR400 DIMM) -- Sun Parts
Sun / X9296A / 540-6428 / 2GB (2 x 1GB DDR400 DIMM) -- Sun Parts
Sun / X9297A / 540-6429 / 4GB (2 x 2GB DDR400 DIMM) -- Sun Parts

I want try to get OEM or ThirdManufacture RAM if not to expensiv.

Your

Gunter

---

Sun hardware site says v40z can be populated to 64 GB

at link
http://www.sun.com/servers/entry/v40z/specs.xml#anchor2

---

Sun hardware site says v40z can be populated to 64 GB

at link
http://www.sun.com/servers/entry/v40z/specs.xml#anchor2

As Matt stated:

It is unclear whether or not our version of the v40z can take 4GB DIMMs (and go over 32GB total).

There are different versions of the V40z server, some that come with an additional RAM card, or a newer chipset that supports higher density RAM. This Kingston link seems to explain some of the differences. Note that Kingston is only selling a 4GB pair (2x2GB modules) @ $574.

---

When it encounter hardware or software blanking bit ON and does some data substitution (no matter will it be white noise or smth more clever) it put info about changed region of data into WU header too.

Good idea, but I can think of one reason *not* to do this. Because of the nature of the radar, the bounces, the echoes, the periodicity, etc. we are toggling the blanking bit every 6000 samples, roughly. That's once every .002 seconds, and it's very random in nature. So to accurately depict the radar blanking info within the headers of a ~107 second workunit, we need to inflate each workunit by 200K, which will increase our bandwidth by 33%. These are rough numbers but you get the idea.

- Matt

---

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

. . . Thanks for the News Matt - All of You @ Berkeley are sent Accolades, with Respect

---

BOINC Wiki . . .

Science Status Page . . .

When it encounter hardware or software blanking bit ON and does some data substitution (no matter will it be white noise or smth more clever) it put info about changed region of data into WU header too.

Good idea, but I can think of one reason *not* to do this. Because of the nature of the radar, the bounces, the echoes, the periodicity, etc. we are toggling the blanking bit every 6000 samples, roughly. That's once every .002 seconds, and it's very random in nature. So to accurately depict the radar blanking info within the headers of a ~107 second workunit, we need to inflate each workunit by 200K, which will increase our bandwidth by 33%. These are rough numbers but you get the idea.

- Matt

For S@H Enhanced the 1024*1024 two-bit samples take 256K, sending one-bit blanking reduced to the same sample rate and similarly packed would take 128K. However, the highest time resolution used for analysis is 1/8 the sample rate so the blanking info could be further reduced by a factor of 8 and only need 16K packed. Rewriting the application to make use of that info wouldn't be trivial, though.

For Astropulse, the internal blanking code works on 32K sample data chunks, because of 50% overlap there are 2047 of those in an AP WU. As few as 256 bytes of additional data could represent the external blanking. In that form it would be easy to merge with the internal blanking.

In both cases, the reduced blanking data would need a threshold of some sort. For instance, if three 6000 sample parts of a 32K Astropulse data chunk had been replaced it might or might not be worth looking for signals in that data.

Just musing here. I agree fully with Raistmer that it's better to analyze the real data thoroughly and skip as much of the waste processing of inserted random data as possible.

                                                              Joe

I keep getting "project has no jobs available" when requesting work and then a http error. Server status is showing 102,953 results ready to send.

Seems something is amiss somewhere, will keep being patient.

---

See entry on home page. Server problem.

---

Again me being too fast at typing, I saw that after I hit send.

Thank you though for replying.

---

For S@H Enhanced the 1024*1024 two-bit samples take 256K, sending one-bit blanking reduced to the same sample rate and similarly packed would take 128K. However, the highest time resolution used for analysis is 1/8 the sample rate so the blanking info could be further reduced by a factor of 8 and only need 16K packed. Rewriting the application to make use of that info wouldn't be trivial, though.

Right.. This was a quick calculation and I messed up byte/bit conversion so I was off by a factor of 8. But that was just one reason - you pointed out another - the rewriting of the application (and the splitter for that matter).

Forgive my half-understanding here, as the nitty-gritty math details elude me (they keeps getting paged out for a zillion other details). But (a) injecting anything by the right shaped noise during radar blanking periods causes hard-to-identify "edge effects," (b) if we don't insert noise we'll be obliterated with radar in every workunit, and (c) if we simply "skip" the radar periods, we can't do any gaussian searching (which requires contiguous time domain), maybe not even pulse searching given the nature of fast folding. I cannot stress how much data is being blitzed by radar - we're randomizing about 10-15% of the data, once every 0.002 seconds is a blanking period lasting about 0.00025 seconds. This all said, other people around here could give you a clearer idea/argument why we're using whatever methods we're using to deal with radar - I'm just working on finding the stuff and telling everybody where it is - the hard part being that radar quite often doesn't pass any obvious threshold, so I have to lock on to known patterns when they do pass threshold and then interpolate.

- Matt

---

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Quick Question: Is the AstroPulse_v5 (5.03) application's work hooked up to the preferences 'Receive Work for AstroPulse' tickbox ? I don't seem to get any except for in blobs of cuda multibeam work when enabling 'Receive work for other applications'. [Incidentally, swamped with cuda work, and CPUs going cold]

Jason

---

"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.

Quick Question: Is the AstroPulse_v5 (5.03) application's work hooked up to the preferences 'Receive Work for AstroPulse' tickbox ? I don't seem to get any except for in blobs of cuda multibeam work when enabling 'Receive work for other applications'. [Incidentally, swamped with cuda work, and CPUs going cold]

Jason

I've been wondering that, I've only got Astropulse selected, I have a three way app_info for setiathome_enhanced, astropulse and astropulse_v5,
but the only astropulse i have received are AP 5.00 resends.

Claggy

Since AstroPulse v5.03 (astropulse_v5) was released as a new application, and since AP v5.03 WUs are incompatible with the older APP, anyone running an optimized application specifying AP WUs for v5 will only get work for the older AP - of which there is only reissues at this point.

In order to get new AP v5.03 work, you have to either use the stock app or wait for the official release of a AP v5.03 compatible optimized app with the appropriate app_info.xml file.

---

...In order to get new AP v5.03 work, you have to either use the stock app or wait for the official release of a AP v5.03 compatible optimized app with the appropriate app_info.xml file.

Correct, which is the one I'm trying to test (including relevant app_info). Being swamped with Cuda work and no way to specify AstroPulse only but still get v5.03 (along with v5.00 old resends), is one factor delaying the release. (Aside from waiting for validation confirmation of the ones that have been received and processed already.)

[Nevermind, pulling multibeam out of the app_info altogether for testing purposes seems to do the trick.]

Jason

---

"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.

Forgive my half-understanding here, as the nitty-gritty math details elude me (they keeps getting paged out for a zillion other details). But (a) injecting anything by the right shaped noise during radar blanking periods causes hard-to-identify "edge effects," (b) if we don't insert noise we'll be obliterated with radar in every workunit, and (c) if we simply "skip" the radar periods, we can't do any gaussian searching (which requires contiguous time domain), maybe not even pulse searching given the nature of fast folding. I cannot stress how much data is being blitzed by radar - we're randomizing about 10-15% of the data, once every 0.002 seconds is a blanking period lasting about 0.00025 seconds. This all said, other people around here could give you a clearer idea/argument why we're using whatever methods we're using to deal with radar - I'm just working on finding the stuff and telling everybody where it is - the hard part being that radar quite often doesn't pass any obvious threshold, so I have to lock on to known patterns when they do pass threshold and then interpolate.

- Matt

a) "right" shape is questionable by itself. New AP 5.03 tasks (for example) give very high overlowed exit ratio still (sampling too small still though, maybe just weird deviation).
b) noise can be inserted by server or can be inserted by client (even less work for server maybe) if client will know what areas it should blank (as doing now in AP client blanking).
c) sure we can't just skip some time areas w/o bad consequencies, but some of processing can be skipped. For example, doing single pulse search in "blanked" data meaningless from scientific point of view and wasteful from performance point of view. Maybe, another kinds of searching can be modified too for alternative accounting of damaged areas.

Yes, this requires some modifications in splitter (perhaps) code, it should write some additional fields in task header. (Manual work, yes, but any optimization is mostly manual work ;) ).
Regarding file size increase - if blanking area bigger than few time samples, compression could be used in form {index of area start,area size} so task size increase souldn't be too dramatical.

In addition to the downloading problem, there now seems to be a problem uploading completed WU's to Berkeley... getting:

2/16/2009 8:57:52 AM|SETI@home|Temporarily failed upload of 16ja09af.28021.72.11.8.100_1_0: system connect

When I try...

[add]
also give Beta a kick, as a similar problem has been over there for a day or two...
[/add]

[2nd add]
re-started working at ~0920 Berkeley time...
[/2nd add]

---

.

Hello, from Albany, CA!...

Someone must have kicked something as the uploads are all working fine on this end.

Haven't had a problem since 9am Berkeley time...then started working again soon after.

---

Good questioning but sorry, I don't see what direction you're aiming for...

a) "right" shape is questionable by itself. New AP 5.03 tasks (for example) give very high overlowed exit ratio still (sampling too small still though, maybe just weird deviation).

Noise "shape" here is likely intended to mean the spectral plot shape. The idea is to have the inserted noise look like background noise for that WU. I imagine that the transition from WU signal to inserted noise and then from inserted noise back to WU signal is going to be a very subtle trick to get it to gently blend without the FFTs 'seeing' the joins.

My own line of attack would be to use noise blanking techniques to cancel out just the radar interference itself. I don't know if that would be less or more difficult than the 'blank noise' insertion.

b) noise can be inserted by server or can be inserted by client (even less work for server maybe) if client will know what areas it should blank (as doing now in AP client blanking).

Anything more complicated for the clients could well become a nightmare for the optimisers!

c) sure we can't just skip some time areas w/o bad consequencies, but some of processing can be skipped. For example, doing single pulse search in "blanked" data meaningless from scientific point of view and wasteful from performance point of view. Maybe, another kinds of searching can be modified too for alternative accounting of damaged areas.

Perhaps just accept the data "as-is" and have the clients ignore all the pulses found in the radar contaminated bits...

Yes, this requires some modifications ... so task size increase souldn't be too dramatical.

I agree to keep all relevant data possible in with the WU for a "just in case" review further down the line. That could save a lot of reprocessing for any hiccups anywhere...


Regardless, trying to keep the data clean is a very difficult problem. Far far better would be for the radar not to be there in the first place.

Keep searchin',
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)