Fallen Leaf (Feb 12 2009)

Message boards : Technical News : Fallen Leaf (Feb 12 2009)
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 864768 - Posted: 12 Feb 2009, 20:20:58 UTC

Looks like "Astropulse V5" was finally released yesterday night. As far as I know so far, so good - work is going out, results are being validated. However, it seems like jocelyn (the master mysql database server) had a long period of mysterious pain over night, and recovered on its own this morning. This happens from time to time on our mysql servers, perhaps due to its own nebulous data scrubbing, or perhaps due to lack of memory which is becoming more a problem as the database continues to grow and less of it fits in RAM. Unless anybody out there has a couple Sun-qualified 2GB DIMMs that work in Sun v40z's kicking around, we're going to purchase a few. Currently the system has 28GB of RAM - 12 slots with 2GB DIMMs, the remaining 4 with 1GB. We hope to at least upgrade those four to 2GB. It is unclear whether or not our version of the v40z can take 4GB DIMMs (and go over 32GB total).

As for radar blanking, let me clear up the general picture.

Now that we are using the ALFA receiver (since 2006) we are susceptible to military radar, which causes many overflows in our SETI@home/astropulse analysis. The transmitter is aimed right at us approximately every 12 seconds, and then echoes bounce all over the mountains surrounding the telescope the rest of the time. Even the echoes cause us to overflow. The radar is fairly unpredictable - the military isn't very forthcoming about their transmission patterns, and when they are going to change to another pattern. Nevertheless, it is predictable enough: there are about 6 known "patterns" us civilians can lock on to.

Luckily, Arecibo solved this problem for us. They have a hardware device that broadcasts a bit letting all projects at the observatory know when it thinks the radar is on (1 for on, 0 for off). This we call the "hardware blanker" - and we inject this bit into an unused channel in our raw data. This has been quite helpful: when the bit is "1" we'd randomize the data, thus squashing the overflows. At least in theory - there were still three problems.

Problem 1: We only got the hardware blanker working sometime in 2007, so there is no such blanking information in the previous years' worth of data, thus rendering it fairly useless.

Problem 2: The hardware blanker sometimes isn't on like it should be, or even worse is mis-locked onto a wrong pattern and going out of phase with the actual radar, which also renders data quite useless.

Here's where my code comes in: The "software radar blanker." Actually, this is code/logic written by a summer student, Luke, and then I cleaned it up and (apparently so far) got it working. In short, the software radar blanker does a statistical analysis of the raw data - basically looking to see when we're blasted by radar and then trying to lock on to known patterns, and extrapolate from there. Luckily there's another free bit available in the raw data, so the ultimate plan is for raw data to come up here, go through the software radar blanker, and then process. The splitter will use the software and hardware radar blanker bits (exactly how is still up for discussion) to randomize the data. This brings us to...

Problem 3: The randomization shouldn't be totally random. Initially we were injecting white noise into the data when we were blanking. Turns out this causes edge effects and other artifacts during the client analysis. This noise was eventually shaped to fall in line with noise we'd expect to see from a quiet Arecibo. The exact mathematical details of this are left to others who aren't me. I was out of this loop.

All the above was taking too long, so Josh actually implemented code in the astropulse client to reduce some of these radar problems until they are completely solved. He isn't radar "blanking" (which happens during workunit creation) as much as having the clients find stuff that is probably radar and treating it accordingly. For what it's worth, one of the CASPER guys, Andrew, has been having the same exact military radar problems with the pulsar data they've been collecting at the ATA, so he's been simultaneously working on his own radar mitigation techniques. Man, the earth is noisy.

In any case, I figure it'll be about a month of testing/tweaking before we're actually using the software blanker.

- Matt

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 864768 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 864998 - Posted: 13 Feb 2009, 11:41:42 UTC - in response to Message 864768.  
Last modified: 13 Feb 2009, 11:42:12 UTC

Ok, I see.
I asked this in your prev post thread but rephrase question here:

Is it possible to change splitter that way ?:
When it encounter hardware or software blanking bit ON and does some data substitution (no matter will it be white noise or smth more clever) it put info about changed region of data into WU header too.

Why it's worth to do:

It will retain possibility of some different processing for these parts by alternative client while will give no difficulties to standart client.
So, if some way to treat altered areas more effective will be found alternative client could use info about altered regions for its work. And by default altered areas can be treated as other data regions (as now).
ID: 864998 · Report as offensive
Profile skysearcher

Send message
Joined: 20 Mar 00
Posts: 3
Credit: 379,708
RAC: 0
Germany
Message 865135 - Posted: 13 Feb 2009, 19:00:00 UTC

Hey,

what kind of RAM is now in your V40z.

Sun / X9251A / 1GB (2 x 512MB DDR333 DIMM) -- Sun Parts
Sun / X9252A / 2GB (2 x 1GB DDR333 DIMM) -- Sun Parts
Sun / X9266A / 4GB (2 x 2GB DDR333 DIMM) -- Sun Parts
Sun / X9295A / 540-6427 / 1GB (2 x 512MB DDR400 DIMM) -- Sun Parts
Sun / X9296A / 540-6428 / 2GB (2 x 1GB DDR400 DIMM) -- Sun Parts
Sun / X9297A / 540-6429 / 4GB (2 x 2GB DDR400 DIMM) -- Sun Parts

I want try to get OEM or ThirdManufacture RAM if not to expensiv.

Your

Gunter
ID: 865135 · Report as offensive
Profile Dexter Griffin

Send message
Joined: 17 Feb 02
Posts: 1
Credit: 6,269,785
RAC: 0
United States
Message 865141 - Posted: 13 Feb 2009, 19:32:10 UTC - in response to Message 864768.  

Sun hardware site says v40z can be populated to 64 GB

at link
http://www.sun.com/servers/entry/v40z/specs.xml#anchor2

ID: 865141 · Report as offensive
OzzFan Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Apr 02
Posts: 15691
Credit: 84,761,841
RAC: 28
United States
Message 865183 - Posted: 13 Feb 2009, 22:20:00 UTC - in response to Message 865141.  
Last modified: 13 Feb 2009, 22:20:24 UTC

Sun hardware site says v40z can be populated to 64 GB

at link
http://www.sun.com/servers/entry/v40z/specs.xml#anchor2


As Matt stated:

It is unclear whether or not our version of the v40z can take 4GB DIMMs (and go over 32GB total).


There are different versions of the V40z server, some that come with an additional RAM card, or a newer chipset that supports higher density RAM. This Kingston link seems to explain some of the differences. Note that Kingston is only selling a 4GB pair (2x2GB modules) @ $574.
ID: 865183 · Report as offensive
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 865195 - Posted: 13 Feb 2009, 22:48:18 UTC - in response to Message 864998.  

When it encounter hardware or software blanking bit ON and does some data substitution (no matter will it be white noise or smth more clever) it put info about changed region of data into WU header too.


Good idea, but I can think of one reason *not* to do this. Because of the nature of the radar, the bounces, the echoes, the periodicity, etc. we are toggling the blanking bit every 6000 samples, roughly. That's once every .002 seconds, and it's very random in nature. So to accurately depict the radar blanking info within the headers of a ~107 second workunit, we need to inflate each workunit by 200K, which will increase our bandwidth by 33%. These are rough numbers but you get the idea.

- Matt
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 865195 · Report as offensive
Profile Dr. C.E.T.I.
Avatar

Send message
Joined: 29 Feb 00
Posts: 16019
Credit: 794,685
RAC: 0
United States
Message 865257 - Posted: 14 Feb 2009, 2:46:23 UTC


. . . Thanks for the News Matt - All of You @ Berkeley are sent Accolades, with Respect


BOINC Wiki . . .

Science Status Page . . .
ID: 865257 · Report as offensive
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 865290 - Posted: 14 Feb 2009, 6:26:04 UTC - in response to Message 865195.  

When it encounter hardware or software blanking bit ON and does some data substitution (no matter will it be white noise or smth more clever) it put info about changed region of data into WU header too.


Good idea, but I can think of one reason *not* to do this. Because of the nature of the radar, the bounces, the echoes, the periodicity, etc. we are toggling the blanking bit every 6000 samples, roughly. That's once every .002 seconds, and it's very random in nature. So to accurately depict the radar blanking info within the headers of a ~107 second workunit, we need to inflate each workunit by 200K, which will increase our bandwidth by 33%. These are rough numbers but you get the idea.

- Matt

For S@H Enhanced the 1024*1024 two-bit samples take 256K, sending one-bit blanking reduced to the same sample rate and similarly packed would take 128K. However, the highest time resolution used for analysis is 1/8 the sample rate so the blanking info could be further reduced by a factor of 8 and only need 16K packed. Rewriting the application to make use of that info wouldn't be trivial, though.

For Astropulse, the internal blanking code works on 32K sample data chunks, because of 50% overlap there are 2047 of those in an AP WU. As few as 256 bytes of additional data could represent the external blanking. In that form it would be easy to merge with the internal blanking.

In both cases, the reduced blanking data would need a threshold of some sort. For instance, if three 6000 sample parts of a 32K Astropulse data chunk had been replaced it might or might not be worth looking for signals in that data.

Just musing here. I agree fully with Raistmer that it's better to analyze the real data thoroughly and skip as much of the waste processing of inserted random data as possible.
                                                              Joe
ID: 865290 · Report as offensive
Profile Neil Blaikie
Volunteer tester
Avatar

Send message
Joined: 17 May 99
Posts: 143
Credit: 6,652,341
RAC: 0
Canada
Message 865419 - Posted: 14 Feb 2009, 17:32:26 UTC

I keep getting "project has no jobs available" when requesting work and then a http error. Server status is showing 102,953 results ready to send.

Seems something is amiss somewhere, will keep being patient.
ID: 865419 · Report as offensive
HarryM
Volunteer tester

Send message
Joined: 24 Jul 08
Posts: 68
Credit: 3,812,695
RAC: 0
United States
Message 865420 - Posted: 14 Feb 2009, 17:38:23 UTC - in response to Message 865419.  

See entry on home page. Server problem.
ID: 865420 · Report as offensive
Profile Neil Blaikie
Volunteer tester
Avatar

Send message
Joined: 17 May 99
Posts: 143
Credit: 6,652,341
RAC: 0
Canada
Message 865425 - Posted: 14 Feb 2009, 17:58:07 UTC - in response to Message 865420.  

Again me being too fast at typing, I saw that after I hit send.

Thank you though for replying.
ID: 865425 · Report as offensive
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 865489 - Posted: 14 Feb 2009, 21:39:19 UTC - in response to Message 865290.  

For S@H Enhanced the 1024*1024 two-bit samples take 256K, sending one-bit blanking reduced to the same sample rate and similarly packed would take 128K. However, the highest time resolution used for analysis is 1/8 the sample rate so the blanking info could be further reduced by a factor of 8 and only need 16K packed. Rewriting the application to make use of that info wouldn't be trivial, though.


Right.. This was a quick calculation and I messed up byte/bit conversion so I was off by a factor of 8. But that was just one reason - you pointed out another - the rewriting of the application (and the splitter for that matter).

Forgive my half-understanding here, as the nitty-gritty math details elude me (they keeps getting paged out for a zillion other details). But (a) injecting anything by the right shaped noise during radar blanking periods causes hard-to-identify "edge effects," (b) if we don't insert noise we'll be obliterated with radar in every workunit, and (c) if we simply "skip" the radar periods, we can't do any gaussian searching (which requires contiguous time domain), maybe not even pulse searching given the nature of fast folding. I cannot stress how much data is being blitzed by radar - we're randomizing about 10-15% of the data, once every 0.002 seconds is a blanking period lasting about 0.00025 seconds. This all said, other people around here could give you a clearer idea/argument why we're using whatever methods we're using to deal with radar - I'm just working on finding the stuff and telling everybody where it is - the hard part being that radar quite often doesn't pass any obvious threshold, so I have to lock on to known patterns when they do pass threshold and then interpolate.

- Matt
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 865489 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 865738 - Posted: 15 Feb 2009, 13:45:05 UTC
Last modified: 15 Feb 2009, 13:47:02 UTC

Quick Question: Is the AstroPulse_v5 (5.03) application's work hooked up to the preferences 'Receive Work for AstroPulse' tickbox ? I don't seem to get any except for in blobs of cuda multibeam work when enabling 'Receive work for other applications'. [Incidentally, swamped with cuda work, and CPUs going cold]

Jason
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 865738 · Report as offensive
Claggy
Volunteer tester

Send message
Joined: 5 Jul 99
Posts: 4654
Credit: 47,537,079
RAC: 4
United Kingdom
Message 865745 - Posted: 15 Feb 2009, 14:03:47 UTC - in response to Message 865738.  

Quick Question: Is the AstroPulse_v5 (5.03) application's work hooked up to the preferences 'Receive Work for AstroPulse' tickbox ? I don't seem to get any except for in blobs of cuda multibeam work when enabling 'Receive work for other applications'. [Incidentally, swamped with cuda work, and CPUs going cold]

Jason


I've been wondering that, I've only got Astropulse selected, I have a three way app_info for setiathome_enhanced, astropulse and astropulse_v5,
but the only astropulse i have received are AP 5.00 resends.

Claggy
ID: 865745 · Report as offensive
OzzFan Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Apr 02
Posts: 15691
Credit: 84,761,841
RAC: 28
United States
Message 865766 - Posted: 15 Feb 2009, 15:35:15 UTC

Since AstroPulse v5.03 (astropulse_v5) was released as a new application, and since AP v5.03 WUs are incompatible with the older APP, anyone running an optimized application specifying AP WUs for v5 will only get work for the older AP - of which there is only reissues at this point.

In order to get new AP v5.03 work, you have to either use the stock app or wait for the official release of a AP v5.03 compatible optimized app with the appropriate app_info.xml file.
ID: 865766 · Report as offensive
Profile jason_gee
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 24 Nov 06
Posts: 7489
Credit: 91,093,184
RAC: 0
Australia
Message 865776 - Posted: 15 Feb 2009, 16:04:17 UTC - in response to Message 865766.  
Last modified: 15 Feb 2009, 16:16:26 UTC

...In order to get new AP v5.03 work, you have to either use the stock app or wait for the official release of a AP v5.03 compatible optimized app with the appropriate app_info.xml file.


Correct, which is the one I'm trying to test (including relevant app_info). Being swamped with Cuda work and no way to specify AstroPulse only but still get v5.03 (along with v5.00 old resends), is one factor delaying the release. (Aside from waiting for validation confirmation of the ones that have been received and processed already.)

[Nevermind, pulling multibeam out of the app_info altogether for testing purposes seems to do the trick.]

Jason
"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.
ID: 865776 · Report as offensive
Profile Raistmer
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 16 Jun 01
Posts: 6325
Credit: 106,370,077
RAC: 121
Russia
Message 866138 - Posted: 16 Feb 2009, 15:35:48 UTC - in response to Message 865489.  


Forgive my half-understanding here, as the nitty-gritty math details elude me (they keeps getting paged out for a zillion other details). But (a) injecting anything by the right shaped noise during radar blanking periods causes hard-to-identify "edge effects," (b) if we don't insert noise we'll be obliterated with radar in every workunit, and (c) if we simply "skip" the radar periods, we can't do any gaussian searching (which requires contiguous time domain), maybe not even pulse searching given the nature of fast folding. I cannot stress how much data is being blitzed by radar - we're randomizing about 10-15% of the data, once every 0.002 seconds is a blanking period lasting about 0.00025 seconds. This all said, other people around here could give you a clearer idea/argument why we're using whatever methods we're using to deal with radar - I'm just working on finding the stuff and telling everybody where it is - the hard part being that radar quite often doesn't pass any obvious threshold, so I have to lock on to known patterns when they do pass threshold and then interpolate.

- Matt

a) "right" shape is questionable by itself. New AP 5.03 tasks (for example) give very high overlowed exit ratio still (sampling too small still though, maybe just weird deviation).
b) noise can be inserted by server or can be inserted by client (even less work for server maybe) if client will know what areas it should blank (as doing now in AP client blanking).
c) sure we can't just skip some time areas w/o bad consequencies, but some of processing can be skipped. For example, doing single pulse search in "blanked" data meaningless from scientific point of view and wasteful from performance point of view. Maybe, another kinds of searching can be modified too for alternative accounting of damaged areas.

Yes, this requires some modifications in splitter (perhaps) code, it should write some additional fields in task header. (Manual work, yes, but any optimization is mostly manual work ;) ).
Regarding file size increase - if blanking area bigger than few time samples, compression could be used in form {index of area start,area size} so task size increase souldn't be too dramatical.
ID: 866138 · Report as offensive
Profile KWSN THE Holy Hand Grenade!
Volunteer tester
Avatar

Send message
Joined: 20 Dec 05
Posts: 3187
Credit: 57,163,290
RAC: 0
United States
Message 866159 - Posted: 16 Feb 2009, 16:59:20 UTC
Last modified: 16 Feb 2009, 17:23:27 UTC

In addition to the downloading problem, there now seems to be a problem uploading completed WU's to Berkeley... getting:

2/16/2009 8:57:52 AM|SETI@home|Temporarily failed upload of 16ja09af.28021.72.11.8.100_1_0: system connect

When I try...

[add]
also give Beta a kick, as a similar problem has been over there for a day or two...
[/add]

[2nd add]
re-started working at ~0920 Berkeley time...
[/2nd add]
.

Hello, from Albany, CA!...
ID: 866159 · Report as offensive
Profile Neil Blaikie
Volunteer tester
Avatar

Send message
Joined: 17 May 99
Posts: 143
Credit: 6,652,341
RAC: 0
Canada
Message 866206 - Posted: 16 Feb 2009, 19:21:39 UTC

Someone must have kicked something as the uploads are all working fine on this end.

Haven't had a problem since 9am Berkeley time...then started working again soon after.
ID: 866206 · Report as offensive
Profile ML1
Volunteer moderator
Volunteer tester

Send message
Joined: 25 Nov 01
Posts: 20147
Credit: 7,508,002
RAC: 20
United Kingdom
Message 866582 - Posted: 17 Feb 2009, 23:59:53 UTC - in response to Message 866138.  
Last modified: 18 Feb 2009, 0:01:14 UTC

Good questioning but sorry, I don't see what direction you're aiming for...

a) "right" shape is questionable by itself. New AP 5.03 tasks (for example) give very high overlowed exit ratio still (sampling too small still though, maybe just weird deviation).

Noise "shape" here is likely intended to mean the spectral plot shape. The idea is to have the inserted noise look like background noise for that WU. I imagine that the transition from WU signal to inserted noise and then from inserted noise back to WU signal is going to be a very subtle trick to get it to gently blend without the FFTs 'seeing' the joins.

My own line of attack would be to use noise blanking techniques to cancel out just the radar interference itself. I don't know if that would be less or more difficult than the 'blank noise' insertion.

b) noise can be inserted by server or can be inserted by client (even less work for server maybe) if client will know what areas it should blank (as doing now in AP client blanking).

Anything more complicated for the clients could well become a nightmare for the optimisers!

c) sure we can't just skip some time areas w/o bad consequencies, but some of processing can be skipped. For example, doing single pulse search in "blanked" data meaningless from scientific point of view and wasteful from performance point of view. Maybe, another kinds of searching can be modified too for alternative accounting of damaged areas.

Perhaps just accept the data "as-is" and have the clients ignore all the pulses found in the radar contaminated bits...

Yes, this requires some modifications ... so task size increase souldn't be too dramatical.

I agree to keep all relevant data possible in with the WU for a "just in case" review further down the line. That could save a lot of reprocessing for any hiccups anywhere...


Regardless, trying to keep the data clean is a very difficult problem. Far far better would be for the radar not to be there in the first place.

Keep searchin',
Martin
See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)
ID: 866582 · Report as offensive
1 · 2 · Next

Message boards : Technical News : Fallen Leaf (Feb 12 2009)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.