Rainy Day Data (May 14 2009)

Message boards : Technical News : Rainy Day Data (May 14 2009)
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 894724 - Posted: 14 May 2009, 20:40:07 UTC

We are quite preoccupied with anniversary stuff so we've been doing the bare minimum amount of systems administration to get by until after the event. Still, it should be mentioned we continue to have SATA/driver issues on our data recorder at Arecibo, and haven't collected new data for about a month now. While we have a pile of data yet to crunch readily available on disk, I started pulling up unanalyzed data from our offsite archives.

Before doing so I went through the whole data inventory rigamarole this morning. We have 1787 raw multi-beam data files (mostly all 50GB in size) archived, of which 338 haven't been split at all. However, a portion of these files were recorded before 2008, i.e. before we had a hardware radar blanking signal embedded in the data. So until we get my software radar blanker working (a project postponed until post-anniversary) we can't chew on these files without dealing with major radio frequency interference. This isn't a major problem: 1225 of the 1787 archived data files are from 2008 or later, and of these 249 have yet to be split. So we got plenty of numbers to crunch until we get the data recorder working again.

- Matt
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 894724 · Report as offensive
Profile Gary Charpentier Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 25 Dec 00
Posts: 30608
Credit: 53,134,872
RAC: 32
United States
Message 894763 - Posted: 14 May 2009, 22:04:23 UTC - in response to Message 894724.  

We are quite preoccupied with anniversary stuff so we've been doing the bare minimum amount of systems administration to get by until after the event. Still, it should be mentioned we continue to have SATA/driver issues on our data recorder at Arecibo, and haven't collected new data for about a month now. While we have a pile of data yet to crunch readily available on disk, I started pulling up unanalyzed data from our offsite archives.

Before doing so I went through the whole data inventory rigamarole this morning. We have 1787 raw multi-beam data files (mostly all 50GB in size) archived, of which 338 haven't been split at all. However, a portion of these files were recorded before 2008, i.e. before we had a hardware radar blanking signal embedded in the data. So until we get my software radar blanker working (a project postponed until post-anniversary) we can't chew on these files without dealing with major radio frequency interference. This isn't a major problem: 1225 of the 1787 archived data files are from 2008 or later, and of these 249 have yet to be split. So we got plenty of numbers to crunch until we get the data recorder working again.

- Matt

Nice to hear about the MB data, but where does this leave you with AP data?

Hope you get the problems as Arecibo fixed soon. Wouldn't want ET to try and call when we aren't recording.


ID: 894763 · Report as offensive
John McLeod VII
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jul 99
Posts: 24806
Credit: 790,712
RAC: 0
United States
Message 894836 - Posted: 15 May 2009, 2:42:11 UTC - in response to Message 894763.  

We are quite preoccupied with anniversary stuff so we've been doing the bare minimum amount of systems administration to get by until after the event. Still, it should be mentioned we continue to have SATA/driver issues on our data recorder at Arecibo, and haven't collected new data for about a month now. While we have a pile of data yet to crunch readily available on disk, I started pulling up unanalyzed data from our offsite archives.

Before doing so I went through the whole data inventory rigamarole this morning. We have 1787 raw multi-beam data files (mostly all 50GB in size) archived, of which 338 haven't been split at all. However, a portion of these files were recorded before 2008, i.e. before we had a hardware radar blanking signal embedded in the data. So until we get my software radar blanker working (a project postponed until post-anniversary) we can't chew on these files without dealing with major radio frequency interference. This isn't a major problem: 1225 of the 1787 archived data files are from 2008 or later, and of these 249 have yet to be split. So we got plenty of numbers to crunch until we get the data recorder working again.

- Matt

Nice to hear about the MB data, but where does this leave you with AP data?

Hope you get the problems as Arecibo fixed soon. Wouldn't want ET to try and call when we aren't recording.


AP data and MultiBeam data are the same, they are split and processed differently.


BOINC WIKI
ID: 894836 · Report as offensive
Profile S@NL - Eesger - www.knoop.nl
Avatar

Send message
Joined: 7 Oct 01
Posts: 385
Credit: 50,200,038
RAC: 0
Netherlands
Message 894934 - Posted: 15 May 2009, 14:02:29 UTC

For the off chance it hasn't been seen yet, all the 'ap_splitter's are not running..
The SETI@Home Gauntlet 2012 april 16 - 30| info / chat | STATS
ID: 894934 · Report as offensive
Profile cliff west

Send message
Joined: 7 May 01
Posts: 211
Credit: 16,180,728
RAC: 15
United States
Message 894962 - Posted: 15 May 2009, 15:40:08 UTC

with the anniversary coming up and the press coverage that it will generat you might want to have a lot of units ready for new/returning or currant accounts. it would look bad it you ran out of work after all the time you all have put in for the anniversary events.
ID: 894962 · Report as offensive
Profile KWSN THE Holy Hand Grenade!
Volunteer tester
Avatar

Send message
Joined: 20 Dec 05
Posts: 3187
Credit: 57,163,290
RAC: 0
United States
Message 895001 - Posted: 15 May 2009, 17:01:43 UTC - in response to Message 894934.  

For the off chance it hasn't been seen yet, all the 'ap_splitter's are not running..


Yeah, but... didn't you notice that almost all the "tapes" in the queue have been split for AP, but not for MB? The staff may just be letting MB catch up...
.

Hello, from Albany, CA!...
ID: 895001 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 895011 - Posted: 15 May 2009, 17:25:18 UTC - in response to Message 894962.  

with the anniversary coming up and the press coverage that it will generat you might want to have a lot of units ready for new/returning or currant accounts. it would look bad it you ran out of work after all the time you all have put in for the anniversary events.

It would be a polite welcome to new crunchers (break them in gently!) to avoid splitting tapes which are known to have a high proportion of VLAR jobs in them during the anniversary publicity blitz: that is, tapes recorded when project P2030 has scheduled observing time.

Fortunately the broken SATA connector means we seem to have missed the threatened VLAR storm I predicted for the beginning of this month, in message 890296.
ID: 895011 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 895029 - Posted: 15 May 2009, 18:05:46 UTC - in response to Message 895001.  

For the off chance it hasn't been seen yet, all the 'ap_splitter's are not running..


Yeah, but... didn't you notice that almost all the "tapes" in the queue have been split for AP, but not for MB? The staff may just be letting MB catch up...

Or the WU storage is full from all the B3_P1 WUs and it is taking a while for those WUs to get the 6 errors reported and be able to be deleted.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 895029 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1639
Credit: 12,921,799
RAC: 89
New Zealand
Message 896101 - Posted: 17 May 2009, 20:52:29 UTC
Last modified: 17 May 2009, 20:53:47 UTC

As of 17 May 2009 20:40:09 there is no ap data been split, mb data is been split at 19.3919. Iwill be sweet data wise until after the weekly outage. Id no new data is added until tomorrow or even until after the weekly outage it will be interesting to see how many results are waiting to be returned. Currently there are 430,111 results waiting to be returned.
ID: 896101 · Report as offensive
Profile Neil Blaikie
Volunteer tester
Avatar

Send message
Joined: 17 May 99
Posts: 143
Credit: 6,652,341
RAC: 0
Canada
Message 896223 - Posted: 18 May 2009, 0:43:24 UTC

As mentioned there are a lot of the dreaded VLAR workunits out at the moment, I had several all auto-killed with the optimized app as error -6. Had one with this which is something I have not seen before.

Error in ap_remove_radar.cpp: generate_envelope: num_ffts_performed < 100. Blanking too much RFI?

ID: 896223 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1639
Credit: 12,921,799
RAC: 89
New Zealand
Message 896249 - Posted: 18 May 2009, 2:01:17 UTC

Thats right, people that get VLAR workunits can help by letting these run to free space up for good work units. Thanks for letting these tasks run 1st
ID: 896249 · Report as offensive
Profile Neil Blaikie
Volunteer tester
Avatar

Send message
Joined: 17 May 99
Posts: 143
Credit: 6,652,341
RAC: 0
Canada
Message 896332 - Posted: 18 May 2009, 6:07:48 UTC - in response to Message 896249.  

I do not think that is a fair point at all, if they slow your PC/GPU to a crawl in some cases then why run them?, to free up space for good workunits (define a good work unit, it is the data on the tape that determines that)

Personally I could let them run easily, as could anyone who received them, the point being though is they are annoying to some crunchers and not everyone likes them. It is personal choice to run optimized apps with a VLAR autokill. I do use my PC for other things other than SETI, so why should I compensate for possibly finding "a signal" in a work unit against actually being able to use the PC.

VLAR's affect my computer and I am sure several other crunchers who choose to kill them. They get crunched eventually either by someone with a machine who doesn't get effected or as much as I hate to say this, by a new user joining who has no idea what is going on when receiving a VLAR.

The project is doing amazing work on a budget that is probably laughed at in the scientific community (no offence meant there), and yes it has hardships but as MATT mentioned, they are reluctant to change anything major until after the anniversary. People are willing to spend their money on donating to the project, but it is not enough to cover every possible problem that arises.

Who knows (apart from a select few) what may happen after the anniversary, things will am sure will have to change in some way or another as it will generate more users depending on the overall coverage worldwide and within the scientific community.

I do not mean this post to be an attack on anyone, anything in any way. It is my opinion and not meant to open up discussion on what is the right or wrong thing to do, I leave that to the project admins and face the consequences of whatever they decide to do.
ID: 896332 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 896348 - Posted: 18 May 2009, 8:15:53 UTC - in response to Message 896223.  

... Had one with this which is something I have not seen before.

Error in ap_remove_radar.cpp: generate_envelope: num_ffts_performed < 100. Blanking too much RFI?

These have been discussed and analysed in Number Crunching. There was a stuck bit in the B3_P1 recording channel for about a month: the staff should be aware (at least, they will be if they read email from Josef W. Segur!)

@ Matt,

The Beta website seems totally dead this morning - HTTP 500 Internal Server Error.
ID: 896348 · Report as offensive
Profile Virtual Boss*
Volunteer tester
Avatar

Send message
Joined: 4 May 08
Posts: 417
Credit: 6,440,287
RAC: 0
Australia
Message 896378 - Posted: 18 May 2009, 11:10:54 UTC - in response to Message 896348.  

... Had one with this which is something I have not seen before.

Error in ap_remove_radar.cpp: generate_envelope: num_ffts_performed < 100. Blanking too much RFI?

These have been discussed and analysed in Number Crunching. There was a stuck bit in the B3_P1 recording channel for about a month: the staff should be aware (at least, they will be if they read email from Josef W. Segur!)

@ Matt,

The Beta website seems totally dead this morning - HTTP 500 Internal Server Error.


The interesting thing is even though they are shown as compute error I have received credit for at least one of them (all 0.05 of it). Will see what happens with other two in my pendings list.
ID: 896378 · Report as offensive
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 896455 - Posted: 18 May 2009, 15:32:56 UTC - in response to Message 896348.  

... Had one with this which is something I have not seen before.

Error in ap_remove_radar.cpp: generate_envelope: num_ffts_performed < 100. Blanking too much RFI?

These have been discussed and analysed in Number Crunching. There was a stuck bit in the B3_P1 recording channel for about a month: the staff should be aware (at least, they will be if they read email from Josef W. Segur!)
...

Eric did reply to my initial email, when we only knew that 12mr09 through 14mr09 were affected. I later updated that, indicating the problem extended at least through 03ap09. Now we know that all 59 'tapes' from 09mr09aa through 07ap09ag have that problem, but 02mr09ae and earlier didn't.

My guess is there's still a stuck bit on the beam 3 polarity 1 channel, if there's quicklook data available perhaps Matt could check?

Another thought is that the mb_splitter processes have not yet started on any of those 'tapes'. When they do, dividing the data into 256 subbands will hide the flaw; I think S@H Enhanced will simply find the data unusually quiet. Still, it will be a waste of host CPU cycles to crunch that data. Perhaps those channels could somehow be marked 'done' before the mb_splitters get to them?
                                                               Joe
ID: 896455 · Report as offensive
Olli

Send message
Joined: 25 Apr 07
Posts: 143
Credit: 2,089,162
RAC: 0
Finland
Message 897506 - Posted: 21 May 2009, 0:29:32 UTC - in response to Message 896455.  

We are all working very hard on this. Well one of my computers is anyhow.
Trying to relax and enjoy computing with you all, resolvement of the matters will be achieved someday somehow.

Close to breaking the first million with seti, whoo.

Awaiting for the public release of the celebration speeches.

Olli
ID: 897506 · Report as offensive

Message boards : Technical News : Rainy Day Data (May 14 2009)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.