Rainy Day Data (May 14 2009)


log in

Advanced search

Message boards : Technical News : Rainy Day Data (May 14 2009)

Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 894724 - Posted: 14 May 2009, 20:40:07 UTC

We are quite preoccupied with anniversary stuff so we've been doing the bare minimum amount of systems administration to get by until after the event. Still, it should be mentioned we continue to have SATA/driver issues on our data recorder at Arecibo, and haven't collected new data for about a month now. While we have a pile of data yet to crunch readily available on disk, I started pulling up unanalyzed data from our offsite archives.

Before doing so I went through the whole data inventory rigamarole this morning. We have 1787 raw multi-beam data files (mostly all 50GB in size) archived, of which 338 haven't been split at all. However, a portion of these files were recorded before 2008, i.e. before we had a hardware radar blanking signal embedded in the data. So until we get my software radar blanker working (a project postponed until post-anniversary) we can't chew on these files without dealing with major radio frequency interference. This isn't a major problem: 1225 of the 1787 archived data files are from 2008 or later, and of these 249 have yet to be split. So we got plenty of numbers to crunch until we get the data recorder working again.

- Matt
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Profile Gary Charpentier
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 12144
Credit: 6,424,042
RAC: 8,126
United States
Message 894763 - Posted: 14 May 2009, 22:04:23 UTC - in response to Message 894724.

We are quite preoccupied with anniversary stuff so we've been doing the bare minimum amount of systems administration to get by until after the event. Still, it should be mentioned we continue to have SATA/driver issues on our data recorder at Arecibo, and haven't collected new data for about a month now. While we have a pile of data yet to crunch readily available on disk, I started pulling up unanalyzed data from our offsite archives.

Before doing so I went through the whole data inventory rigamarole this morning. We have 1787 raw multi-beam data files (mostly all 50GB in size) archived, of which 338 haven't been split at all. However, a portion of these files were recorded before 2008, i.e. before we had a hardware radar blanking signal embedded in the data. So until we get my software radar blanker working (a project postponed until post-anniversary) we can't chew on these files without dealing with major radio frequency interference. This isn't a major problem: 1225 of the 1787 archived data files are from 2008 or later, and of these 249 have yet to be split. So we got plenty of numbers to crunch until we get the data recorder working again.

- Matt

Nice to hear about the MB data, but where does this leave you with AP data?

Hope you get the problems as Arecibo fixed soon. Wouldn't want ET to try and call when we aren't recording.


____________

John McLeod VII
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 15 Jul 99
Posts: 24102
Credit: 518,020
RAC: 150
United States
Message 894836 - Posted: 15 May 2009, 2:42:11 UTC - in response to Message 894763.

We are quite preoccupied with anniversary stuff so we've been doing the bare minimum amount of systems administration to get by until after the event. Still, it should be mentioned we continue to have SATA/driver issues on our data recorder at Arecibo, and haven't collected new data for about a month now. While we have a pile of data yet to crunch readily available on disk, I started pulling up unanalyzed data from our offsite archives.

Before doing so I went through the whole data inventory rigamarole this morning. We have 1787 raw multi-beam data files (mostly all 50GB in size) archived, of which 338 haven't been split at all. However, a portion of these files were recorded before 2008, i.e. before we had a hardware radar blanking signal embedded in the data. So until we get my software radar blanker working (a project postponed until post-anniversary) we can't chew on these files without dealing with major radio frequency interference. This isn't a major problem: 1225 of the 1787 archived data files are from 2008 or later, and of these 249 have yet to be split. So we got plenty of numbers to crunch until we get the data recorder working again.

- Matt

Nice to hear about the MB data, but where does this leave you with AP data?

Hope you get the problems as Arecibo fixed soon. Wouldn't want ET to try and call when we aren't recording.


AP data and MultiBeam data are the same, they are split and processed differently.
____________


BOINC WIKI

Profile S@NL - Eesger - www.knoop.nl
Avatar
Send message
Joined: 7 Oct 01
Posts: 384
Credit: 35,659,000
RAC: 19,468
Netherlands
Message 894934 - Posted: 15 May 2009, 14:02:29 UTC

For the off chance it hasn't been seen yet, all the 'ap_splitter's are not running..
____________
The SETI@Home Gauntlet 2012 april 16 - 30| info / chat | STATS

Profile cliff west
Send message
Joined: 7 May 01
Posts: 197
Credit: 11,263,590
RAC: 13
United States
Message 894962 - Posted: 15 May 2009, 15:40:08 UTC

with the anniversary coming up and the press coverage that it will generat you might want to have a lot of units ready for new/returning or currant accounts. it would look bad it you ran out of work after all the time you all have put in for the anniversary events.
____________

Profile KWSN THE Holy Hand Grenade!
Volunteer tester
Avatar
Send message
Joined: 20 Dec 05
Posts: 1902
Credit: 9,208,517
RAC: 13,483
United States
Message 895001 - Posted: 15 May 2009, 17:01:43 UTC - in response to Message 894934.

For the off chance it hasn't been seen yet, all the 'ap_splitter's are not running..


Yeah, but... didn't you notice that almost all the "tapes" in the queue have been split for AP, but not for MB? The staff may just be letting MB catch up...
____________
.

Richard Haselgrove
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8375
Credit: 46,757,796
RAC: 22,383
United Kingdom
Message 895011 - Posted: 15 May 2009, 17:25:18 UTC - in response to Message 894962.

with the anniversary coming up and the press coverage that it will generat you might want to have a lot of units ready for new/returning or currant accounts. it would look bad it you ran out of work after all the time you all have put in for the anniversary events.

It would be a polite welcome to new crunchers (break them in gently!) to avoid splitting tapes which are known to have a high proportion of VLAR jobs in them during the anniversary publicity blitz: that is, tapes recorded when project P2030 has scheduled observing time.

Fortunately the broken SATA connector means we seem to have missed the threatened VLAR storm I predicted for the beginning of this month, in message 890296.

Cosmic_Ocean
Avatar
Send message
Joined: 23 Dec 00
Posts: 2238
Credit: 8,453,224
RAC: 4,095
United States
Message 895029 - Posted: 15 May 2009, 18:05:46 UTC - in response to Message 895001.

For the off chance it hasn't been seen yet, all the 'ap_splitter's are not running..


Yeah, but... didn't you notice that almost all the "tapes" in the queue have been split for AP, but not for MB? The staff may just be letting MB catch up...

Or the WU storage is full from all the B3_P1 WUs and it is taking a while for those WUs to get the 6 errors reported and be able to be deleted.
____________

Linux laptop uptime: 1484d 22h 42m
Ended due to UPS failure, found 14 hours after the fact

Speedy
Volunteer tester
Avatar
Send message
Joined: 26 Jun 04
Posts: 644
Credit: 5,384,658
RAC: 7,034
New Zealand
Message 896101 - Posted: 17 May 2009, 20:52:29 UTC
Last modified: 17 May 2009, 20:53:47 UTC

As of 17 May 2009 20:40:09 there is no ap data been split, mb data is been split at 19.3919. Iwill be sweet data wise until after the weekly outage. Id no new data is added until tomorrow or even until after the weekly outage it will be interesting to see how many results are waiting to be returned. Currently there are 430,111 results waiting to be returned.
____________

Live in NZ y not join Smile City?

Profile Neil Blaikie
Volunteer tester
Avatar
Send message
Joined: 17 May 99
Posts: 142
Credit: 6,466,200
RAC: 6
Canada
Message 896223 - Posted: 18 May 2009, 0:43:24 UTC

As mentioned there are a lot of the dreaded VLAR workunits out at the moment, I had several all auto-killed with the optimized app as error -6. Had one with this which is something I have not seen before.

Error in ap_remove_radar.cpp: generate_envelope: num_ffts_performed < 100. Blanking too much RFI?

____________

Speedy
Volunteer tester
Avatar
Send message
Joined: 26 Jun 04
Posts: 644
Credit: 5,384,658
RAC: 7,034
New Zealand
Message 896249 - Posted: 18 May 2009, 2:01:17 UTC

Thats right, people that get VLAR workunits can help by letting these run to free space up for good work units. Thanks for letting these tasks run 1st
____________

Live in NZ y not join Smile City?

Profile Neil Blaikie
Volunteer tester
Avatar
Send message
Joined: 17 May 99
Posts: 142
Credit: 6,466,200
RAC: 6
Canada
Message 896332 - Posted: 18 May 2009, 6:07:48 UTC - in response to Message 896249.

I do not think that is a fair point at all, if they slow your PC/GPU to a crawl in some cases then why run them?, to free up space for good workunits (define a good work unit, it is the data on the tape that determines that)

Personally I could let them run easily, as could anyone who received them, the point being though is they are annoying to some crunchers and not everyone likes them. It is personal choice to run optimized apps with a VLAR autokill. I do use my PC for other things other than SETI, so why should I compensate for possibly finding "a signal" in a work unit against actually being able to use the PC.

VLAR's affect my computer and I am sure several other crunchers who choose to kill them. They get crunched eventually either by someone with a machine who doesn't get effected or as much as I hate to say this, by a new user joining who has no idea what is going on when receiving a VLAR.

The project is doing amazing work on a budget that is probably laughed at in the scientific community (no offence meant there), and yes it has hardships but as MATT mentioned, they are reluctant to change anything major until after the anniversary. People are willing to spend their money on donating to the project, but it is not enough to cover every possible problem that arises.

Who knows (apart from a select few) what may happen after the anniversary, things will am sure will have to change in some way or another as it will generate more users depending on the overall coverage worldwide and within the scientific community.

I do not mean this post to be an attack on anyone, anything in any way. It is my opinion and not meant to open up discussion on what is the right or wrong thing to do, I leave that to the project admins and face the consequences of whatever they decide to do.
____________

Richard Haselgrove
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8375
Credit: 46,757,796
RAC: 22,383
United Kingdom
Message 896348 - Posted: 18 May 2009, 8:15:53 UTC - in response to Message 896223.

... Had one with this which is something I have not seen before.

Error in ap_remove_radar.cpp: generate_envelope: num_ffts_performed < 100. Blanking too much RFI?

These have been discussed and analysed in Number Crunching. There was a stuck bit in the B3_P1 recording channel for about a month: the staff should be aware (at least, they will be if they read email from Josef W. Segur!)

@ Matt,

The Beta website seems totally dead this morning - HTTP 500 Internal Server Error.

Profile Virtual Boss*
Volunteer tester
Avatar
Send message
Joined: 4 May 08
Posts: 417
Credit: 6,170,041
RAC: 792
Australia
Message 896378 - Posted: 18 May 2009, 11:10:54 UTC - in response to Message 896348.

... Had one with this which is something I have not seen before.

Error in ap_remove_radar.cpp: generate_envelope: num_ffts_performed < 100. Blanking too much RFI?

These have been discussed and analysed in Number Crunching. There was a stuck bit in the B3_P1 recording channel for about a month: the staff should be aware (at least, they will be if they read email from Josef W. Segur!)

@ Matt,

The Beta website seems totally dead this morning - HTTP 500 Internal Server Error.


The interesting thing is even though they are shown as compute error I have received credit for at least one of them (all 0.05 of it). Will see what happens with other two in my pendings list.

Josef W. Segur
Volunteer developer
Volunteer tester
Send message
Joined: 30 Oct 99
Posts: 4206
Credit: 1,030,755
RAC: 271
United States
Message 896455 - Posted: 18 May 2009, 15:32:56 UTC - in response to Message 896348.

... Had one with this which is something I have not seen before.

Error in ap_remove_radar.cpp: generate_envelope: num_ffts_performed < 100. Blanking too much RFI?

These have been discussed and analysed in Number Crunching. There was a stuck bit in the B3_P1 recording channel for about a month: the staff should be aware (at least, they will be if they read email from Josef W. Segur!)
...

Eric did reply to my initial email, when we only knew that 12mr09 through 14mr09 were affected. I later updated that, indicating the problem extended at least through 03ap09. Now we know that all 59 'tapes' from 09mr09aa through 07ap09ag have that problem, but 02mr09ae and earlier didn't.

My guess is there's still a stuck bit on the beam 3 polarity 1 channel, if there's quicklook data available perhaps Matt could check?

Another thought is that the mb_splitter processes have not yet started on any of those 'tapes'. When they do, dividing the data into 256 subbands will hide the flaw; I think S@H Enhanced will simply find the data unusually quiet. Still, it will be a waste of host CPU cycles to crunch that data. Perhaps those channels could somehow be marked 'done' before the mb_splitters get to them?
Joe

Olli
Send message
Joined: 25 Apr 07
Posts: 143
Credit: 2,089,162
RAC: 0
Finland
Message 897506 - Posted: 21 May 2009, 0:29:32 UTC - in response to Message 896455.

We are all working very hard on this. Well one of my computers is anyhow.
Trying to relax and enjoy computing with you all, resolvement of the matters will be achieved someday somehow.

Close to breaking the first million with seti, whoo.

Awaiting for the public release of the celebration speeches.

Olli
____________

Message boards : Technical News : Rainy Day Data (May 14 2009)

Copyright © 2014 University of California