I wonder if someone would estimate for us how much work is left to fully analyze the existing data at S@H?

I have this vision that someone is actually planning this behemuth calculation and that the information can be made public.

I wonder if someone would estimate for us how much work is left to fully analyze the existing data at S@H?

I have this vision that someone is actually planning this behemuth calculation and that the information can be made public.

What we're crunching right now is old data...

Here's Matt talking about it....a month ago

The data I'm putting in now has only gone through the whole system once a long time ago using clients that didn't do as deep an analysis as Enhanced and in most cases the set of final results were incomplete (failed redundancy checking, database problems, etc.). Still, it is at least partially redundant and therefore there are diminishing returns to sending these workunits out again.

There are two schools of thought: keep users happy at all costs (since they are our most precious resource), or do the most scientifically useful thing regardless of effect on short-term public interest. Both sides have valid arguments. It's tough making such decisions. [Edit: I'm not the one making these decisions.]

- Matt

And then once multibeam starts here hopefully soon, we have 5+ Terabytes of data to crunch.
Here's Matt again....

We have about 5+ terabytes of multibeam data. Actually that number may be way off but I know it is multiple terabytes. That should last us through the season once we start using it.

- Matt

---

I was given the impression, and I could have misinterpreted, that the multibeam data collected so far has random RFI interference from a radar station that renders the data uncrunchable, and that processing would have to begin with new data collected after the 'Radar blanking' implementation is completed and operational. [ from: http://setiathome.berkeley.edu/sah_sci_newsletters.php?frag=news-24.inc ]

With regard to crunching old data, LOL, I guess the workunits currently coming through are mostly late '99 - early '00, so we could have about another 6 to 7 years worth :P (kidding)

---

"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.

I was given the impression, and I could have misinterpreted, that the multibeam data collected so far has random RFI interference from a radar station that renders the data uncrunchable, and that processing would have to begin with new data collected after the 'Radar blanking' implementation is completed and operational. [ from: http://setiathome.berkeley.edu/sah_sci_newsletters.php?frag=news-24.inc ]

With regard to crunching old data, LOL, I guess the workunits currently coming through are mostly late '99 - early '00, so we could have about another 6 to 7 years worth :P (kidding)

There is radar interference in ALL the new multibeam data. There's a workaround for it so that it can be used, but the workaround for the data already collected blanks out a lot of extra stuff just to make sure they get all of the radar signal. The new workaround consists of recording the exact information about when and where the radar signal appears so that it can be blanked out without destroying data unnecessarily.

---

Jim

Some people plan their life out and look back at the wealth they've had.
Others live life day by day and look back at the wealth of experiences and enjoyment they've had.

Thanks Jim, You were pretty clear but I am slow this morning - So let me get this clear in my head. There are two workarounds? One for already collected MB data that is fairly agressive, but hopefully we'll get some useful data out of it.... And second, the new radar blanking system which ensures only specific portions are blanked so we get much more useful data ?

Pretty good if they've managed to salvage any of the already collected data, Oh Joy :D

---

"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.

... The new workaround consists of recording the exact information about when and where the radar signal appears so that it can be blanked out without destroying data unnecessarily.

Anyone know if there is anything for quietening the noise at source?

For example, having the radar blank it's transmissions for a wider arc?

Keep searchin',
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)

... The new workaround consists of recording the exact information about when and where the radar signal appears so that it can be blanked out without destroying data unnecessarily.

Anyone know if there is anything for quietening the noise at source?

For example, having the radar blank it's transmissions for a wider arc?

Keep searchin',
Martin

The "blanking" I spoke of doesn't affect the radar itself. Instead there's an algorithm that searches the tapes for the radar signal and substitutes white noise (similar to the interstellar background noise) in it's place. The new system actually uses an extra track on the data recorder to record the time and duration of the pulses when they are received. This way only the specific pulse has to be "blanked".

Here is the actual info...

(Text removed to save space. I didn't realize that the link to this exact same info was posted in an earlier post in this thread! See the post by jason_glee and my reply for the entire text of the message.)

---

Jim

Some people plan their life out and look back at the wealth they've had.
Others live life day by day and look back at the wealth of experiences and enjoyment they've had.

There is radar interference in ALL the new multibeam data.

Aaa hhaa…..Now we have to wonder….

Could this be a coincidence? How could it BE?

Could this be ET’s desperate (last ditch) attempt to conceal themselves?

Could this be a government conspiracy?

What is the real cause of this……SUDDENLY?

Were we about to break the B’Biggest news in HISTORY?

Does this have to do with Area 51’s, 60th anniversary?

A coincidence like this is in the same realm as the JFK assassination, Princess Diana, OJ Simpson…….think about it!

Do we need to contact 60 minutes?

What is actually going on here???

All I know for sure is we where days or weeks away from crunching MultiBeam data…..and now???

LOL, were the "engineers" that visited to assess the telescope's future wearing black suits and 70's style shades ?

---

"Living by the wisdom of computer science doesn't sound so bad after all. And unlike most advice, it's backed up by proofs." -- Algorithms to live by: The computer science of human decisions.

... A small bit of interfacing electronics needs to be built. Dan and an AO engineer Dana Whitlow will do that when Dan is in Puerto Rico in the middle of this month for a bioastronomy conference. When AO comes back on line after the painting is completed we should be all set.

Good stuff.

Thanks for the detail.

Regards,
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)

... The new workaround consists of recording the exact information about when and where the radar signal appears so that it can be blanked out without destroying data unnecessarily.

Anyone know if there is anything for quietening the noise at source?

For example, having the radar blank it's transmissions for a wider arc?

Keep searchin',
Martin

See this page for links to Arecibo RFI mitigation.

                                                                  Joe

Ok, i have a question about the radar blanking. From what i have read, there is now an algorith that replaces the radar signal with random white noise from the collected multibeam data.

Doesn't this make the data less correct? From what i understand is that some parts will be 'restored', but with 'false' data. Isn't it possible that an ET signal was received during the radar signal, and so removed from the data?

I hope someone can explain me. Thanx in advance!

---

Let's focus on how much processing of 'old' data remains and set aside the 'new' data and the RFI issue, as the corresponding workload is less quantifiable at present.

My question becomes, if we intend to process all the 'old' data to the level of completeness our current algorithms provide, how much more work do we have to do?

It seems that there must be a finite number of tapes (or disks), that map into a finite (but large) number of wu's. Some vast number of these wu's have been processed using the current state of the art and put into the science database. So for my simple mind, what percentage of possible wu's have been completed (up through entry into the science database)?

As a natural follow-up, assuming the current computational capacity persists, and without discounting for new software and datasets being brought on-stream, how many more days/weeks/months do we have left to completely process the 'old' data? (I'm assuming here that the answer is not negative, ie. we are not wasting our time and energy just to keep the seti enging running.)

Ok, i have a question about the radar blanking. From what i have read, there is now an algorith that replaces the radar signal with random white noise from the collected multibeam data.

Doesn't this make the data less correct? From what i understand is that some parts will be 'restored', but with 'false' data. Isn't it possible that an ET signal was received during the radar signal, and so removed from the data?

I hope someone can explain me. Thanx in advance!

You are entirely correct. "If" a signal were sent on the frequency of the radar signal and arrives at the exact time as the radar signal, then yes it will be removed. But the chances of that happening are extremely large. However even if there were to be a signal at the frequency of the radar but coming from a different source, then the chances that it would be received "exactly" when the pulse from the radar arrives, and "only" when the radar pulse arrives would be astronomical (no pun intended!)

Any signal that we would be receiving would be covered up by the offending very powerful local radar signal anyway. This is why there can be so many radio and tv stations using the same frequency. They are separated in distance so the local station overpowers any signal that you may receive from the more distant station.

Also look at all the other information that would be wasted by not removing the radar signal. Yes we do loose a tiny fraction of data but using the algorithm to remove the known signal lets us search for unknown signals in the remaining data (either at the same freq but slightly removed in time, or at the same time but slightly removed in freq.) If the offending radar signal were left in, the entire work unit would be trashed (it would return a -9 overflow or more than 30 signals so it would be invalidated and trashed.) Using the algorithm to replace the small percentage of the wu with noise allows the rest of the wu to be processed.

---

Jim

Some people plan their life out and look back at the wealth they've had.
Others live life day by day and look back at the wealth of experiences and enjoyment they've had.

Let's focus on how much processing of 'old' data remains and set aside the 'new' data and the RFI issue, as the corresponding workload is less quantifiable at present.

My question becomes, if we intend to process all the 'old' data to the level of completeness our current algorithms provide, how much more work do we have to do?

It seems that there must be a finite number of tapes (or disks), that map into a finite (but large) number of wu's. Some vast number of these wu's have been processed using the current state of the art and put into the science database. So for my simple mind, what percentage of possible wu's have been completed (up through entry into the science database)?

As a natural follow-up, assuming the current computational capacity persists, and without discounting for new software and datasets being brought on-stream, how many more days/weeks/months do we have left to completely process the 'old' data? (I'm assuming here that the answer is not negative, ie. we are not wasting our time and energy just to keep the seti enging running.)

First I don't think the intention is to reprocess "all" the work units that were crunched with the older app. Just the ones that for some reason didn't make it into the database or were lost due to database corruption. And yes, to give everybody something to do until the multibeam info arrives. But before I start people grumbling about redoing old work, this IS scientifically valid work since it IS being crunched a second time with a much more powerful tool!

But if you look at it, there was about seven years spent crunching on the original data (1999 to 2006 when we started with the Enhanced app) (and I may be wrong about the starting date, I wasn't here then!) Taking Moore's Law into consideration, we could probably crunch that same data in about a fourth of that time. That is if enough users kept up with the newest and fastest developments for crunching machines. (edit) And if the law that "nature abhors a vacuum" doesn't come into the picture. meaning that if processing power increases, we could expand the search to even greater depths with different apps.

This question could probably only be answered by Eric or someone else on the staff that knows exactly how many data tapes did not get crunched by the new app. and exactly how fast we can crunch them. I'm just giving a wild guess here.

---

Jim

Some people plan their life out and look back at the wealth they've had.
Others live life day by day and look back at the wealth of experiences and enjoyment they've had.

Well, we have been running ESAH for about a year now. Therefore, one could conclude it would take a least several years (probably more) to go back and run all the old data thorough ESAH, based on how you framed your question.

However, I think the goal is to make sure all the old data has had a chance to go through the mill at least once to completion. In which case the question becomes how many of those tapes are left to do.

IIRC, wasn't there a page on the Classic site which showed a grid diagram of which tapes had been run and which hadn't, along with their progress status? Maybe you could get a rough idea from that. I would think it should be pretty accurate at least for up until they closed down Classic.

Alinator

Ah ok, now i understand. Thank you for explaining Jim!

---

Well, we have been running ESAH for about a year now. [snip]
Alinator

May 6th 2006

Well, we have been running ESAH for about a year now. [snip]
Alinator

May 6th 2006

LOL... Seems like yesterday! :-)

You are entirely correct.

<snip>

Also look at all the other information that would be wasted by not removing the radar signal. Yes we do loose a tiny fraction of data but using the algorithm to remove the known signal lets us search for unknown signals in the remaining data (either at the same freq but slightly removed in time, or at the same time but slightly removed in freq.) If the offending radar signal were left in, the entire work unit would be trashed (it would return a -9 overflow or more than 30 signals so it would be invalidated and trashed.) Using the algorithm to replace the small percentage of the wu with noise allows the rest of the wu to be processed.

Yep, the Newsletter and the related RFI info on the Arecibo site was very interesting.

It seems to me though that the blanking technique they're choosing at the moment would most likely be their second choice, if they had their druthers.

Since the signal source is known, characterizable, and they have a couple of spare channels in the recorder, it should be possible to DSP it right out without degrading the remaining signal significantly.

Of course, the catch is finding the time and the scratch to do that! ;-)

Alinator