more data on the way

Message boards : News : more data on the way
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · 4 · Next

AuthorMessage
Jeff Cobb Project Donor
Volunteer moderator
Project administrator
Project developer
Project scientist

Send message
Joined: 1 Mar 99
Posts: 122
Credit: 40,367
RAC: 0
United States
Message 1150111 - Posted: 8 Sep 2011, 16:56:49 UTC

In an attempt to push some older unanalyzed files through the pipeline we encountered data that could not be successfully preprocessed or split. This has resulted in work distribution going to near zero. We are currently transferring newer files from off site storage and soon will be receiving a disk shipment from Arecibo. Once these data are on hand work distribution will pick up.
ID: 1150111 · Report as offensive
EdwardPF
Volunteer tester

Send message
Joined: 26 Jul 99
Posts: 389
Credit: 236,772,605
RAC: 374
United States
Message 1150114 - Posted: 8 Sep 2011, 16:58:47 UTC - in response to Message 1150111.  

Thanks for the info!!

Ed F
ID: 1150114 · Report as offensive
Profile Jim_S
Avatar

Send message
Joined: 23 Feb 00
Posts: 4705
Credit: 64,560,357
RAC: 31
United States
Message 1150115 - Posted: 8 Sep 2011, 17:00:54 UTC

Thanks for the info Jeff!!!

I Desire Peace and Justice, Jim Scott (Mod-Ret.)
ID: 1150115 · Report as offensive
Profile S@NL Etienne Dokkum
Volunteer tester
Avatar

Send message
Joined: 11 Jun 99
Posts: 212
Credit: 43,822,095
RAC: 0
Netherlands
Message 1150116 - Posted: 8 Sep 2011, 17:16:09 UTC

Thanx for the update Jeff !

a lot of people screaming for WU's...should relax a little and realise this is a volunteer project and gaining credit isn't the point of it all. If "we" find ET in my lifetime I'll be the happiest man alive. That's all there is to it.

Keep up the good job guys !!!!
ID: 1150116 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51478
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1150119 - Posted: 8 Sep 2011, 17:18:38 UTC

Thank you very much for the update, Jeff!!!
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1150119 · Report as offensive
Profile Dimly Lit Lightbulb 😀
Volunteer tester
Avatar

Send message
Joined: 30 Aug 08
Posts: 15399
Credit: 7,423,413
RAC: 1
United Kingdom
Message 1150127 - Posted: 8 Sep 2011, 17:36:02 UTC

Thanks for the update Jeff.
ID: 1150127 · Report as offensive
Claggy
Volunteer tester

Send message
Joined: 5 Jul 99
Posts: 4654
Credit: 47,537,079
RAC: 4
United Kingdom
Message 1150160 - Posted: 8 Sep 2011, 19:15:09 UTC - in response to Message 1150111.  

Thanks for the update Jeff,

Claggy
ID: 1150160 · Report as offensive
Profile Jeff Mercer

Send message
Joined: 14 Aug 08
Posts: 90
Credit: 162,139
RAC: 0
United States
Message 1150200 - Posted: 8 Sep 2011, 20:50:00 UTC

Thanks for the update. By any chance could you tell us if the data from the GBT is being sent out yet. Also, thanks for the hard work in getting the project back up and running. Everything working just FINE on my end !!

ID: 1150200 · Report as offensive
Profile Dirk Sadowski
Volunteer tester

Send message
Joined: 6 Apr 07
Posts: 7105
Credit: 147,663,825
RAC: 5
Germany
Message 1150229 - Posted: 8 Sep 2011, 22:08:19 UTC - in response to Message 1150111.  

Jeff, thanks for the news!


- Best regards! - Sutaru Tsureku, team seti.international founder. - Optimize your PC for higher RAC. - SETI@home needs your help. -
ID: 1150229 · Report as offensive
Fr. Oz211

Send message
Joined: 6 Mar 09
Posts: 1
Credit: 770,112
RAC: 1
United States
Message 1150299 - Posted: 9 Sep 2011, 3:10:37 UTC

Thanks for the update. I wondered why my Boinc manager kept telling me there was no data to process!
Ron Z.
ID: 1150299 · Report as offensive
tbret
Volunteer tester
Avatar

Send message
Joined: 28 May 99
Posts: 3380
Credit: 296,162,071
RAC: 40
United States
Message 1150305 - Posted: 9 Sep 2011, 3:22:47 UTC

Thank you for seeing this all the way through to the fix.
ID: 1150305 · Report as offensive
Profile Boomer
Volunteer tester

Send message
Joined: 1 Jun 09
Posts: 43
Credit: 21,892,319
RAC: 10
United States
Message 1150346 - Posted: 9 Sep 2011, 5:28:27 UTC

Excuse me for asking a STUPID question, but why isn't the data transferred through the internet directly to you at Berkeley? Sneaker net from Areceibo isn't the most efficient way of transferring data.

Please forgive. Meesa so stupid.
ID: 1150346 · Report as offensive
OzzFan Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Apr 02
Posts: 15691
Credit: 84,761,841
RAC: 28
United States
Message 1150394 - Posted: 9 Sep 2011, 13:07:49 UTC - in response to Message 1150346.  

Excuse me for asking a STUPID question, but why isn't the data transferred through the internet directly to you at Berkeley? Sneaker net from Areceibo isn't the most efficient way of transferring data.

Please forgive. Meesa so stupid.


The data is recorded at the Arecibo satellite dish onto hard drives. Because we're talking about hundreds of gigabytes worth of data, it is actually faster to ship the hard drives to Berkeley than it is to transfer the work across the internet itself.


Ironically enough, once the hard drives are received at SETI@Home's lab in the Space Sciences Laboratory in Berkeley, the raw data is then transferred to a secure offline storage via the internet. If at any time they need more data, they simply download the raw data files back into the lab to be split and processed as workunits.


I'd imagine it would be possible to connect a fat internet pipe directly from Arecibo to Berkeley, but somebody would have to pay for it. While people may be tired of hearing about it, the truth is SETI doesn't have that kind of money to be paying for big projects like that. I'm not really sure it's practical to get an internet connection from Arecibo to Berkeley when there's more pressing matters in need of attention.
ID: 1150394 · Report as offensive
Jon

Send message
Joined: 18 Aug 11
Posts: 2
Credit: 335,920
RAC: 0
United States
Message 1150404 - Posted: 9 Sep 2011, 13:27:21 UTC

Thanks for keeping us in the loop.
ID: 1150404 · Report as offensive
TribbleRED

Send message
Joined: 9 May 10
Posts: 3
Credit: 3,385,328
RAC: 7
United States
Message 1150440 - Posted: 9 Sep 2011, 15:43:35 UTC

Feel free to send me some unsplit data... Ill take care of it on my end.
ID: 1150440 · Report as offensive
TribbleRED

Send message
Joined: 9 May 10
Posts: 3
Credit: 3,385,328
RAC: 7
United States
Message 1150443 - Posted: 9 Sep 2011, 15:49:15 UTC - in response to Message 1150394.  
Last modified: 9 Sep 2011, 15:53:26 UTC

Excuse me for asking a STUPID question, but why isn't the data transferred through the internet directly to you at Berkeley? Sneaker net from Areceibo isn't the most efficient way of transferring data.

Please forgive. Meesa so stupid.


The data is recorded at the Arecibo satellite dish onto hard drives. Because we're talking about hundreds of gigabytes worth of data, it is actually faster to ship the hard drives to Berkeley than it is to transfer the work across the internet itself.


Ironically enough, once the hard drives are received at SETI@Home's lab in the Space Sciences Laboratory in Berkeley, the raw data is then transferred to a secure offline storage via the internet. If at any time they need more data, they simply download the raw data files back into the lab to be split and processed as workunits.


I'd imagine it would be possible to connect a fat internet pipe directly from Arecibo to Berkeley, but somebody would have to pay for it. While people may be tired of hearing about it, the truth is SETI doesn't have that kind of money to be paying for big projects like that. I'm not really sure it's practical to get an internet connection from Arecibo to Berkeley when there's more pressing matters in need of attention.



I wonder if a dfs or a p2p setup would be a viable option so as to "buffer" the transfer of data instead of letting it pile up. Use your volunteers to pull the data to share the load and have them send it to Berkeley(or wherever) to be processed and then sent back to the volunteers for computation. A lot of turns in the road, sure, but before you scoff at the idea, name a better one than the possibility of losing all your data in an unforeseen transit accident.
ID: 1150443 · Report as offensive
OzzFan Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Apr 02
Posts: 15691
Credit: 84,761,841
RAC: 28
United States
Message 1150454 - Posted: 9 Sep 2011, 16:27:20 UTC - in response to Message 1150443.  

Excuse me for asking a STUPID question, but why isn't the data transferred through the internet directly to you at Berkeley? Sneaker net from Areceibo isn't the most efficient way of transferring data.

Please forgive. Meesa so stupid.


The data is recorded at the Arecibo satellite dish onto hard drives. Because we're talking about hundreds of gigabytes worth of data, it is actually faster to ship the hard drives to Berkeley than it is to transfer the work across the internet itself.


Ironically enough, once the hard drives are received at SETI@Home's lab in the Space Sciences Laboratory in Berkeley, the raw data is then transferred to a secure offline storage via the internet. If at any time they need more data, they simply download the raw data files back into the lab to be split and processed as workunits.


I'd imagine it would be possible to connect a fat internet pipe directly from Arecibo to Berkeley, but somebody would have to pay for it. While people may be tired of hearing about it, the truth is SETI doesn't have that kind of money to be paying for big projects like that. I'm not really sure it's practical to get an internet connection from Arecibo to Berkeley when there's more pressing matters in need of attention.



I wonder if a dfs or a p2p setup would be a viable option so as to "buffer" the transfer of data instead of letting it pile up. Use your volunteers to pull the data to share the load and have them send it to Berkeley(or wherever) to be processed and then sent back to the volunteers for computation. A lot of turns in the road, sure, but before you scoff at the idea, name a better one than the possibility of losing all your data in an unforeseen transit accident.


Two reasons why that's a bad idea:

1) It would be possible to "poison" the data, making it hard to prove the claim if an alien signal is found. There will be peer review to contend with. The data needs to be controlled a bit better.

2) The internet pipe that we all use to upload/download data is already saturated most of the time. Adding in your dfs or p2p idea would make the situation worse. As it is, the data is backed up into a private cloud using a separate internet connection than the one we use for our upload/downloads, so it doesn't have to fight for contention with user's traffic.


If I were a scientist on the project, I would trust data backups more than I would dfs or p2p.
ID: 1150454 · Report as offensive
Jeff Cobb Project Donor
Volunteer moderator
Project administrator
Project developer
Project scientist

Send message
Joined: 1 Mar 99
Posts: 122
Credit: 40,367
RAC: 0
United States
Message 1150494 - Posted: 9 Sep 2011, 18:34:25 UTC

Several data drives did arrive from Arecibo today. These files are now being moved into the pipeline.
ID: 1150494 · Report as offensive
TribbleRED

Send message
Joined: 9 May 10
Posts: 3
Credit: 3,385,328
RAC: 7
United States
Message 1150495 - Posted: 9 Sep 2011, 18:35:23 UTC - in response to Message 1150454.  
Last modified: 9 Sep 2011, 18:40:58 UTC

Excuse me for asking a STUPID question, but why isn't the data transferred through the internet directly to you at Berkeley? Sneaker net from Areceibo isn't the most efficient way of transferring data.

Please forgive. Meesa so stupid.


The data is recorded at the Arecibo satellite dish onto hard drives. Because we're talking about hundreds of gigabytes worth of data, it is actually faster to ship the hard drives to Berkeley than it is to transfer the work across the internet itself.


Ironically enough, once the hard drives are received at SETI@Home's lab in the Space Sciences Laboratory in Berkeley, the raw data is then transferred to a secure offline storage via the internet. If at any time they need more data, they simply download the raw data files back into the lab to be split and processed as workunits.


I'd imagine it would be possible to connect a fat internet pipe directly from Arecibo to Berkeley, but somebody would have to pay for it. While people may be tired of hearing about it, the truth is SETI doesn't have that kind of money to be paying for big projects like that. I'm not really sure it's practical to get an internet connection from Arecibo to Berkeley when there's more pressing matters in need of attention.



I wonder if a dfs or a p2p setup would be a viable option so as to "buffer" the transfer of data instead of letting it pile up. Use your volunteers to pull the data to share the load and have them send it to Berkeley(or wherever) to be processed and then sent back to the volunteers for computation. A lot of turns in the road, sure, but before you scoff at the idea, name a better one than the possibility of losing all your data in an unforeseen transit accident.


Two reasons why that's a bad idea:

1) It would be possible to "poison" the data, making it hard to prove the claim if an alien signal is found. There will be peer review to contend with. The data needs to be controlled a bit better.

2) The internet pipe that we all use to upload/download data is already saturated most of the time. Adding in your dfs or p2p idea would make the situation worse. As it is, the data is backed up into a private cloud using a separate internet connection than the one we use for our upload/downloads, so it doesn't have to fight for contention with user's traffic.


If I were a scientist on the project, I would trust data backups more than I would dfs or p2p.


Poison? lol. It doesn't work like that... And as far as the "saturated internet connection" for the volunteers is concerned: unless you are streaming pron 24/7 I am not sure what you are doing to saturate yours. Even if you are torrenting...

Look, may it be p2p or dfs, either is probably not the best of ideas regarding our current situation - nor one I would use for data retention on this sort of project... initially. Having said that, if at all possible I would be doing this before I start shipping my primary data physically... even down the street.

But by having some sort of stream out of any of the antennae to people in the project or the volunteers, it would be a step in the right direction to where the future of the world's corporations and companies are going.

http://www.fastcompany.com/1763349/20-years-of-data-storage-visualized
-and-
http://www.pcmag.com/article2/0,2817,2389276,00.asp

Money is said to be a hindrance... Sure, but I am not so sure the argument on the size of the pipe needed can be given till an accurate monthly amount of stored data is given. Bids for the bandwidth required to transmit said data over the course of a month will, in turn, follow. I would imagine that if looked at correctly, there would be the possibility of data transmission as a primary. Especially in today's world where the Mb is getting cheaper and cheaper via the demand.

All I know is that eventually all of it will be pushed (via dfs or equivalent) to a cloud or private data center. It's not going to happen tomorrow so the question posted here was asking, more simply, what can we do for the interim? We are the volunteers here..
ID: 1150495 · Report as offensive
OzzFan Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Apr 02
Posts: 15691
Credit: 84,761,841
RAC: 28
United States
Message 1150512 - Posted: 9 Sep 2011, 19:05:19 UTC - in response to Message 1150495.  

Poison? lol. It doesn't work like that...


So you're saying that if the raw data files are stored on user's PCs, that no malicious person out there would even think of modifying them?

And as far as the "saturated internet connection" for the volunteers is concerned: unless you are streaming pron 24/7 I am not sure what you are doing to saturate yours. Even if you are torrenting...


No, you misunderstand what I said. I said SETI@Home's connection is saturated, or is often too close to the saturation level to even contemplate storing raw files on user's PCs.

Look, may it be p2p or dfs, either is probably not the best of ideas regarding our current situation - nor one I would use for data retention on this sort of project... initially. Having said that, if at all possible I would be doing this before I start shipping my primary data physically... even down the street.

But by having some sort of stream out of any of the antennae to people in the project or the volunteers, it would be a step in the right direction to where the future of the world's corporations and companies are going.

http://www.fastcompany.com/1763349/20-years-of-data-storage-visualized
-and-
http://www.pcmag.com/article2/0,2817,2389276,00.asp

Money is said to be a hindrance... Sure, but I am not so sure the argument on the size of the pipe needed can be given till an accurate monthly amount of stored data is given. Bids for the bandwidth required to transmit said data over the course of a month will, in turn, follow. I would imagine that if looked at correctly, there would be the possibility of data transmission as a primary. Especially in today's world where the Mb is getting cheaper and cheaper via the demand.

All I know is that eventually all of it will be pushed (via dfs or equivalent) to a cloud or private data center. It's not going to happen tomorrow so the question posted here was asking, more simply, what can we do for the interim? We are the volunteers here..


When it comes to scientific work or data integrity, controls and environment are far more important than what the rest of the world is doing.

Remember, SETI@Home is not a business, so typical business ideas and technologies need not apply.
ID: 1150512 · Report as offensive
1 · 2 · 3 · 4 · Next

Message boards : News : more data on the way


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.