Tenaya (Feb 24 2009)

Message boards : Technical News : Tenaya (Feb 24 2009)
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · 4 · Next

AuthorMessage
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 869143 - Posted: 25 Feb 2009, 0:16:11 UTC

Had our weekly maintenance outage today, including the usual chores. I took the opportunity to replace a failed drive on one of our administrative file servers. I also issued the long-overdue final "shutdown" command on another administrative server, kang, which we no longer use. Many years ago, during the early days of SETI@home, several Sun representatives came by one day to discuss our progress. We thought it was just an informal touching-base kind of meeting, but they told us at the end they were going to donate a whole rack full of 6 state-of-the-art Sun servers and 2 disk arrays. Sun has always been nice to us, but this was completely unexpected. We eventually dubbed this the "k-rack" as we named every server after a sci-fi character starting with "k" (kang, kodos, kosh, klaatu, kryten, koloth). Well, kang, was the last one to go - the end of an era. We're still using the rack itself, though - very useful.

Network bandwidth woes continue, moreso now that we're coming out of the weekly outage. Lots of discussion about this in the previous thread - let me see if I can wrap up all the major points quickly. There are three potential solutions to our bandwidth limitations that we are actively entertaining/researching with the related parties. They are: 1. get a full 1Gbit link up to our server closet (pros: zero migration, cons: time/cost - about $80K in parts/labor), 2. collocation on campus (pros: minimal cost/migration, cons: almost impossible nuisance having to administer from a distance), 3. have a third-party entity host/administer everything (pros: we can ditch sysadmin for once and get back to work, cons: major cost, major migration). Each of these solutions requires a major amount of "getting ducks in row" (due to equipment policies, contract terms, general scheduling issues, etc.) - it's hardly just a money issue. Of course there are other options, too, like putting all efforts into final data analysis and shutting down SETI@home. One major issue is that our server closet (roughly 100 CPUs, 100 TB disk, 200 GB RAM) operates atomically - it's all or nothing. We can't just move one piece somewhere else. It's long and complicated - please don't make me explain why unless there's a free pitcher of beer involved.

- Matt

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 869143 · Report as offensive
C

Send message
Joined: 3 Apr 99
Posts: 240
Credit: 7,716,977
RAC: 0
United States
Message 869149 - Posted: 25 Feb 2009, 0:31:46 UTC - in response to Message 869143.  

...One major issue is that our server closet (roughly 100 CPUs, 100 TB disk, 200 GB RAM) operates atomically - it's all or nothing. We can't just move one piece somewhere else. It's long and complicated - please don't make me explain why unless there's a free pitcher of beer involved.

- Matt


Matt: If I'm ever in Berkeley, or you're ever in Northern Virginia, I'd be happy to buy a pitcher and listen to you...
C

Join Team MacNN
ID: 869149 · Report as offensive
Profile [KWSN]John Galt 007
Volunteer tester
Avatar

Send message
Joined: 9 Nov 99
Posts: 2444
Credit: 25,086,197
RAC: 0
United States
Message 869156 - Posted: 25 Feb 2009, 0:47:58 UTC

PM me your address and what type, and I will ship you what you want...beer, wine, etc. in an amount equal to a pitcher...

Now get to explaining...
Clk2HlpSetiCty:::PayIt4ward

ID: 869156 · Report as offensive
Wolverine
Avatar

Send message
Joined: 9 Jan 00
Posts: 35
Credit: 7,361,717
RAC: 0
Canada
Message 869177 - Posted: 25 Feb 2009, 1:57:30 UTC

Seeing the constant issues with bandwidth, I was wondering if the SETI data packets could be compressed (to reduce size for transfer), then BIONIC decompress for processing.

I know it would take more CPU time to compress the packets, but it would reduce bandwidth use.

Just an idea, thought I would share.

- Wol
ID: 869177 · Report as offensive
Profile Blurf
Volunteer tester

Send message
Joined: 2 Sep 06
Posts: 8962
Credit: 12,678,685
RAC: 0
United States
Message 869179 - Posted: 25 Feb 2009, 2:17:33 UTC

Matt-thanks for the news and thoughts. You and Eric (and Angela of course) are always welcome for a visit.


ID: 869179 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 869213 - Posted: 25 Feb 2009, 4:02:48 UTC - in response to Message 869177.  

Seeing the constant issues with bandwidth, I was wondering if the SETI data packets could be compressed (to reduce size for transfer), then BIONIC decompress for processing.

I know it would take more CPU time to compress the packets, but it would reduce bandwidth use.

Just an idea, thought I would share.

- Wol

File compression works based on the fact that the data in those files (word processing, databases, etc.) are not entirely random. A "flat file" database may compress 90% because one filler character appears over and over.

Common bytes get shorter codes, uncommon bytes longer, and the average number of bits/character goes down. (gross oversimplification)

Binary data consisting almost entirely of noise is going to be equally distributed across the whole range, so they aren't very compressable.
ID: 869213 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 869214 - Posted: 25 Feb 2009, 4:03:26 UTC - in response to Message 869143.  

It's long and complicated - please don't make me explain why unless there's a free pitcher of beer involved.

- Matt

I'll happily send beer without the explanation.
ID: 869214 · Report as offensive
PhonAcq

Send message
Joined: 14 Apr 01
Posts: 1656
Credit: 30,658,217
RAC: 1
United States
Message 869216 - Posted: 25 Feb 2009, 4:05:37 UTC

I'll be glad to help drink the beer.
ID: 869216 · Report as offensive
Profile Allie in Vancouver
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 3949
Credit: 1,604,668
RAC: 0
Canada
Message 869229 - Posted: 25 Feb 2009, 4:45:05 UTC

Free beer?

Just send it to me and I’ll make sure it gets down to Berkley.

Trust me.

Really. I will. I promise.


Pure mathematics is, in its way, the poetry of logical ideas.

Albert Einstein
ID: 869229 · Report as offensive
John McLeod VII
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jul 99
Posts: 24806
Credit: 790,712
RAC: 0
United States
Message 869233 - Posted: 25 Feb 2009, 4:56:43 UTC - in response to Message 869229.  

Free beer?

Just send it to me and I’ll make sure it gets down to Berkley.

Trust me.

Really. I will. I promise.


Minus your cut of course.


BOINC WIKI
ID: 869233 · Report as offensive
Profile yank Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 15 Aug 99
Posts: 522
Credit: 22,545,639
RAC: 0
United States
Message 869238 - Posted: 25 Feb 2009, 5:21:19 UTC

Free beer...tell me where and when and this old navy veteran will join you for whatever you want to give away. Darn I even will buy. GO ...GO...US Navy.
http://boinc.mundayweb.com/teamStats.php?userID=14824
ID: 869238 · Report as offensive
Profile [AF>France>Bourgogne]Patouchon
Avatar

Send message
Joined: 25 Aug 01
Posts: 7
Credit: 3,461,672
RAC: 4
France
Message 869256 - Posted: 25 Feb 2009, 7:29:02 UTC - in response to Message 869176.  

hello from France
i'm happy than the outage was issued, you did a great job.
infortunately i can't get SETI wus, i can only get Astropulse wus and they are too big to crunch with my old PC (about 650 h announced !!!).
do you think i'll can get some in a few hours ?
thanks for you patience
Patrick from "l'Alliance francophone" team.

seti1 was pretty good, seti2 will be better ?
ID: 869256 · Report as offensive
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 869257 - Posted: 25 Feb 2009, 7:29:21 UTC - in response to Message 869213.  

Seeing the constant issues with bandwidth, I was wondering if the SETI data packets could be compressed (to reduce size for transfer), then BIONIC decompress for processing.

I know it would take more CPU time to compress the packets, but it would reduce bandwidth use.

Just an idea, thought I would share.

- Wol

File compression works based on the fact that the data in those files (word processing, databases, etc.) are not entirely random. A "flat file" database may compress 90% because one filler character appears over and over.

Common bytes get shorter codes, uncommon bytes longer, and the average number of bits/character goes down. (gross oversimplification)

Binary data consisting almost entirely of noise is going to be equally distributed across the whole range, so they aren't very compressable.

To put some rough numbers on it: Astropulse WUs compress less than 2% because the data is sent as true binary and the xml header information is relatively short. S@H Enhanced WUs compress around 27% because the data is packed 6 bits per byte, there are line feeds inserted after each 64, and the xml header is larger. That's gzip compression, IINM the download servers can be configured to apply that and libCurl can then ungzip the received data. If so, the effect might be similar to adding nearly 15 Mbits/sec of download bandwidth (assuming vader and bane wouldn't choke doing the extra calculations).
                                                                 Joe
ID: 869257 · Report as offensive
Profile Hammeh
Volunteer tester
Avatar

Send message
Joined: 21 May 01
Posts: 135
Credit: 1,143,316
RAC: 0
United Kingdom
Message 869260 - Posted: 25 Feb 2009, 7:55:01 UTC

Of course there are other options, too, like putting all efforts into final data analysis and shutting down SETI@home.

No!! You can do that..
What else will my computers do all day every day? Other projects suck, this is by far the best one even with all the problems.
ID: 869260 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14644
Credit: 200,643,578
RAC: 874
United Kingdom
Message 869278 - Posted: 25 Feb 2009, 10:12:20 UTC - in response to Message 869274.  

Would it not be cheaper to lease a server closet, properly equiped ,
to handle the Upload/download function at your major local ISP.
Keep the 100 Mb line to the lab as control/backup.

How would the data get from Berkeley (where it is created in manageable chunks from the raw telescope recordings) to your ISP's server?

That's why Matt said

One major issue is that our server closet (roughly 100 CPUs, 100 TB disk, 200 GB RAM) operates atomically - it's all or nothing. We can't just move one piece somewhere else.

@ Matt,

We don't usually do pitchers in England, but I'd love to hear the stories - if you're ever in Yorkshire, these are waiting for you:

ID: 869278 · Report as offensive
Profile Paul_Tergeist
Avatar

Send message
Joined: 17 Sep 04
Posts: 8
Credit: 11,760,089
RAC: 0
United States
Message 869286 - Posted: 25 Feb 2009, 11:14:20 UTC - in response to Message 869143.  

One major issue is that our server closet (roughly 100 CPUs, 100 TB disk, 200 GB RAM) operates atomically - it's all or nothing. We can't just move one piece somewhere else.


Any possibility of teaming up with another university research group? That might reduce some of the hardware stress, but I don't know how feasible this might be. Just another option to toss out.

As always, please keep us posted on the situation.
<<Photo: ISS & Shuttle Discovery on STS-116 parting ways>>

The SETI@home distributed cluster is listening.
ID: 869286 · Report as offensive
Profile Jack Zhang
Volunteer tester
Avatar

Send message
Joined: 2 Jul 06
Posts: 206
Credit: 6,142,449
RAC: 0
Canada
Message 869294 - Posted: 25 Feb 2009, 11:54:30 UTC

I'd say the best out of the three solutions is the 1Gbit link up to the server closet.

The processing power of all of today's computers would be a reason why some WUs finish so fast. We need another array of radio telescopes to keep a constant WU flow to the super fast computers (like data from the Allen Telescope Array).
What if Fiction was Fact and Fact was Fiction and vice versa?
ID: 869294 · Report as offensive
Fred W
Volunteer tester

Send message
Joined: 13 Jun 99
Posts: 2524
Credit: 11,954,210
RAC: 0
United Kingdom
Message 869299 - Posted: 25 Feb 2009, 12:26:58 UTC - in response to Message 869278.  


We don't usually do pitchers in England, but I'd love to hear the stories - if you're ever in Yorkshire, these are waiting for you:


And a dyed-in-the-wool Lancastrian would risk crossing the border to double the contribution.

F.
ID: 869299 · Report as offensive
Claggy
Volunteer tester

Send message
Joined: 5 Jul 99
Posts: 4654
Credit: 47,537,079
RAC: 4
United Kingdom
Message 869311 - Posted: 25 Feb 2009, 13:38:38 UTC - in response to Message 869299.  


We don't usually do pitchers in England, but I'd love to hear the stories - if you're ever in Yorkshire, these are waiting for you:


And a dyed-in-the-wool Lancastrian would risk crossing the border to double the contribution.

F.


And I'll come across from Nottinghamshire to triple the contribution.

Claggy
ID: 869311 · Report as offensive
mick depear

Send message
Joined: 16 Oct 06
Posts: 15
Credit: 355,105
RAC: 0
United Kingdom
Message 869334 - Posted: 25 Feb 2009, 15:02:43 UTC - in response to Message 869311.  


We don't usually do pitchers in England, but I'd love to hear the stories - if you're ever in Yorkshire, these are waiting for you:


And a dyed-in-the-wool Lancastrian would risk crossing the border to double the contribution.

F.


And I'll come across from Nottinghamshire to triple the contribution.

Claggy


Add another 4 from Cambridgeshire :).

Mick
ID: 869334 · Report as offensive
1 · 2 · 3 · 4 · Next

Message boards : Technical News : Tenaya (Feb 24 2009)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.