Tenaya (Feb 24 2009)


log in

Advanced search

Message boards : Technical News : Tenaya (Feb 24 2009)

1 · 2 · 3 · 4 · Next
Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 869143 - Posted: 25 Feb 2009, 0:16:11 UTC

Had our weekly maintenance outage today, including the usual chores. I took the opportunity to replace a failed drive on one of our administrative file servers. I also issued the long-overdue final "shutdown" command on another administrative server, kang, which we no longer use. Many years ago, during the early days of SETI@home, several Sun representatives came by one day to discuss our progress. We thought it was just an informal touching-base kind of meeting, but they told us at the end they were going to donate a whole rack full of 6 state-of-the-art Sun servers and 2 disk arrays. Sun has always been nice to us, but this was completely unexpected. We eventually dubbed this the "k-rack" as we named every server after a sci-fi character starting with "k" (kang, kodos, kosh, klaatu, kryten, koloth). Well, kang, was the last one to go - the end of an era. We're still using the rack itself, though - very useful.

Network bandwidth woes continue, moreso now that we're coming out of the weekly outage. Lots of discussion about this in the previous thread - let me see if I can wrap up all the major points quickly. There are three potential solutions to our bandwidth limitations that we are actively entertaining/researching with the related parties. They are: 1. get a full 1Gbit link up to our server closet (pros: zero migration, cons: time/cost - about $80K in parts/labor), 2. collocation on campus (pros: minimal cost/migration, cons: almost impossible nuisance having to administer from a distance), 3. have a third-party entity host/administer everything (pros: we can ditch sysadmin for once and get back to work, cons: major cost, major migration). Each of these solutions requires a major amount of "getting ducks in row" (due to equipment policies, contract terms, general scheduling issues, etc.) - it's hardly just a money issue. Of course there are other options, too, like putting all efforts into final data analysis and shutting down SETI@home. One major issue is that our server closet (roughly 100 CPUs, 100 TB disk, 200 GB RAM) operates atomically - it's all or nothing. We can't just move one piece somewhere else. It's long and complicated - please don't make me explain why unless there's a free pitcher of beer involved.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

C
Send message
Joined: 3 Apr 99
Posts: 240
Credit: 6,607,588
RAC: 827
United States
Message 869149 - Posted: 25 Feb 2009, 0:31:46 UTC - in response to Message 869143.

...One major issue is that our server closet (roughly 100 CPUs, 100 TB disk, 200 GB RAM) operates atomically - it's all or nothing. We can't just move one piece somewhere else. It's long and complicated - please don't make me explain why unless there's a free pitcher of beer involved.

- Matt


Matt: If I'm ever in Berkeley, or you're ever in Northern Virginia, I'd be happy to buy a pitcher and listen to you...
C
____________

Join Team MacNN

Profile [KWSN]John Galt 007
Volunteer tester
Avatar
Send message
Joined: 9 Nov 99
Posts: 2441
Credit: 18,348,694
RAC: 62
United States
Message 869156 - Posted: 25 Feb 2009, 0:47:58 UTC

PM me your address and what type, and I will ship you what you want...beer, wine, etc. in an amount equal to a pitcher...

Now get to explaining...
____________
Clk2HlpSetiCty:::PayIt4ward

Wolverine
Avatar
Send message
Joined: 9 Jan 00
Posts: 35
Credit: 7,306,704
RAC: 1,394
Canada
Message 869177 - Posted: 25 Feb 2009, 1:57:30 UTC

Seeing the constant issues with bandwidth, I was wondering if the SETI data packets could be compressed (to reduce size for transfer), then BIONIC decompress for processing.

I know it would take more CPU time to compress the packets, but it would reduce bandwidth use.

Just an idea, thought I would share.

- Wol
____________

Profile Blurf
Volunteer tester
Send message
Joined: 2 Sep 06
Posts: 7411
Credit: 6,484,220
RAC: 4,515
United States
Message 869179 - Posted: 25 Feb 2009, 2:17:33 UTC

Matt-thanks for the news and thoughts. You and Eric (and Angela of course) are always welcome for a visit.
____________


1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 869213 - Posted: 25 Feb 2009, 4:02:48 UTC - in response to Message 869177.

Seeing the constant issues with bandwidth, I was wondering if the SETI data packets could be compressed (to reduce size for transfer), then BIONIC decompress for processing.

I know it would take more CPU time to compress the packets, but it would reduce bandwidth use.

Just an idea, thought I would share.

- Wol

File compression works based on the fact that the data in those files (word processing, databases, etc.) are not entirely random. A "flat file" database may compress 90% because one filler character appears over and over.

Common bytes get shorter codes, uncommon bytes longer, and the average number of bits/character goes down. (gross oversimplification)

Binary data consisting almost entirely of noise is going to be equally distributed across the whole range, so they aren't very compressable.
____________

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 869214 - Posted: 25 Feb 2009, 4:03:26 UTC - in response to Message 869143.

It's long and complicated - please don't make me explain why unless there's a free pitcher of beer involved.

- Matt

I'll happily send beer without the explanation.
____________

PhonAcq
Send message
Joined: 14 Apr 01
Posts: 1622
Credit: 21,964,629
RAC: 3,938
United States
Message 869216 - Posted: 25 Feb 2009, 4:05:37 UTC

I'll be glad to help drink the beer.

Profile Allie in Vancouver
Volunteer tester
Avatar
Send message
Joined: 16 Mar 07
Posts: 3949
Credit: 1,604,668
RAC: 0
Canada
Message 869229 - Posted: 25 Feb 2009, 4:45:05 UTC

Free beer?

Just send it to me and I’ll make sure it gets down to Berkley.

Trust me.

Really. I will. I promise.


____________
Pure mathematics is, in its way, the poetry of logical ideas.

Albert Einstein

John McLeod VII
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 15 Jul 99
Posts: 24090
Credit: 517,737
RAC: 158
United States
Message 869233 - Posted: 25 Feb 2009, 4:56:43 UTC - in response to Message 869229.

Free beer?

Just send it to me and I’ll make sure it gets down to Berkley.

Trust me.

Really. I will. I promise.


Minus your cut of course.
____________


BOINC WIKI

Profile yank
Volunteer tester
Avatar
Send message
Joined: 15 Aug 99
Posts: 510
Credit: 12,629,769
RAC: 2
United States
Message 869238 - Posted: 25 Feb 2009, 5:21:19 UTC

Free beer...tell me where and when and this old navy veteran will join you for whatever you want to give away. Darn I even will buy. GO ...GO...US Navy.
____________

Profile [AF>France>Bourgogne]Patouchon
Avatar
Send message
Joined: 25 Aug 01
Posts: 7
Credit: 2,023,082
RAC: 362
France
Message 869256 - Posted: 25 Feb 2009, 7:29:02 UTC - in response to Message 869176.

hello from France
i'm happy than the outage was issued, you did a great job.
infortunately i can't get SETI wus, i can only get Astropulse wus and they are too big to crunch with my old PC (about 650 h announced !!!).
do you think i'll can get some in a few hours ?
thanks for you patience
Patrick from "l'Alliance francophone" team.

____________
seti1 was pretty good, seti2 will be better ?

Josef W. Segur
Volunteer developer
Volunteer tester
Send message
Joined: 30 Oct 99
Posts: 4203
Credit: 1,030,189
RAC: 263
United States
Message 869257 - Posted: 25 Feb 2009, 7:29:21 UTC - in response to Message 869213.

Seeing the constant issues with bandwidth, I was wondering if the SETI data packets could be compressed (to reduce size for transfer), then BIONIC decompress for processing.

I know it would take more CPU time to compress the packets, but it would reduce bandwidth use.

Just an idea, thought I would share.

- Wol

File compression works based on the fact that the data in those files (word processing, databases, etc.) are not entirely random. A "flat file" database may compress 90% because one filler character appears over and over.

Common bytes get shorter codes, uncommon bytes longer, and the average number of bits/character goes down. (gross oversimplification)

Binary data consisting almost entirely of noise is going to be equally distributed across the whole range, so they aren't very compressable.

To put some rough numbers on it: Astropulse WUs compress less than 2% because the data is sent as true binary and the xml header information is relatively short. S@H Enhanced WUs compress around 27% because the data is packed 6 bits per byte, there are line feeds inserted after each 64, and the xml header is larger. That's gzip compression, IINM the download servers can be configured to apply that and libCurl can then ungzip the received data. If so, the effect might be similar to adding nearly 15 Mbits/sec of download bandwidth (assuming vader and bane wouldn't choke doing the extra calculations).
Joe

Profile Hammeh
Volunteer tester
Avatar
Send message
Joined: 21 May 01
Posts: 135
Credit: 1,143,316
RAC: 0
United Kingdom
Message 869260 - Posted: 25 Feb 2009, 7:55:01 UTC

Of course there are other options, too, like putting all efforts into final data analysis and shutting down SETI@home.

No!! You can do that..
What else will my computers do all day every day? Other projects suck, this is by far the best one even with all the problems.
____________

Richard Haselgrove
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8375
Credit: 46,690,623
RAC: 19,738
United Kingdom
Message 869278 - Posted: 25 Feb 2009, 10:12:20 UTC - in response to Message 869274.

Would it not be cheaper to lease a server closet, properly equiped ,
to handle the Upload/download function at your major local ISP.
Keep the 100 Mb line to the lab as control/backup.

How would the data get from Berkeley (where it is created in manageable chunks from the raw telescope recordings) to your ISP's server?

That's why Matt said

One major issue is that our server closet (roughly 100 CPUs, 100 TB disk, 200 GB RAM) operates atomically - it's all or nothing. We can't just move one piece somewhere else.

@ Matt,

We don't usually do pitchers in England, but I'd love to hear the stories - if you're ever in Yorkshire, these are waiting for you:

Profile Paul_Tergeist
Avatar
Send message
Joined: 17 Sep 04
Posts: 8
Credit: 11,760,089
RAC: 0
United States
Message 869286 - Posted: 25 Feb 2009, 11:14:20 UTC - in response to Message 869143.

One major issue is that our server closet (roughly 100 CPUs, 100 TB disk, 200 GB RAM) operates atomically - it's all or nothing. We can't just move one piece somewhere else.


Any possibility of teaming up with another university research group? That might reduce some of the hardware stress, but I don't know how feasible this might be. Just another option to toss out.

As always, please keep us posted on the situation.
____________
<<Photo: ISS & Shuttle Discovery on STS-116 parting ways>>

The SETI@home distributed cluster is listening.

Profile Jack Zhang
Volunteer tester
Avatar
Send message
Joined: 2 Jul 06
Posts: 206
Credit: 6,030,745
RAC: 887
Canada
Message 869294 - Posted: 25 Feb 2009, 11:54:30 UTC

I'd say the best out of the three solutions is the 1Gbit link up to the server closet.

The processing power of all of today's computers would be a reason why some WUs finish so fast. We need another array of radio telescopes to keep a constant WU flow to the super fast computers (like data from the Allen Telescope Array).
____________
What if Fiction was Fact and Fact was Fiction and vice versa?

Fred W
Volunteer tester
Send message
Joined: 13 Jun 99
Posts: 2524
Credit: 11,954,210
RAC: 0
United Kingdom
Message 869299 - Posted: 25 Feb 2009, 12:26:58 UTC - in response to Message 869278.


We don't usually do pitchers in England, but I'd love to hear the stories - if you're ever in Yorkshire, these are waiting for you:


And a dyed-in-the-wool Lancastrian would risk crossing the border to double the contribution.

F.
____________

Claggy
Volunteer tester
Send message
Joined: 5 Jul 99
Posts: 4048
Credit: 32,693,129
RAC: 604
United Kingdom
Message 869311 - Posted: 25 Feb 2009, 13:38:38 UTC - in response to Message 869299.


We don't usually do pitchers in England, but I'd love to hear the stories - if you're ever in Yorkshire, these are waiting for you:


And a dyed-in-the-wool Lancastrian would risk crossing the border to double the contribution.

F.


And I'll come across from Nottinghamshire to triple the contribution.

Claggy
____________


mick depear
Send message
Joined: 16 Oct 06
Posts: 15
Credit: 355,105
RAC: 0
United Kingdom
Message 869334 - Posted: 25 Feb 2009, 15:02:43 UTC - in response to Message 869311.


We don't usually do pitchers in England, but I'd love to hear the stories - if you're ever in Yorkshire, these are waiting for you:


And a dyed-in-the-wool Lancastrian would risk crossing the border to double the contribution.

F.


And I'll come across from Nottinghamshire to triple the contribution.

Claggy


Add another 4 from Cambridgeshire :).

Mick

1 · 2 · 3 · 4 · Next

Message boards : Technical News : Tenaya (Feb 24 2009)

Copyright © 2014 University of California