Message boards :
Technical News :
Tenaya (Feb 24 2009)
Message board moderation
Author | Message |
---|---|
![]() ![]() Send message Joined: 1 Mar 99 Posts: 1444 Credit: 957,058 RAC: 0 ![]() |
Had our weekly maintenance outage today, including the usual chores. I took the opportunity to replace a failed drive on one of our administrative file servers. I also issued the long-overdue final "shutdown" command on another administrative server, kang, which we no longer use. Many years ago, during the early days of SETI@home, several Sun representatives came by one day to discuss our progress. We thought it was just an informal touching-base kind of meeting, but they told us at the end they were going to donate a whole rack full of 6 state-of-the-art Sun servers and 2 disk arrays. Sun has always been nice to us, but this was completely unexpected. We eventually dubbed this the "k-rack" as we named every server after a sci-fi character starting with "k" (kang, kodos, kosh, klaatu, kryten, koloth). Well, kang, was the last one to go - the end of an era. We're still using the rack itself, though - very useful. Network bandwidth woes continue, moreso now that we're coming out of the weekly outage. Lots of discussion about this in the previous thread - let me see if I can wrap up all the major points quickly. There are three potential solutions to our bandwidth limitations that we are actively entertaining/researching with the related parties. They are: 1. get a full 1Gbit link up to our server closet (pros: zero migration, cons: time/cost - about $80K in parts/labor), 2. collocation on campus (pros: minimal cost/migration, cons: almost impossible nuisance having to administer from a distance), 3. have a third-party entity host/administer everything (pros: we can ditch sysadmin for once and get back to work, cons: major cost, major migration). Each of these solutions requires a major amount of "getting ducks in row" (due to equipment policies, contract terms, general scheduling issues, etc.) - it's hardly just a money issue. Of course there are other options, too, like putting all efforts into final data analysis and shutting down SETI@home. One major issue is that our server closet (roughly 100 CPUs, 100 TB disk, 200 GB RAM) operates atomically - it's all or nothing. We can't just move one piece somewhere else. It's long and complicated - please don't make me explain why unless there's a free pitcher of beer involved. - Matt -- BOINC/SETI@home network/web/science/development person -- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude |
C Send message Joined: 3 Apr 99 Posts: 240 Credit: 7,716,977 RAC: 0 ![]() |
...One major issue is that our server closet (roughly 100 CPUs, 100 TB disk, 200 GB RAM) operates atomically - it's all or nothing. We can't just move one piece somewhere else. It's long and complicated - please don't make me explain why unless there's a free pitcher of beer involved. Matt: If I'm ever in Berkeley, or you're ever in Northern Virginia, I'd be happy to buy a pitcher and listen to you... C Join Team MacNN |
![]() ![]() Send message Joined: 9 Nov 99 Posts: 2444 Credit: 25,086,197 RAC: 0 ![]() |
PM me your address and what type, and I will ship you what you want...beer, wine, etc. in an amount equal to a pitcher... Now get to explaining... Clk2HlpSetiCty:::PayIt4ward ![]() |
Wolverine ![]() Send message Joined: 9 Jan 00 Posts: 35 Credit: 7,361,717 RAC: 0 ![]() |
Seeing the constant issues with bandwidth, I was wondering if the SETI data packets could be compressed (to reduce size for transfer), then BIONIC decompress for processing. I know it would take more CPU time to compress the packets, but it would reduce bandwidth use. Just an idea, thought I would share. - Wol ![]() |
![]() Send message Joined: 2 Sep 06 Posts: 8964 Credit: 12,678,685 RAC: 0 ![]() |
Matt-thanks for the news and thoughts. You and Eric (and Angela of course) are always welcome for a visit. ![]() ![]() |
1mp0£173 Send message Joined: 3 Apr 99 Posts: 8423 Credit: 356,897 RAC: 0 ![]() |
Seeing the constant issues with bandwidth, I was wondering if the SETI data packets could be compressed (to reduce size for transfer), then BIONIC decompress for processing. File compression works based on the fact that the data in those files (word processing, databases, etc.) are not entirely random. A "flat file" database may compress 90% because one filler character appears over and over. Common bytes get shorter codes, uncommon bytes longer, and the average number of bits/character goes down. (gross oversimplification) Binary data consisting almost entirely of noise is going to be equally distributed across the whole range, so they aren't very compressable. |
1mp0£173 Send message Joined: 3 Apr 99 Posts: 8423 Credit: 356,897 RAC: 0 ![]() |
It's long and complicated - please don't make me explain why unless there's a free pitcher of beer involved. I'll happily send beer without the explanation. |
PhonAcq Send message Joined: 14 Apr 01 Posts: 1656 Credit: 30,658,217 RAC: 1 ![]() |
I'll be glad to help drink the beer. |
![]() ![]() Send message Joined: 16 Mar 07 Posts: 3949 Credit: 1,604,668 RAC: 0 ![]() |
Free beer? Just send it to me and I’ll make sure it gets down to Berkley. Trust me. Really. I will. I promise. ![]() Pure mathematics is, in its way, the poetry of logical ideas. Albert Einstein |
John McLeod VII Send message Joined: 15 Jul 99 Posts: 24806 Credit: 790,712 RAC: 0 ![]() |
Free beer? Minus your cut of course. ![]() ![]() BOINC WIKI |
![]() ![]() ![]() ![]() ![]() Send message Joined: 15 Aug 99 Posts: 522 Credit: 22,545,639 RAC: 0 ![]() |
Free beer...tell me where and when and this old navy veteran will join you for whatever you want to give away. Darn I even will buy. GO ...GO...US Navy. http://boinc.mundayweb.com/teamStats.php?userID=14824 |
![]() ![]() Send message Joined: 25 Aug 01 Posts: 7 Credit: 3,461,672 RAC: 4 ![]() |
hello from France i'm happy than the outage was issued, you did a great job. infortunately i can't get SETI wus, i can only get Astropulse wus and they are too big to crunch with my old PC (about 650 h announced !!!). do you think i'll can get some in a few hours ? thanks for you patience Patrick from "l'Alliance francophone" team. seti1 was pretty good, seti2 will be better ? ![]() |
Josef W. Segur Send message Joined: 30 Oct 99 Posts: 4504 Credit: 1,414,761 RAC: 0 ![]() |
Seeing the constant issues with bandwidth, I was wondering if the SETI data packets could be compressed (to reduce size for transfer), then BIONIC decompress for processing. To put some rough numbers on it: Astropulse WUs compress less than 2% because the data is sent as true binary and the xml header information is relatively short. S@H Enhanced WUs compress around 27% because the data is packed 6 bits per byte, there are line feeds inserted after each 64, and the xml header is larger. That's gzip compression, IINM the download servers can be configured to apply that and libCurl can then ungzip the received data. If so, the effect might be similar to adding nearly 15 Mbits/sec of download bandwidth (assuming vader and bane wouldn't choke doing the extra calculations). Joe |
![]() ![]() Send message Joined: 21 May 01 Posts: 135 Credit: 1,143,316 RAC: 0 ![]() |
Of course there are other options, too, like putting all efforts into final data analysis and shutting down SETI@home. No!! You can do that.. What else will my computers do all day every day? Other projects suck, this is by far the best one even with all the problems. |
Richard Haselgrove ![]() Send message Joined: 4 Jul 99 Posts: 14686 Credit: 200,643,578 RAC: 874 ![]() ![]() |
Would it not be cheaper to lease a server closet, properly equiped , How would the data get from Berkeley (where it is created in manageable chunks from the raw telescope recordings) to your ISP's server? That's why Matt said One major issue is that our server closet (roughly 100 CPUs, 100 TB disk, 200 GB RAM) operates atomically - it's all or nothing. We can't just move one piece somewhere else. @ Matt, We don't usually do pitchers in England, but I'd love to hear the stories - if you're ever in Yorkshire, these are waiting for you: ![]() |
![]() ![]() Send message Joined: 17 Sep 04 Posts: 8 Credit: 11,760,089 RAC: 0 ![]() |
One major issue is that our server closet (roughly 100 CPUs, 100 TB disk, 200 GB RAM) operates atomically - it's all or nothing. We can't just move one piece somewhere else. Any possibility of teaming up with another university research group? That might reduce some of the hardware stress, but I don't know how feasible this might be. Just another option to toss out. As always, please keep us posted on the situation. <<Photo: ISS & Shuttle Discovery on STS-116 parting ways>> The SETI@home distributed cluster is listening. |
![]() ![]() Send message Joined: 2 Jul 06 Posts: 206 Credit: 6,142,449 RAC: 0 ![]() |
I'd say the best out of the three solutions is the 1Gbit link up to the server closet. The processing power of all of today's computers would be a reason why some WUs finish so fast. We need another array of radio telescopes to keep a constant WU flow to the super fast computers (like data from the Allen Telescope Array). What if Fiction was Fact and Fact was Fiction and vice versa? |
Fred W Send message Joined: 13 Jun 99 Posts: 2524 Credit: 11,954,210 RAC: 0 ![]() |
And a dyed-in-the-wool Lancastrian would risk crossing the border to double the contribution. F. ![]() |
Claggy Send message Joined: 5 Jul 99 Posts: 4654 Credit: 47,537,079 RAC: 4 ![]() |
And I'll come across from Nottinghamshire to triple the contribution. Claggy ![]() |
mick depear Send message Joined: 16 Oct 06 Posts: 15 Credit: 355,105 RAC: 0 ![]() |
Add another 4 from Cambridgeshire :). Mick |
©2025 University of California
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.