Bit Ceiling (Oct 08 2008)

Message boards : Technical News : Bit Ceiling (Oct 08 2008)
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · 4 · Next

AuthorMessage
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 816015 - Posted: 8 Oct 2008, 22:01:36 UTC

Some nagging network issues, mostly due to the known liabilities/usual suspects. Very often we are maxing out our 100Mbit private connection to the world, due to peak periods (catchup after an outage, a spate of "fast" workunits, new client downloads added to the usual set of workunit downloads) or sending our raw data to offsite archival storage. This is why download/upload rates are abysmally slow at times - if you can get through at all.

One solution would be to increase our bandwidth - we do pay for a 1Gbit connection, but due to campus infrastructure can only use 100Mbit of that. Getting campus to improve this infrastructure is currently prohibitive due to cost (which includes new routers and dragging new cables up a mountainside to our lab), bureaucratic red tape, and the backlog of higher priority networking tasks campus wishes to tackle first. In other words as far as I can tell it ain't never gonna happen. Another solution would be to reduce our result redundancy, as already discussed in recent threads.

We also had our science db/raw data storage server choke a bit today - perhaps because of the recent swarm of fast workunits and therefore increased demand on the splitters. We do try to randomize the data to prevent such swarms but you can't win all of the time. And our web log rotating script sometimes barfs for one reason or another and fails to restart one server or another. For a moment there both the scheduler and upload server were off - I caught it fairly quickly though and restarted them.

To clarify, result uploads and workunit downloads go over the private SETI net, along with scheduler traffic. Web pages and other stuff goes over the campus net (it's not that much - only a couple Mbits/sec at peak times). The archival storage (where we copy all our raw data offsite) sometimes goes over the campus net, sometimes over the private SETI net, sometimes both if we need to empty the disks as fast as possible to return to Arecibo.

Other than all that.. I fixed the fonts of the status pages, and Jeff elsewhere posted a quick note about NTPCker progress.

- Matt

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 816015 · Report as offensive
Profile Blurf
Volunteer tester

Send message
Joined: 2 Sep 06
Posts: 8962
Credit: 12,678,685
RAC: 0
United States
Message 816024 - Posted: 8 Oct 2008, 22:21:15 UTC

As always, thanks Matt.


ID: 816024 · Report as offensive
Profile Dr. C.E.T.I.
Avatar

Send message
Joined: 29 Feb 00
Posts: 16019
Credit: 794,685
RAC: 0
United States
Message 816052 - Posted: 8 Oct 2008, 23:49:32 UTC


. . .Thank You for the Updates Matt (and also to Jeff for the NTPCker progress)


BOINC Wiki . . .

Science Status Page . . .
ID: 816052 · Report as offensive
Stephen

Send message
Joined: 1 Sep 06
Posts: 103
Credit: 11,155,194
RAC: 0
Message 816054 - Posted: 8 Oct 2008, 23:55:10 UTC

Keep up the great work, I know you guys do the best with what you have. That in mind, what is the cost difference between the 1Gbit and the 100Mbit? If you're paying for 1Gbit and not getting it, maybe you could shift some resources around to help in other areas.

ID: 816054 · Report as offensive
OzzFan Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Apr 02
Posts: 15691
Credit: 84,761,841
RAC: 28
United States
Message 816061 - Posted: 9 Oct 2008, 0:07:32 UTC - in response to Message 816054.  

Keep up the great work, I know you guys do the best with what you have. That in mind, what is the cost difference between the 1Gbit and the 100Mbit? If you're paying for 1Gbit and not getting it, maybe you could shift some resources around to help in other areas.


As I remember, the Gigabit deal they signed up for is cheaper than the 100Mbit deal they had earlier.
ID: 816061 · Report as offensive
Stephen

Send message
Joined: 1 Sep 06
Posts: 103
Credit: 11,155,194
RAC: 0
Message 816214 - Posted: 9 Oct 2008, 11:07:52 UTC - in response to Message 816061.  

As I remember, the Gigabit deal they signed up for is cheaper than the 100Mbit deal they had earlier.


Thanks, I guess I missed out on that tidbit of info.
ID: 816214 · Report as offensive
W-K 666 Project Donor
Volunteer tester

Send message
Joined: 18 May 99
Posts: 19013
Credit: 40,757,560
RAC: 67
United Kingdom
Message 816216 - Posted: 9 Oct 2008, 11:20:12 UTC

Is there any spare cabling between the Seti download and upload servers and the hurricane interface?

i.e. would this be feasable;
download1_____100Mb____________	[ r ]
				[ o ]
download2_____100Mb ___________	[ u ]______Hurricane_1Gb
				[ t ]
Upload________100Mb ___________	[ e ]
				[ r ]

ID: 816216 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 816218 - Posted: 9 Oct 2008, 11:23:40 UTC - in response to Message 816216.  

No, the Hurricane interface is at the bottom of the hill, and the servers are at the top of the hill. There is only one available cable (actually, fibre) between the top and bottom of the hill, and it's that fibre which is the problem.

Sounds like a job for a SETI volunteer work-camp......
ID: 816218 · Report as offensive
Claggy
Volunteer tester

Send message
Joined: 5 Jul 99
Posts: 4654
Credit: 47,537,079
RAC: 4
United Kingdom
Message 816237 - Posted: 9 Oct 2008, 12:55:49 UTC - in response to Message 816218.  

No, the Hurricane interface is at the bottom of the hill, and the servers are at the top of the hill. There is only one available cable (actually, fibre) between the top and bottom of the hill, and it's that fibre which is the problem.

Sounds like a job for a SETI volunteer work-camp......


I was thinking about that yesterday, we just need to arrange enough local
volunteers with enough spades!! (or a digger)

Claggy.
ID: 816237 · Report as offensive
vandehoef
Volunteer tester

Send message
Joined: 27 Dec 00
Posts: 1
Credit: 6,850,702
RAC: 2
Canada
Message 816239 - Posted: 9 Oct 2008, 13:00:32 UTC - in response to Message 816218.  

In the spirit of Grid Computing, perhaps the Grid is not doing enough in this SETI model. The more nodes that come on to SETI the bigger the burden on the central core ... network pipe not big enough, too many validations required, indices required to sort out huge databases of information. Perhaps you can distribute some work done centrally to the Grid .. beyond the calculations. Also, are you compressing data before it hits the network. Perhaps you can put a server down the mountain that sits on the 1Gig interface and filter what goes back over the 100M up the Mountain. (my apologies if I am covering old ground).
Keep up the good work!
ID: 816239 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 816242 - Posted: 9 Oct 2008, 13:06:51 UTC - in response to Message 816237.  

No, the Hurricane interface is at the bottom of the hill, and the servers are at the top of the hill. There is only one available cable (actually, fibre) between the top and bottom of the hill, and it's that fibre which is the problem.

Sounds like a job for a SETI volunteer work-camp......

I was thinking about that yesterday, we just need to arrange enough local
volunteers with enough spades!! (or a digger)

I don't see why it has to be restricted to local volunteers - I'm sure there are people from other parts of the States, even overseas, who'd be interested in a trip to visit the sharp end, and pay for their guided tour of the lab with a bit of labour. Excess baggage charges on the digger might be a bit of a problem, though.
ID: 816242 · Report as offensive
Profile Jon Golding
Avatar

Send message
Joined: 20 Apr 00
Posts: 105
Credit: 841,861
RAC: 0
United Kingdom
Message 816250 - Posted: 9 Oct 2008, 13:52:33 UTC

I have no idea about the feasibility of this, but one way to avoid having to lay another cable between the lab and the Hurricane router might be to make the link wirelessly using WiFi or WiMAX. Some of this new technology claims to achieve speeds similar to fibre. The publicity they'd get, for being an integral part of the World's biggest computing project, might even persuade a supplier to set up and run a wireless link for "free".

ID: 816250 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 816285 - Posted: 9 Oct 2008, 15:44:49 UTC - in response to Message 816250.  

I have no idea about the feasibility of this, but one way to avoid having to lay another cable between the lab and the Hurricane router might be to make the link wirelessly using WiFi or WiMAX. Some of this new technology claims to achieve speeds similar to fibre. The publicity they'd get, for being an integral part of the World's biggest computing project, might even persuade a supplier to set up and run a wireless link for "free".

Along the same lines........
I would think there would be wireless point-to-point links available that would cost far less than physically running new fibre up the mountain....
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 816285 · Report as offensive
Profile speedimic
Volunteer tester
Avatar

Send message
Joined: 28 Sep 02
Posts: 362
Credit: 16,590,653
RAC: 0
Germany
Message 816288 - Posted: 9 Oct 2008, 15:57:00 UTC

Sounds like a job for a SETI volunteer work-camp......


I'm in.

mic.


ID: 816288 · Report as offensive
Profile Dr. C.E.T.I.
Avatar

Send message
Joined: 29 Feb 00
Posts: 16019
Credit: 794,685
RAC: 0
United States
Message 816290 - Posted: 9 Oct 2008, 16:14:57 UTC
Last modified: 9 Oct 2008, 16:15:11 UTC

.

. . . somebody should create a Special Forum for a 'call-outs' and Volunteers Needed Thread etc ,on these Boards

What say all - Eric K. (i know your busy ;) plenty of people want to Volunteer . . . so, . . .
BOINC Wiki . . .

Science Status Page . . .
ID: 816290 · Report as offensive
PhonAcq

Send message
Joined: 14 Apr 01
Posts: 1656
Credit: 30,658,217
RAC: 1
United States
Message 816302 - Posted: 9 Oct 2008, 16:42:13 UTC

I've advocated for a more grid/cloud-like environment before and am glad to see at least someone out there (vandehof) seems to have the same idea.

Before getting the picks, shovels, bureaucrats, and lawyers out to rebuild Berkeley, why not think about re-structuring the project a 'tiny' bit to permit another institution, geographically removed, to provide some of the backbone server capacity. Since more than 50% of the project credits derive from North America, but Europe has a strong participation, I suggest some place on the US East or Texas, obviously where the bandwidth is strong.

I'm not so simple as to say this would be 'easy', but sharing the science in a more advanced way has to be worth the effort. We could call it DT. (Distributed Thinking!)

Speaking of DT's, its time for my first beer of the day. Cheers.
ID: 816302 · Report as offensive
Profile Dr. C.E.T.I.
Avatar

Send message
Joined: 29 Feb 00
Posts: 16019
Credit: 794,685
RAC: 0
United States
Message 816303 - Posted: 9 Oct 2008, 16:46:15 UTC - in response to Message 816302.  

I've advocated for a more grid/cloud-like environment before and am glad to see at least someone out there (vandehof) seems to have the same idea.

Before getting the picks, shovels, bureaucrats, and lawyers out to rebuild Berkeley, why not think about re-structuring the project a 'tiny' bit to permit another institution, geographically removed, to provide some of the backbone server capacity. Since more than 50% of the project credits derive from North America, but Europe has a strong participation, I suggest some place on the US East or Texas, obviously where the bandwidth is strong.

I'm not so simple as to say this would be 'easy', but sharing the science in a more advanced way has to be worth the effort. We could call it DT. (Distributed Thinking!)

Speaking of DT's, its time for my first beer of the day. Cheers.


. . . somebody @ one of the North Carolina Universities (Raleigh etc) could implement a plan - strike a Proposal ?


BOINC Wiki . . .

Science Status Page . . .
ID: 816303 · Report as offensive
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 816305 - Posted: 9 Oct 2008, 16:50:45 UTC - in response to Message 816290.  

First off, there is already a "help wanted" section of the forum which we set up a while ago - the problem is coming up with (a) problems that volunteers could effectively help with, and (b) time to manage potential solutions. We've been remiss in adding topics to that forum, mostly due to (b).

Secondly, we actually did have a microwave link way back in the day. Well, not us, but another project I worked for here at the lab which once had a NASA grant of sufficient worth to temporarily occupy a building off campus. So, strangely enough, we'd get bits up the hill to the space lab, and then transmit them through the air back down the hill to reach this other building. It was a 1Mbit link, roughly, if I remember correctly. This was circa 1995.

That said, our current private SETI ISP - Hurricane Electric (a.k.a "HE") - is not directly connected to anything on campus. In fact, the whole thing is a marriage of many parties. We connect to HE down at the PAIX (major exchange in Palo Alto). But to do so we needed space/access at the PAIX. This is where the generous support and donated routers (and rack space) from Bill Woodcock (of Packet Clearing House fame) comes in. He also walked me and Jeff through using IOS to get all this to connect properly. Anyway, bits go through the PAIX to CENIC, which is where they are tunneled (thanks to much help/support from central campus) over an extra 100Mbit cable up to our SETI router in our closet. That's the "simple version" of what's going on. Many parties are involved, all with varied hectic schedules and different priorities. Getting a better link from here to central campus is but one part of the puzzle.

(So yeah.. when people on campus running BOINC download a workunit from our servers, their requests go all the way down to Palo Alto first and then back to us.)

To clarify another point - we are paying a tiny fraction for the current 1Gbit than we were paying for the previous 100Mbit connection. The only real benefit of the latter was that, if I remember correctly, they had a connection to CENIC, so the previously mentioned chaos wasn't necessary (though there was still some minor chaos which we coordinated with campus).

And Finally! The problem with relocating parts of our infrastructure elsewhere is that it's all or nothing - every backend service is extremely tightly coupled with each other. The science database, the BOINC database, the upload/download servers and their respective storage, the splitter servers, the websites... etc. etc. etc. We're experiencing dropouts between servers that are in our lab and the servers in our closet across the hall (which talk over a single dedicated switch as opposed to through our lab infrastructure).

- Matt
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 816305 · Report as offensive
Profile Dr. C.E.T.I.
Avatar

Send message
Joined: 29 Feb 00
Posts: 16019
Credit: 794,685
RAC: 0
United States
Message 816307 - Posted: 9 Oct 2008, 16:56:44 UTC


. . . Understood Matt - Thanks for the Clarification re:

And Finally! The problem with relocating parts of our infrastructure elsewhere is that it's all or nothing - every backend service is extremely tightly coupled with each other. The science database, the BOINC database, the upload/download servers and their respective storage, the splitter servers, the websites... etc. etc. etc. We're experiencing dropouts between servers that are in our lab and the servers in our closet across the hall (which talk over a single dedicated switch as opposed to through our lab infrastructure).



BOINC Wiki . . .

Science Status Page . . .
ID: 816307 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 816318 - Posted: 9 Oct 2008, 17:24:38 UTC - in response to Message 816305.  

... It was a 1Mbit link, roughly, if I remember correctly. This was circa 1995 ...

So Moore's Law should get us up to a gigabit by about 2010? That sounds worth exploring - though I still like the sound of a volunteers' workcamp.

Thanks for the detailled explanation, Matt. I'm sure it will stimulate us all to get our thinking caps on and see if we can come up with a solution for you.
ID: 816318 · Report as offensive
1 · 2 · 3 · 4 · Next

Message boards : Technical News : Bit Ceiling (Oct 08 2008)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.