Bit Ceiling (Oct 08 2008)


log in

Advanced search

Message boards : Technical News : Bit Ceiling (Oct 08 2008)

1 · 2 · 3 · 4 · Next
Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 816015 - Posted: 8 Oct 2008, 22:01:36 UTC

Some nagging network issues, mostly due to the known liabilities/usual suspects. Very often we are maxing out our 100Mbit private connection to the world, due to peak periods (catchup after an outage, a spate of "fast" workunits, new client downloads added to the usual set of workunit downloads) or sending our raw data to offsite archival storage. This is why download/upload rates are abysmally slow at times - if you can get through at all.

One solution would be to increase our bandwidth - we do pay for a 1Gbit connection, but due to campus infrastructure can only use 100Mbit of that. Getting campus to improve this infrastructure is currently prohibitive due to cost (which includes new routers and dragging new cables up a mountainside to our lab), bureaucratic red tape, and the backlog of higher priority networking tasks campus wishes to tackle first. In other words as far as I can tell it ain't never gonna happen. Another solution would be to reduce our result redundancy, as already discussed in recent threads.

We also had our science db/raw data storage server choke a bit today - perhaps because of the recent swarm of fast workunits and therefore increased demand on the splitters. We do try to randomize the data to prevent such swarms but you can't win all of the time. And our web log rotating script sometimes barfs for one reason or another and fails to restart one server or another. For a moment there both the scheduler and upload server were off - I caught it fairly quickly though and restarted them.

To clarify, result uploads and workunit downloads go over the private SETI net, along with scheduler traffic. Web pages and other stuff goes over the campus net (it's not that much - only a couple Mbits/sec at peak times). The archival storage (where we copy all our raw data offsite) sometimes goes over the campus net, sometimes over the private SETI net, sometimes both if we need to empty the disks as fast as possible to return to Arecibo.

Other than all that.. I fixed the fonts of the status pages, and Jeff elsewhere posted a quick note about NTPCker progress.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Profile Blurf
Volunteer tester
Send message
Joined: 2 Sep 06
Posts: 7574
Credit: 6,968,658
RAC: 3,770
United States
Message 816024 - Posted: 8 Oct 2008, 22:21:15 UTC

As always, thanks Matt.
____________


Profile Dr. C.E.T.I.
Avatar
Send message
Joined: 29 Feb 00
Posts: 15993
Credit: 690,597
RAC: 0
United States
Message 816052 - Posted: 8 Oct 2008, 23:49:32 UTC


. . .Thank You for the Updates Matt (and also to Jeff for the NTPCker progress)


____________
BOINC Wiki . . .

Science Status Page . . .

Stephen
Send message
Joined: 1 Sep 06
Posts: 103
Credit: 11,155,194
RAC: 0
Message 816054 - Posted: 8 Oct 2008, 23:55:10 UTC

Keep up the great work, I know you guys do the best with what you have. That in mind, what is the cost difference between the 1Gbit and the 100Mbit? If you're paying for 1Gbit and not getting it, maybe you could shift some resources around to help in other areas.

OzzFan
Volunteer tester
Avatar
Send message
Joined: 9 Apr 02
Posts: 13625
Credit: 30,963,439
RAC: 20,635
United States
Message 816061 - Posted: 9 Oct 2008, 0:07:32 UTC - in response to Message 816054.

Keep up the great work, I know you guys do the best with what you have. That in mind, what is the cost difference between the 1Gbit and the 100Mbit? If you're paying for 1Gbit and not getting it, maybe you could shift some resources around to help in other areas.


As I remember, the Gigabit deal they signed up for is cheaper than the 100Mbit deal they had earlier.

Stephen
Send message
Joined: 1 Sep 06
Posts: 103
Credit: 11,155,194
RAC: 0
Message 816214 - Posted: 9 Oct 2008, 11:07:52 UTC - in response to Message 816061.

As I remember, the Gigabit deal they signed up for is cheaper than the 100Mbit deal they had earlier.


Thanks, I guess I missed out on that tidbit of info.

WinterKnight
Volunteer tester
Send message
Joined: 18 May 99
Posts: 8682
Credit: 24,887,781
RAC: 28,464
United Kingdom
Message 816216 - Posted: 9 Oct 2008, 11:20:12 UTC

Is there any spare cabling between the Seti download and upload servers and the hurricane interface?

i.e. would this be feasable;

download1_____100Mb____________ [ r ] [ o ] download2_____100Mb ___________ [ u ]______Hurricane_1Gb [ t ] Upload________100Mb ___________ [ e ] [ r ]

Richard HaselgroveProject donor
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8629
Credit: 51,402,244
RAC: 50,518
United Kingdom
Message 816218 - Posted: 9 Oct 2008, 11:23:40 UTC - in response to Message 816216.

No, the Hurricane interface is at the bottom of the hill, and the servers are at the top of the hill. There is only one available cable (actually, fibre) between the top and bottom of the hill, and it's that fibre which is the problem.

Sounds like a job for a SETI volunteer work-camp......

ClaggyProject donor
Volunteer tester
Send message
Joined: 5 Jul 99
Posts: 4139
Credit: 33,435,122
RAC: 20,045
United Kingdom
Message 816237 - Posted: 9 Oct 2008, 12:55:49 UTC - in response to Message 816218.

No, the Hurricane interface is at the bottom of the hill, and the servers are at the top of the hill. There is only one available cable (actually, fibre) between the top and bottom of the hill, and it's that fibre which is the problem.

Sounds like a job for a SETI volunteer work-camp......


I was thinking about that yesterday, we just need to arrange enough local
volunteers with enough spades!! (or a digger)

Claggy.
____________


vandehoef
Volunteer tester
Send message
Joined: 27 Dec 00
Posts: 1
Credit: 1,185,682
RAC: 4,050
Canada
Message 816239 - Posted: 9 Oct 2008, 13:00:32 UTC - in response to Message 816218.

In the spirit of Grid Computing, perhaps the Grid is not doing enough in this SETI model. The more nodes that come on to SETI the bigger the burden on the central core ... network pipe not big enough, too many validations required, indices required to sort out huge databases of information. Perhaps you can distribute some work done centrally to the Grid .. beyond the calculations. Also, are you compressing data before it hits the network. Perhaps you can put a server down the mountain that sits on the 1Gig interface and filter what goes back over the 100M up the Mountain. (my apologies if I am covering old ground).
Keep up the good work!
____________

Richard HaselgroveProject donor
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8629
Credit: 51,402,244
RAC: 50,518
United Kingdom
Message 816242 - Posted: 9 Oct 2008, 13:06:51 UTC - in response to Message 816237.

No, the Hurricane interface is at the bottom of the hill, and the servers are at the top of the hill. There is only one available cable (actually, fibre) between the top and bottom of the hill, and it's that fibre which is the problem.

Sounds like a job for a SETI volunteer work-camp......

I was thinking about that yesterday, we just need to arrange enough local
volunteers with enough spades!! (or a digger)

I don't see why it has to be restricted to local volunteers - I'm sure there are people from other parts of the States, even overseas, who'd be interested in a trip to visit the sharp end, and pay for their guided tour of the lab with a bit of labour. Excess baggage charges on the digger might be a bit of a problem, though.

Profile Jon Golding
Avatar
Send message
Joined: 20 Apr 00
Posts: 56
Credit: 365,460
RAC: 7
United Kingdom
Message 816250 - Posted: 9 Oct 2008, 13:52:33 UTC

I have no idea about the feasibility of this, but one way to avoid having to lay another cable between the lab and the Hurricane router might be to make the link wirelessly using WiFi or WiMAX. Some of this new technology claims to achieve speeds similar to fibre. The publicity they'd get, for being an integral part of the World's biggest computing project, might even persuade a supplier to set up and run a wireless link for "free".

____________

Profile speedimic
Volunteer tester
Avatar
Send message
Joined: 28 Sep 02
Posts: 362
Credit: 16,590,653
RAC: 0
Germany
Message 816288 - Posted: 9 Oct 2008, 15:57:00 UTC

Sounds like a job for a SETI volunteer work-camp......


I'm in.

____________
mic.


Profile Dr. C.E.T.I.
Avatar
Send message
Joined: 29 Feb 00
Posts: 15993
Credit: 690,597
RAC: 0
United States
Message 816290 - Posted: 9 Oct 2008, 16:14:57 UTC
Last modified: 9 Oct 2008, 16:15:11 UTC

.

. . . somebody should create a Special Forum for a 'call-outs' and Volunteers Needed Thread etc ,on these Boards

What say all - Eric K. (i know your busy ;) plenty of people want to Volunteer . . . so, . . .
____________
BOINC Wiki . . .

Science Status Page . . .

PhonAcq
Send message
Joined: 14 Apr 01
Posts: 1622
Credit: 22,351,467
RAC: 6,555
United States
Message 816302 - Posted: 9 Oct 2008, 16:42:13 UTC

I've advocated for a more grid/cloud-like environment before and am glad to see at least someone out there (vandehof) seems to have the same idea.

Before getting the picks, shovels, bureaucrats, and lawyers out to rebuild Berkeley, why not think about re-structuring the project a 'tiny' bit to permit another institution, geographically removed, to provide some of the backbone server capacity. Since more than 50% of the project credits derive from North America, but Europe has a strong participation, I suggest some place on the US East or Texas, obviously where the bandwidth is strong.

I'm not so simple as to say this would be 'easy', but sharing the science in a more advanced way has to be worth the effort. We could call it DT. (Distributed Thinking!)

Speaking of DT's, its time for my first beer of the day. Cheers.

Profile Dr. C.E.T.I.
Avatar
Send message
Joined: 29 Feb 00
Posts: 15993
Credit: 690,597
RAC: 0
United States
Message 816303 - Posted: 9 Oct 2008, 16:46:15 UTC - in response to Message 816302.

I've advocated for a more grid/cloud-like environment before and am glad to see at least someone out there (vandehof) seems to have the same idea.

Before getting the picks, shovels, bureaucrats, and lawyers out to rebuild Berkeley, why not think about re-structuring the project a 'tiny' bit to permit another institution, geographically removed, to provide some of the backbone server capacity. Since more than 50% of the project credits derive from North America, but Europe has a strong participation, I suggest some place on the US East or Texas, obviously where the bandwidth is strong.

I'm not so simple as to say this would be 'easy', but sharing the science in a more advanced way has to be worth the effort. We could call it DT. (Distributed Thinking!)

Speaking of DT's, its time for my first beer of the day. Cheers.


. . . somebody @ one of the North Carolina Universities (Raleigh etc) could implement a plan - strike a Proposal ?


____________
BOINC Wiki . . .

Science Status Page . . .

Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 816305 - Posted: 9 Oct 2008, 16:50:45 UTC - in response to Message 816290.

First off, there is already a "help wanted" section of the forum which we set up a while ago - the problem is coming up with (a) problems that volunteers could effectively help with, and (b) time to manage potential solutions. We've been remiss in adding topics to that forum, mostly due to (b).

Secondly, we actually did have a microwave link way back in the day. Well, not us, but another project I worked for here at the lab which once had a NASA grant of sufficient worth to temporarily occupy a building off campus. So, strangely enough, we'd get bits up the hill to the space lab, and then transmit them through the air back down the hill to reach this other building. It was a 1Mbit link, roughly, if I remember correctly. This was circa 1995.

That said, our current private SETI ISP - Hurricane Electric (a.k.a "HE") - is not directly connected to anything on campus. In fact, the whole thing is a marriage of many parties. We connect to HE down at the PAIX (major exchange in Palo Alto). But to do so we needed space/access at the PAIX. This is where the generous support and donated routers (and rack space) from Bill Woodcock (of Packet Clearing House fame) comes in. He also walked me and Jeff through using IOS to get all this to connect properly. Anyway, bits go through the PAIX to CENIC, which is where they are tunneled (thanks to much help/support from central campus) over an extra 100Mbit cable up to our SETI router in our closet. That's the "simple version" of what's going on. Many parties are involved, all with varied hectic schedules and different priorities. Getting a better link from here to central campus is but one part of the puzzle.

(So yeah.. when people on campus running BOINC download a workunit from our servers, their requests go all the way down to Palo Alto first and then back to us.)

To clarify another point - we are paying a tiny fraction for the current 1Gbit than we were paying for the previous 100Mbit connection. The only real benefit of the latter was that, if I remember correctly, they had a connection to CENIC, so the previously mentioned chaos wasn't necessary (though there was still some minor chaos which we coordinated with campus).

And Finally! The problem with relocating parts of our infrastructure elsewhere is that it's all or nothing - every backend service is extremely tightly coupled with each other. The science database, the BOINC database, the upload/download servers and their respective storage, the splitter servers, the websites... etc. etc. etc. We're experiencing dropouts between servers that are in our lab and the servers in our closet across the hall (which talk over a single dedicated switch as opposed to through our lab infrastructure).

- Matt
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Profile Dr. C.E.T.I.
Avatar
Send message
Joined: 29 Feb 00
Posts: 15993
Credit: 690,597
RAC: 0
United States
Message 816307 - Posted: 9 Oct 2008, 16:56:44 UTC


. . . Understood Matt - Thanks for the Clarification re:

And Finally! The problem with relocating parts of our infrastructure elsewhere is that it's all or nothing - every backend service is extremely tightly coupled with each other. The science database, the BOINC database, the upload/download servers and their respective storage, the splitter servers, the websites... etc. etc. etc. We're experiencing dropouts between servers that are in our lab and the servers in our closet across the hall (which talk over a single dedicated switch as opposed to through our lab infrastructure).



____________
BOINC Wiki . . .

Science Status Page . . .

Richard HaselgroveProject donor
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8629
Credit: 51,402,244
RAC: 50,518
United Kingdom
Message 816318 - Posted: 9 Oct 2008, 17:24:38 UTC - in response to Message 816305.

... It was a 1Mbit link, roughly, if I remember correctly. This was circa 1995 ...

So Moore's Law should get us up to a gigabit by about 2010? That sounds worth exploring - though I still like the sound of a volunteers' workcamp.

Thanks for the detailled explanation, Matt. I'm sure it will stimulate us all to get our thinking caps on and see if we can come up with a solution for you.

1 · 2 · 3 · 4 · Next

Message boards : Technical News : Bit Ceiling (Oct 08 2008)

Copyright © 2014 University of California