Bit Ceiling (Oct 08 2008)

Some nagging network issues, mostly due to the known liabilities/usual suspects. Very often we are maxing out our 100Mbit private connection to the world, due to peak periods (catchup after an outage, a spate of "fast" workunits, new client downloads added to the usual set of workunit downloads) or sending our raw data to offsite archival storage. This is why download/upload rates are abysmally slow at times - if you can get through at all.

One solution would be to increase our bandwidth - we do pay for a 1Gbit connection, but due to campus infrastructure can only use 100Mbit of that. Getting campus to improve this infrastructure is currently prohibitive due to cost (which includes new routers and dragging new cables up a mountainside to our lab), bureaucratic red tape, and the backlog of higher priority networking tasks campus wishes to tackle first. In other words as far as I can tell it ain't never gonna happen. Another solution would be to reduce our result redundancy, as already discussed in recent threads.

We also had our science db/raw data storage server choke a bit today - perhaps because of the recent swarm of fast workunits and therefore increased demand on the splitters. We do try to randomize the data to prevent such swarms but you can't win all of the time. And our web log rotating script sometimes barfs for one reason or another and fails to restart one server or another. For a moment there both the scheduler and upload server were off - I caught it fairly quickly though and restarted them.

To clarify, result uploads and workunit downloads go over the private SETI net, along with scheduler traffic. Web pages and other stuff goes over the campus net (it's not that much - only a couple Mbits/sec at peak times). The archival storage (where we copy all our raw data offsite) sometimes goes over the campus net, sometimes over the private SETI net, sometimes both if we need to empty the disks as fast as possible to return to Arecibo.

Other than all that.. I fixed the fonts of the status pages, and Jeff elsewhere posted a quick note about NTPCker progress.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

As always, thanks Matt.
____________

Keep up the great work, I know you guys do the best with what you have. That in mind, what is the cost difference between the 1Gbit and the 100Mbit? If you're paying for 1Gbit and not getting it, maybe you could shift some resources around to help in other areas.

Is there any spare cabling between the Seti download and upload servers and the hurricane interface?

i.e. would this be feasable;

No, the Hurricane interface is at the bottom of the hill, and the servers are at the top of the hill. There is only one available cable (actually, fibre) between the top and bottom of the hill, and it's that fibre which is the problem.

Sounds like a job for a SETI volunteer work-camp......

No, the Hurricane interface is at the bottom of the hill, and the servers are at the top of the hill. There is only one available cable (actually, fibre) between the top and bottom of the hill, and it's that fibre which is the problem.

Sounds like a job for a SETI volunteer work-camp......

In the spirit of Grid Computing, perhaps the Grid is not doing enough in this SETI model. The more nodes that come on to SETI the bigger the burden on the central core ... network pipe not big enough, too many validations required, indices required to sort out huge databases of information. Perhaps you can distribute some work done centrally to the Grid .. beyond the calculations. Also, are you compressing data before it hits the network. Perhaps you can put a server down the mountain that sits on the 1Gig interface and filter what goes back over the 100M up the Mountain. (my apologies if I am covering old ground).
Keep up the good work!
____________

No, the Hurricane interface is at the bottom of the hill, and the servers are at the top of the hill. There is only one available cable (actually, fibre) between the top and bottom of the hill, and it's that fibre which is the problem.

Sounds like a job for a SETI volunteer work-camp......

I was thinking about that yesterday, we just need to arrange enough local
volunteers with enough spades!! (or a digger)

I have no idea about the feasibility of this, but one way to avoid having to lay another cable between the lab and the Hurricane router might be to make the link wirelessly using WiFi or WiMAX. Some of this new technology claims to achieve speeds similar to fibre. The publicity they'd get, for being an integral part of the World's biggest computing project, might even persuade a supplier to set up and run a wireless link for "free".

Sounds like a job for a SETI volunteer work-camp......

First off, there is already a "help wanted" section of the forum which we set up a while ago - the problem is coming up with (a) problems that volunteers could effectively help with, and (b) time to manage potential solutions. We've been remiss in adding topics to that forum, mostly due to (b).

Secondly, we actually did have a microwave link way back in the day. Well, not us, but another project I worked for here at the lab which once had a NASA grant of sufficient worth to temporarily occupy a building off campus. So, strangely enough, we'd get bits up the hill to the space lab, and then transmit them through the air back down the hill to reach this other building. It was a 1Mbit link, roughly, if I remember correctly. This was circa 1995.

That said, our current private SETI ISP - Hurricane Electric (a.k.a "HE") - is not directly connected to anything on campus. In fact, the whole thing is a marriage of many parties. We connect to HE down at the PAIX (major exchange in Palo Alto). But to do so we needed space/access at the PAIX. This is where the generous support and donated routers (and rack space) from Bill Woodcock (of Packet Clearing House fame) comes in. He also walked me and Jeff through using IOS to get all this to connect properly. Anyway, bits go through the PAIX to CENIC, which is where they are tunneled (thanks to much help/support from central campus) over an extra 100Mbit cable up to our SETI router in our closet. That's the "simple version" of what's going on. Many parties are involved, all with varied hectic schedules and different priorities. Getting a better link from here to central campus is but one part of the puzzle.

(So yeah.. when people on campus running BOINC download a workunit from our servers, their requests go all the way down to Palo Alto first and then back to us.)

To clarify another point - we are paying a tiny fraction for the current 1Gbit than we were paying for the previous 100Mbit connection. The only real benefit of the latter was that, if I remember correctly, they had a connection to CENIC, so the previously mentioned chaos wasn't necessary (though there was still some minor chaos which we coordinated with campus).

And Finally! The problem with relocating parts of our infrastructure elsewhere is that it's all or nothing - every backend service is extremely tightly coupled with each other. The science database, the BOINC database, the upload/download servers and their respective storage, the splitter servers, the websites... etc. etc. etc. We're experiencing dropouts between servers that are in our lab and the servers in our closet across the hall (which talk over a single dedicated switch as opposed to through our lab infrastructure).

- Matt
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

... It was a 1Mbit link, roughly, if I remember correctly. This was circa 1995 ...