On Bandwidth (Jul 08 2009)

Once again it took the replica all night to recover. I started it up this morning, and it's catching up now. Well, almost. I'll turn the "show tasks/results" feature back on once it really starts catching up.

There's been a lot of discussion lately about our bandwidth woes. I actually talked to Blurf this morning on the phone regarding the (rather generous) push to donate money/hardware towards solving this problem. Let me try to paint a big picture here.

We pay for a gigabit of bandwidth from our private ISP (Hurricane Electric), but can only use 100 Mbits given current campus infrastructure. Most of campus is on gigabit already, but our lab is all the way up the hill - so it's much harder and more expensive to improve the old wiring/routing. The entire rest of the Space Lab uses about 10 Mbits/sec, so there is absolutely zero push by anybody else to spend money/effort on this project. Luckily, there was a spare 100 Mbit cable which is what we are using for the Hurricane Electric link.

While we pay for our bits, they still have to route through campus in order to ultimately hook up with the right backbones. That means we have to adhere to campus's network specs, which in turn means we can only use very specific brands/models of hardware, and can only act once they've fully researched our needs. We opened up a ticket months ago asking to start this research. We got word a couple days ago this research has more or less finally begun. Not much progress, but still non-zero. This may seem impossibly slow, but campus really pretty much always has much bigger fish to fry. Plus our requests usually present them with something new they haven't dealt with before, and therefore they are far more careful.

Ultimately, we should be presented with a couple options from campus which include exact pieces of hardware to be obtained. It's still not clear how much cable has to be upgraded and where, but we know we'll need two new routers, if not also other hardware. When campus gives us this final report, only then can we start figuring out how to obtain the necessary hardware.

As for other options, like going wireless... There actually used to be a building down in the flats that got wireless bandwidth from us. The experience was that it was quite slow and prone to suffering during bad weather, etc. This was a while ago, but still there is enough concern about reliability that nobody seems to want to go down this path.

Of course, another option is relocating our whole project down the hill (where gigabit links are readily available), or at least the server closet. Since the backend is quite complicated with many essential and nested dependencies it's all or nothing - we can't just move one server or functionality elsewhere - we'd have to move everything (this has been explained by me and others in countless other threads over the years). If we do end up moving (always a possibility) then all the above issues are moot.

Another important thing to consider is that we can always reduce are bandwidth demands via other means, which I also explained in another recent threads. Things like removing redundancy (and putting a cap on workunit downloads per day per host), or adding scientific analysis. Or, to be a little extreme, calling SETI@home done, turning off the downloads for good, and moving on to the next thing (something I am actually in favor of doing sooner than later, but the others around here seem to disagree).

I definitely appreciate past and current efforts to help us get beyond the current bandwidth crisis. However, as noted above, there are enough variables involved that I'd hate for you all to start collecting money directed towards a solution to a problem which might just go away. In the meantime, thanks as always for your patience (and crunching time when you actually do get workunits) - we'll keep working with what we got and see if we can't get beyond the storm sooner.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Thank you for the time for our discussion today, Matt. Kind of nice to put a voice to a set of posts.
____________

That was very informative and quite clear, Matt. Thanks for giving us the low-down on the progress of things for the bandwidth issue.
____________

Or, to be a little extreme, calling SETI@home done, turning off the downloads for good, and moving on to the next thing (something I am actually in favor of doing sooner than later, but the others around here seem to disagree).

I didn't understand this point; could someone elaborate?

... CallingSETI@home done, sounds as if you're saying that in your opinion all the works people are doing now is useless, or at least that you don't need the results any longer, for some reasons. ...

... And for those that can't think that one through...

Who knows?!

Keep searchin',
Martin


BTW: That's just one possible answer for both the science and for my postings.
____________
Mandriva Linux A user friendly OS!
See new freedom Mageia2
The Future is what We make IT (GPLv3)

... And...

Have we broken the limits yet for a monolithic database hosted on a single server?


Happy crunchin',
Martin

____________
Mandriva Linux A user friendly OS!
See new freedom Mageia2
The Future is what We make IT (GPLv3)

No no no I didn't mean that at all. I'm not saying you're wasting electricity. We still haven't fully covered the Arecibo sky yet with multibeam observations, for example.

What I am saying is we don't need to adhere to a 24/7/365 uptime policy because of the fear that we'll lose our participants forever. I have enough professional experience to prove that if you want to win customers (or fans) back after a long hiatus, you simply ask them nicely, and they return. In fact I've proven this time and time again with the reminder e-mails. Others feel we need to keep the workflow happening at all costs or we're ruined. What do you think?

Basically all I meant was that we can afford to take more long hiatuses to take care of long standing sub projects (like final data analysis?!), and perhaps come up with new/improved SETI@home-esque projects - all of which would be far easier to do if we were not as preoccupied with keeping data flow happening 100% of the time.

- Matt

Or, to be a little extreme, calling SETI@home done, turning off the downloads for good, and moving on to the next thing (something I am actually in favor of doing sooner than later, but the others around here seem to disagree).

Hmm, can you explain this in a little more details please? CallingSETI@home done, sounds as if you're saying that in your opinion all the works people are doing now is useless, or at least that you don't need the results any longer, for some reasons.

If that's the case please tell us rather sooner than later, so we can stop wasting electricity on something you really don't need.

Sten-Arne

No no no I didn't mean that at all...

What I am saying is we don't need to adhere to a 24/7/365 uptime policy because of the fear that we'll lose our participants forever. I have enough professional experience to prove that if you want to win customers (or fans) back after a long hiatus, you simply ask them nicely, and they return. ...

No no no I didn't mean that at all. I'm not saying you're wasting electricity. We still haven't fully covered the Arecibo sky yet with multibeam observations, for example.

What I am saying is we don't need to adhere to a 24/7/365 uptime policy because of the fear that we'll lose our participants forever. I have enough professional experience to prove that if you want to win customers (or fans) back after a long hiatus, you simply ask them nicely, and they return. In fact I've proven this time and time again with the reminder e-mails. Others feel we need to keep the workflow happening at all costs or we're ruined. What do you think?

Basically all I meant was that we can afford to take more long hiatuses to take care of long standing sub projects (like final data analysis?!), and perhaps come up with new/improved SETI@home-esque projects - all of which would be far easier to do if we were not as preoccupied with keeping data flow happening 100% of the time.

- Matt

What I am saying is we don't need to adhere to a 24/7/365 uptime policy because of the fear that we'll lose our participants forever.

i agree with matt.... until fresh data(and un-noisy) is here... i say that the servers and system could use a break and it would give time for yall to get some back-end duties done instead of babysitting.
____________

I recommend Secunia PSI: http://secunia.com/vulnerability_scanning/personal/
Go Georgia Tech.

i agree with matt.... until fresh data(and un-noisy) is here... i say that the servers and system could use a break and it would give time for yall to get some back-end duties done instead of babysitting.

But isn't a bog standard telephone line capable of up to 10Gbits/sec

But isn't a bog standard telephone line capable of up to 10Gbits/sec?