Working as Expected (Jul 13 2009)

Gigabit is one solution, but looking at the progress rate of SETI it is only temporary one - S@H will stand in front of the hardware problem in the future - I think that the better way is to distribute the backend of the project on many sites preferably distributed across the globe.

Many people suggested putting multiple servers around the world for clients to download from, so the bandwidth is distributed. But how do you get the data to those servers in the first place? It would have to go from the servers at Berkeley to the distributed download servers through the current 100mbit pipe. I don't see how that lowers bandwidth at all.

Same for upload. Yes, you can put multiple upload servers around the world, but eventually, somehow, the data has to go back to setiathome.berkeley.edu.

Also I don't believe the project scientists would like to lose direct control of the project data. It's like a "chain of custody" in an investigation. If you can't account for the security of the data throughout the process then questions will arise if something of some importance is claimed at a later date.

Same for upload. Yes, you can put multiple upload servers around the world, but eventually, somehow, the data has to go back to setiathome.berkeley.edu.

This may seem silly, but what would the bandwidth of a couple of large HDDs or tapes (say, terabyte or larger) be if sent by courier or post from various other sites back to berkeley? It would get around the issue of bandwidth consumption, and the data could be stripped off the drive and confirmed before it was wiped from the satellite systems doing the actual receiving... Low-tech I know, and there would be the delay of receiving and loading the data, but its a thought...

Would an set of remote upload servers as "data aggregators" work?

Receives many small files, handles many TCP connections: then, once an hour or so, zips them all up into one big compressed file and only requires one connection to Berkeley? Maybe even negotiate for permission for those files to arrive over the campus network (upload only), so that Hurricane Electric becomes effectively download only?

It would need a change to the transitioner/validator logic and timing, to avoid those 'validate error' we get when the report arrives before the upload: but with BOINC doing delayed/batched reporting anyway, the overall effect wouldn't be big. It would finally give me an excuse to ditch v5.10.13 (no point in early reporting!). All it really needs is that if the validator can't find the file it needs at the first attempt, it goes into backoff/retry (like much of the rest of BOINC) instead of immediate error.

Edit - with the current scale of operations, it might even work with a single co-lo server down at the bottom of the hill at Campus networking HQ. Keep all those upload syn/ack packets off the 100MB. If it works there, it's scalable to other campuses, other continents as SETI grows.

Would an set of remote upload servers as "data aggregators" work?

Receives many small files, handles many TCP connections: then, once an hour or so, zips them all up into one big compressed file and only requires one connection to Berkeley? Maybe even negotiate for permission for those files to arrive over the campus network (upload only), so that Hurricane Electric becomes effectively download only?

It would need a change to the transitioner/validator logic and timing, to avoid those 'validate error' we get when the report arrives before the upload: but with BOINC doing delayed/batched reporting anyway, the overall effect wouldn't be big. It would finally give me an excuse to ditch v5.10.13 (no point in early reporting!). All it really needs is that if the validator can't find the file it needs at the first attempt, it goes into backoff/retry (like much of the rest of BOINC) instead of immediate error.

Edit - with the current scale of operations, it might even work with a single co-lo server down at the bottom of the hill at Campus networking HQ. Keep all those upload syn/ack packets off the 100MB. If it works there, it's scalable to other campuses, other continents as SETI grows.

A Quick zip indicates that 58 finished results takes up aprox 275K in size so that would equal 46 Mb for every 10000 results in size.
That file could reside in an incoming folder with a cron script unpacking *.zip into that same folder.

Eric is close to implementing a configuration change which will increase the resolution of chirp rates (thus increasing analysis/sensitivity)

A Quick zip indicates that 58 finished results takes up aprox 275K in size so that would equal 46 Mb for every 10000 results in size.
That file could reside in an incoming folder with a cron script unpacking *.zip into that same folder.

Looking at Scarecrow's graphs, we've had sustained bursts of 70,000 - 80,000 results per hour recently, but nothing above that. So all you need is a server capable of running Apache/fastcgi, with a public-facing NIC capable of handling that many connections (the only stress point), and a private-facing NIC which can transfer one 45MB file every seven or eight minutes. That sounds do-able: didn't someone post a link saying that Campus had some free refurb servers available for bidding?

Edit - better get a RAID / hot spare / fresh set of server-class, 5-year warranty, 15,000 rpm disk drives though - they're going to take a hammering!

Would an set of remote upload servers as "data aggregators" work?

Receives many small files, handles many TCP connections: then, once an hour or so, zips them all up into one big compressed file and only requires one connection to Berkeley? Maybe even negotiate for permission for those files to arrive over the campus network (upload only), so that Hurricane Electric becomes effectively download only?

It would need a change to the transitioner/validator logic and timing, to avoid those 'validate error' we get when the report arrives before the upload: but with BOINC doing delayed/batched reporting anyway, the overall effect wouldn't be big. It would finally give me an excuse to ditch v5.10.13 (no point in early reporting!). All it really needs is that if the validator can't find the file it needs at the first attempt, it goes into backoff/retry (like much of the rest of BOINC) instead of immediate error.

Edit - with the current scale of operations, it might even work with a single co-lo server down at the bottom of the hill at Campus networking HQ. Keep all those upload syn/ack packets off the 100MB. If it works there, it's scalable to other campuses, other continents as SETI grows.

Looking at Scarecrow's graphs, we've had sustained bursts of 70,000 - 80,000 results per hour recently, but nothing above that.

Looking at the 90 day graphs shows quite a few peaks over 100,000, and a couple over 200,000.

Unfortunately, zip is not an option for a couple of reasons. First, that particular data does not zip very weil...

The compression ratio on the data as tested range from aout 3% for AP to about 17% for SETI. This is not enough to fix the problem.

My firm belief is there's no bandwidth problem on the link going to SSL, rather it's a transaction rate problem.

The compression ratio on the data as tested range from aout 3% for AP to about 17% for SETI. This is not enough to fix the problem.

The last round of bottlenecks were server CPU issues. We don't need to aggravate that area either.

You still have to cram all of the data through the same pipe, it is only in a larger file instead of a bunch of smaller ones.

John,

No, no, NO!

These are uploads I'm talking about. They are tiny text files, ASCII/XML, and they compress very sweetly