Bandwith solution suggestion

Message boards : Number crunching : Bandwith solution suggestion
Message board moderation

To post messages, you must log in.

Previous · 1 · 2

AuthorMessage
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 918613 - Posted: 17 Jul 2009, 2:19:42 UTC - in response to Message 918435.  

@ Joe, Ned,

Could you sanity-check my "data aggregator" suggestion at #918399, please?


I wonder if the superhost (if it ever happens) would help this? It should be able to upload all its files in one chunk, thus reducing the number of network connections.

It probably wouldn't help that much. Files are broken up into multiple packets, each of which have to make it through, and any of which may hit the floor.

TCP is designed to be reliable, but of course there are limits.

I have never seen more than 30 MBits/sec going through the upload side of the 100 MBits/sec pipe. Unless the 10 minute resolution of the Cricket graphs is hiding some nasty details, I'd expect packets going that direction to reach destination with few problems. But of course TCP needs acknowledgement that data was received, and those acks are trying to squeeze in on the often saturated download side of the pipe. OTOH, there doesn't need to be an ack for each packet. Given a large enough receive buffer setting the transfer can be efficient even with many of the acks getting lost. But if there are hundreds of hosts trying to connect at the same time it increases the difficulties.

I think Richard's plan is definitely one way to skin the cat. The required change to the validation logic is something which I believe should have been implemented long ago anyhow. Still, the mechanism to retry validation if the result files aren't yet in place would be a burden if it were invoked too often. I think the aggregator should forward batches of results fairly quickly, even a reduction of "only" a factor of 50 or 100 in the number of connections up the hill ought to be very effective. The final issue I've been able to conceive is that the aggregator trying to take the potential number of connection requests on the campus 1 GBit may need to be a fairly new system with a large amount of RAM, and as Richard already pointed out it may need some quite fast disks; that gets expensive.
                                                                Joe
ID: 918613 · Report as offensive
Previous · 1 · 2

Message boards : Number crunching : Bandwith solution suggestion


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.