Improve Boinc so that better traffic management is included in the first place. Automatically throttle the client transfer rate in realtime if packets get dropped/lost?

Boinc has two different methods - the exponential backoff and the project requested delay.

Seti only uses a 7 second delay. So once a CUDA host gets a single WU it will finish it and come back a minute later and try again and again and...

If Seti set the project delay to 15 minutes, say, all the fast hosts would be backing off allowing everyone else to get something.

---

Improve Boinc so that better traffic management is included in the first place. Automatically throttle the client transfer rate in realtime if packets get dropped/lost?

Boinc has two different methods - the exponential backoff and the project requested delay.

Seti only uses a 7 second delay. So once a CUDA host gets a single WU it will finish it and come back a minute later and try again and again and...

If Seti set the project delay to 15 minutes, say, all the fast hosts would be backing off allowing everyone else to get something.

That's only one aspect and works with a similar effect as does the Ned rolling IP 'blackout' idea to reduce the number of simultaneous requests.

What I'm proposing is traffic management of the data rates so that whatever network bottleneck is suffered wherever, the clients reduce their data rates for themselves to a low enough level such that they themselves no longer see packet loss or resend requests at the TCP level. In other words, implement traffic shaping from the client side to avoid a link overload. Hit a backoff timeout if the data rate has to be tuned down to an unrealistically long send time.

In this way, Boinc can learn for itself how to best utilise limited network bandwidth as demand varies, and all done automatically with no human 'guesswork' required.

(For example, I bet that the present backoff parameters are just a human 'damp finger in the air guesstimate'...)

A big problem is that the effective bandwidth of an overloaded link is very much less than the physical link bandwidth due to all the wasteful resends required when even just a small few data packets get dropped/lost due to overload.

Regards,
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)

Matt:

I wonder if the University of California - Berkley

would be eligible to apply for some of this NSF Money ?

to run a Fiber Optic Cable up the hill to the Space Sciences Laboratory (SSL) at Berkeley ?

Re:

Inter-Campus and Intra-Campus Cyber Connectivity ?

Message: 1
From: National Science Foundation Update <nsf-update@nsf.gov>
Date: Mon, 6 Jul 2009 16:44:58 -0500 (CDT)
Subject: EPSCoR Research Infrastructure Improvement Program: Inter-Campus and Intra-Campus Cyber Connectivity




Program Title:

Inter-Campus and Intra-Campus Cyber Connectivity (RII C2)

EPSCoR Research Infrastructure Improvement Program: Inter-Campus and Intra-Campus Cyber Connectivity (RII C2)

Synopsis of Program:

The Experimental Program to Stimulate Competitive Research (EPSCoR) is a program designed to fulfill the National Science Foundation's (NSF) mandate to promote scientific progress nationwide. The EPSCoR program is directed at those jurisdictions that have historically received lesser amounts of NSF Research and Development (R&D) funding. Twenty-seven states, the Commonwealth of Puerto Rico and the U. S. Virgin Islands are currently eligible to participate. Through this program, NSF establishes partnerships with government, higher education and industry that are designed to effect lasting improvements in a state's or region's research infrastructure, R&D capacity and hence, its national R&D competitiveness.

The American Recovery and Reinvestment Act of 2009 will enable NSF to invest $20 million in the Research Infrastructure Improvement Program: Inter-Campus and Intra-Campus Cyber Connectivity (RII C2). Awards made under this program will provide up to $1 million for up to 2 years to support the enhancement of inter-campus and intra-campus cyber connectivity within an EPSCoR jurisdiction. These awards are intended to enhance broadband access for academic research and the utilization of cyberinfrastructure consistent with the jurisdiction's Science and Technology (S&T) plan. The inter-campus and intra-campus connectivity targeted by these awards is expected to broaden individual and institutional participation in STEM research and education activities within and among jurisdictions and to facilitate synergy among NSF EPSCoR Research Infrastructure Improvement activities.

More information here:

Inter-Campus and Intra-Campus Cyber Connectivity

http://www.nsf.gov/pubs/2009/nsf09569/nsf09569.htm?govDel=USNSF_25

---

How about

Sunday, Monday Tuesday - download only
Wednesday - outage
Thursday, Friday, Saturday - upload only

---

How about

Sunday, Monday Tuesday - download only
Wednesday - outage
Thursday, Friday, Saturday - upload only

Everyone keeps forgetting the ability of Boinc to Cache work. As long as user are able to download up to 10 days of work, it's possible ride out any outage that is less than 10 days. I suspect a good solution would be to limit the Cache to one or two days so the data base clutter caused by unprocessed work units is limited but users can survive the several hour long outages that we often see. As for the link speed, not to long ago we were living with a 10 meg link. This is a problem that can be solved by throwing money at it. Would you donate $10 to fix the problem? Thats all it would take if enough people donate. With our numbers, we don't need large donations, we just need many small ones.

---

If the big boys at UC Berkley say, Ok seti you can have your Gig pipline but you need to pay for it i will more than gladly donate. And it would be a tad more than $10.00.

---

[/quote]

Old James

the fact is, as this or any program grows, it eventually needs a bandwidth and server upgrades. look at how many times the hotmail servers and network links have been upgraded over the years for an example.

seti will never be at a point where you can say there is enough bandwith. as more and more people come online, the pipe slowly gets clogged up again. even if we has a gigabit link to the outside world, this would only be good for a few years. imagine how fast computers will be a few years from now. what currently takes a cuda card 6 minutes might take 6 seconds.

so a few years down the road, we will be running into this same problem and asking ourselves, "what can we do to improve speed over our old gigabit line?" and someone will suggest a 10 gigabit line, and years later it will be a terabit line.

so a few years down the road, we will be running into this same problem and asking ourselves, "what can we do to improve speed over our old gigabit line?" and someone will suggest a 10 gigabit line, and years later it will be a terabit line.

So when they put the fiber in, it should be the fastest fiber they can afford.

---

i had something else i forgot to mention to matt. i hope you are reading this. since you post nearly daily updates to the technical news section, (thanks for that btw) you have gotten quite an audience of people. i am starting to think of this as an army of computer techs who can pool ideas to help you out.

since the gigabit link is probably not going to happen in the next month or 2...
why not tap into this army and see if we can come up with new ideas that hopefully don't cost any money.

i noticed the other day the pending credit link was disabled. i thought this was a good idea however, why not take it a step farther and temporarily disable other links, such as account into, statistics, maybe even the forums, etc when bandwidth is an issue? i would rather gain a bit of bandwidth then have a forum to blabber on. and its not as though the forum, account info, etc would be permanently disabled. i would suggest only disabling them during peak hours with a small blurb about "forum disabled due to bandwidth issues please try back later" to people who try to access forum when you disable it.

basically, just start selectively turning off features one by one when we run out of bandwidth. the boinc client shows you user and host totals anyways, so theres very little need to check up on the stats online except for people like me who have a burning curiosity to know how myself and my team are doing.

i know you are a busy guy however i was also going to ask if there is any way you could give us a brief rundown of the top bandwidth hogging processes?

i am going to take an obvious guess that multibeam and astropulse uploads/downloads with all of the seti@home users trying to connect to the project would be the number 1 bandwidth hog.
what would be number 2, 3, 4,5 etc...
maybe we could even vote on what parts of the seti site should be turned off first when need be? even a redesign of the seti@home graphics in the boinc client as well as the seti homepage and various links to smaller KB images could be useful in reducing bandwidth. i know there are hundreds of seti users who would jump at the chance to help create new low KB images.

i have been watching the cricket graph for the last week or 2, and more often then not it is maxed out. the only time it isnt maxed out is when the servers are down for maintenance it seems.


i hope you dont get pissed at me and everyone. else who make suggestions. as a computer engineer, i know how frustrating it can be when outsiders come in and try to tell you what to do. just keep up the good work matt. you have gotten seti@home this far so you are obviously doing things right. (or as best you can given the circumstances of the hardware and internet connection).

If it is political, that there isn't better comms to the building, maybe we need to shame the 'powers that be' into action.
A few thousand e-mails from the us users might wake them up. Or ask the students to have a demonstration on our behalf.

I think if we want to have a "demonstration" it'd be best to mail in a $5 bill rather than try to "shame" the administration into spending more money on what is clearly an under-funded project flying under the radar.

---

so a few years down the road, we will be running into this same problem and asking ourselves, "what can we do to improve speed over our old gigabit line?" and someone will suggest a 10 gigabit line, and years later it will be a terabit line.

So when they put the fiber in, it should be the fastest fiber they can afford.

As a general rule, the cost of "pulling fiber" is mostly labor.

So, you don't pull one fiber, you pull a bundle of 6 or 12, you light up the one you need and leave the rest "dark" -- then if you need more, or better, or if one breaks, you have plenty of spares.

---

so a few years down the road, we will be running into this same problem and asking ourselves, "what can we do to improve speed over our old gigabit line?" and someone will suggest a 10 gigabit line, and years later it will be a terabit line.

So when they put the fiber in, it should be the fastest fiber they can afford.

As a general rule, the cost of "pulling fiber" is mostly labor.

So, you don't pull one fiber, you pull a bundle of 6 or 12, you light up the one you need and leave the rest "dark" -- then if you need more, or better, or if one breaks, you have plenty of spares.

And for what we're talking about, you don't even pull a bundle - you lay one single armoured sheath, with a variable number of cores inside. Multi-core may be slightly heavier to ship and handle, but the labor costs are pretty static.

Last time I looked at this (last time we talked about this!), 24-core cost about double what 4-core cost. 6-fold capacity increase for 2-fold price hike is a no-brainer: go for the biggest multi-fibre your supplier makes.

so a few years down the road, we will be running into this same problem and asking ourselves, "what can we do to improve speed over our old gigabit line?" and someone will suggest a 10 gigabit line, and years later it will be a terabit line.

So when they put the fiber in, it should be the fastest fiber they can afford.

As a general rule, the cost of "pulling fiber" is mostly labor.

So, you don't pull one fiber, you pull a bundle of 6 or 12, you light up the one you need and leave the rest "dark" -- then if you need more, or better, or if one breaks, you have plenty of spares.

And for what we're talking about, you don't even pull a bundle - you lay one single armoured sheath, with a variable number of cores inside. Multi-core may be slightly heavier to ship and handle, but the labor costs are pretty static.

Last time I looked at this (last time we talked about this!), 24-core cost about double what 4-core cost. 6-fold capacity increase for 2-fold price hike is a no-brainer: go for the biggest multi-fibre your supplier makes.

And one of the other beauties about fibre (vs copper) is that you can add wavelengths to each fibre as your requirements grow - and electronics costs continue to fall. So whatever is pulled in now could support tera-bit if/when that becomes necessary.

F.

---

i hope you dont get pissed at me and everyone. else who make suggestions. as a computer engineer, i know how frustrating it can be when outsiders come in and try to tell you what to do.

All of this is observation, or based on my recollection of statements made by SETI@Home staff. Errors are mine, and my thoughts don't represent U.C. Berkeley or SETI@Home.

If you do a traceroute to setiathome.ssl.berkeley.edu and another traceroute to setiboincdata.ssl.berkeley.edu, you'll find very different paths.

The forums run using "campus" bandwidth, so the forums and the statistics links and etc. don't touch the 100 megabit bandwidth.

What they do use is database access, and the database is one of the bottlenecks.

SETI@Home buys bandwidth "off campus" because of what Campus charges them for "campus" bandwidth. The bandwidth through Hurricane Electric is much cheaper, and Campus has been gracious enough to route the off-campus bandwidth to the lab. Here is an article from 2002: http://www.mail-archive.com/setiathome@klx.com/msg01049.html.

U.C. Berkeley owns their own facilities. To bring in outside "wire" is going to cost a lot more than extending campus facilities.

The lab was built in 1966, and I'm sure they pulled in plenty of phone lines based on 1966 standards. Today, they likely would not pull in bundles and bundles of copper, but would pull a healthy bundle of fiber.

---

i had something else i forgot to mention to matt. i hope you are reading this. since you post nearly daily updates to the technical news section, (thanks for that btw) you have gotten quite an audience of people. i am starting to think of this as an army of computer techs who can pool ideas to help you out.

since the gigabit link is probably not going to happen in the next month or 2...
why not tap into this army and see if we can come up with new ideas that hopefully don't cost any money.

i noticed the other day the pending credit link was disabled. i thought this was a good idea however, why not take it a step farther and temporarily disable other links, such as account into, statistics, maybe even the forums, etc when bandwidth is an issue? i would rather gain a bit of bandwidth then have a forum to blabber on. and its not as though the forum, account info, etc would be permanently disabled. i would suggest only disabling them during peak hours with a small blurb about "forum disabled due to bandwidth issues please try back later" to people who try to access forum when you disable it.

basically, just start selectively turning off features one by one when we run out of bandwidth. the boinc client shows you user and host totals anyways, so theres very little need to check up on the stats online except for people like me who have a burning curiosity to know how myself and my team are doing.

i know you are a busy guy however i was also going to ask if there is any way you could give us a brief rundown of the top bandwidth hogging processes?

i am going to take an obvious guess that multibeam and astropulse uploads/downloads with all of the seti@home users trying to connect to the project would be the number 1 bandwidth hog.
what would be number 2, 3, 4,5 etc...
maybe we could even vote on what parts of the seti site should be turned off first when need be? even a redesign of the seti@home graphics in the boinc client as well as the seti homepage and various links to smaller KB images could be useful in reducing bandwidth. i know there are hundreds of seti users who would jump at the chance to help create new low KB images.

i have been watching the cricket graph for the last week or 2, and more often then not it is maxed out. the only time it isnt maxed out is when the servers are down for maintenance it seems.


i hope you dont get pissed at me and everyone. else who make suggestions. as a computer engineer, i know how frustrating it can be when outsiders come in and try to tell you what to do. just keep up the good work matt. you have gotten seti@home this far so you are obviously doing things right. (or as best you can given the circumstances of the hardware and internet connection).

Let me clear something up. The website is setiathome.berkeley.edu and is on the normal campus bandwidth. The upload/downloads are on ssl.berkeley.edu and are on the Huricane Electric bandwidth SETI pays for. Shutting down the forums doesn't get any additional bandwidth.

Now you ask why were they down? Load on the database servers. All those links we have to our work units and the stats are a load on the database server. They are the reason that sometimes it makes sense after the weekly outage to leave them off until the replica DB catches itself back up to sync. The stats are normally served off the replica and not the master.

---

...Multi-core may be slightly heavier to ship and handle, but the labor costs are pretty static.

Last time I looked at this (last time we talked about this!), 24-core cost about double what 4-core cost. 6-fold capacity increase for 2-fold price hike is a no-brainer: go for the biggest multi-fibre your supplier makes.

And one of the other beauties about fibre (vs copper) is that you can add wavelengths to each fibre as your requirements grow - and electronics costs continue to fall. So whatever is pulled in now could support tera-bit if/when that becomes necessary.

And from earlier in the thread:

Using the existing grey fibre looks to be the best bet.

With DWDM, you can get 8 Tb/s down one fibre... Could the Berkeley servers keep up with that?

So... 8 Tb/s x24 = 192 Tb/s


As mentioned previously, I believe it has already been mentioned that the SSL are already using grey fibre, but only at 100 Mb/s.

... OK, so perhaps 192 Tb/s isn't actually needed right now, but there are 1.5 Gb/s fibre modems readily available.

Keep searchin',
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)

As mentioned previously, I believe it has already been mentioned that the SSL are already using grey fibre, but only at 100 Mb/s.

Martin,

If you can find that anywhere, I'd be interested. I think the current wire is copper.

-- Ned

---

Currently, the Southern sky is very under-represented in the SETI data. Would there be any benefit in encouraging the Australians (for instance) to bid for money to set up a Southern SETI to distribute and analyse data from the Parkes radio telescope? Thus, if clients were unable to connect to Berkeley, they wouldn't necessarily sit idle, because they'd then automatically check if Parkes had any WUs to crunch.
Obviously, this wouldn't happen overnight because it depends upon funding and peer review. Nevertheless, increasing the amount of sky surveyed and (importantly) increasing the number of distribution hubs for SETI WUs would reduce the pressure on the Berkeley servers.
On a related note, would it be useful to get NASA or ESA involved in distributing radio telescope data? Perhaps a model of multiple distribution servers, but one receiving server might simplify things?
Of course, everything depends on funding; but if no-one tries submitting the grant applications then nothing happens.

---

On a related note, would it be useful to get NASA or ESA involved in distributing radio telescope data?

Current U.S. Law severely restricts the use of federal funds for SETI of any sort.

... that's one reason for Astropulse: because it could produce results other than SETI, it can potentially get funding from sources that can't fund SETI.

I'm sure ESA has no such restrictions.

---

As mentioned previously, I believe it has already been mentioned that the SSL are already using grey fibre, but only at 100 Mb/s.

If you can find that anywhere, I'd be interested. I think the current wire is copper.

This is from:

Bit Ceiling (Oct 08 2008) - Richard Haselgrove

Bit Ceiling (Oct 08 2008) - ML1


This is the sort of source post I'm remembering:

The really big problem is us

And from Matt himself:

Tweenday Two (Dec 27 2007)

In reality, we have a 1GBit connection to the world via Hurricane Electric, but alas this is constrained by a 100 Mbit fiber coming out of the lab down to campus - it will take some big $$$ to upgrade that, which may happen sooner or later (as it would not only benefit us).

... We have gigabit all over our server closet (more or less - some older servers are 100Mbit going into the 1Gbit switch).


So, appears there's already fibre in place, but not the high speed connections.

I wonder which department is waiting for which department to provide the $$$?

Regards,
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)