On Bandwidth (Jul 08 2009)


log in

Advanced search

Message boards : Technical News : On Bandwidth (Jul 08 2009)

Previous · 1 · 2 · 3 · 4 · 5 · Next
Author Message
Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 8601
Credit: 4,257,457
RAC: 1,351
United Kingdom
Message 916120 - Posted: 9 Jul 2009, 11:29:39 UTC

Other ideas for working with what is there at present...

Smooth out the traffic peaks with traffic shaping so that the link can be maxed out WITHOUT dropping packets and suffering a disgraceful degradation with effectively lost bandwidth. A simple example is to use a variation of the Wondershaper script and the linux "tc" commands. (I've an example for anyone interested. Works very well.)

Use the Ned idea of restricting access by IP ranges to ease the overload peaks. A bit like a rolling blackout!

Improve Boinc so that better traffic management is included in the first place. Automatically throttle the client transfer rate in realtime if packets get dropped/lost?


I prefer the last option as Boinc is supposed to be unobtrusive by design.

A combination of all the above options would be the most thorough traffic solution.


And again... Is not the grey fibre already inplace? It's just a question of replacing the equipment at both ends on campus.

Regards,
Martin

____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 8601
Credit: 4,257,457
RAC: 1,351
United Kingdom
Message 916123 - Posted: 9 Jul 2009, 11:32:12 UTC

Meanwhile...

Isn't the biggest headache the db bottleneck and maintenance nightmare that appears to be quickly heading towards a 24hour operation in itself.

Do we get a duplicate Matt when the maintenance burden exceeds 24hours in any 24hour period?

Happy crunchin',
Martin

____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

William Roeder
Volunteer tester
Avatar
Send message
Joined: 19 May 99
Posts: 69
Credit: 523,414
RAC: 0
United States
Message 916134 - Posted: 9 Jul 2009, 12:28:30 UTC - in response to Message 916120.

Improve Boinc so that better traffic management is included in the first place. Automatically throttle the client transfer rate in realtime if packets get dropped/lost?

Boinc has two different methods - the exponential backoff and the project requested delay.

Seti only uses a 7 second delay. So once a CUDA host gets a single WU it will finish it and come back a minute later and try again and again and...

If Seti set the project delay to 15 minutes, say, all the fast hosts would be backing off allowing everyone else to get something.

____________

Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 8601
Credit: 4,257,457
RAC: 1,351
United Kingdom
Message 916136 - Posted: 9 Jul 2009, 12:48:51 UTC - in response to Message 916134.
Last modified: 9 Jul 2009, 12:52:50 UTC

Improve Boinc so that better traffic management is included in the first place. Automatically throttle the client transfer rate in realtime if packets get dropped/lost?

Boinc has two different methods - the exponential backoff and the project requested delay.

Seti only uses a 7 second delay. So once a CUDA host gets a single WU it will finish it and come back a minute later and try again and again and...

If Seti set the project delay to 15 minutes, say, all the fast hosts would be backing off allowing everyone else to get something.

That's only one aspect and works with a similar effect as does the Ned rolling IP 'blackout' idea to reduce the number of simultaneous requests.

What I'm proposing is traffic management of the data rates so that whatever network bottleneck is suffered wherever, the clients reduce their data rates for themselves to a low enough level such that they themselves no longer see packet loss or resend requests at the TCP level. In other words, implement traffic shaping from the client side to avoid a link overload. Hit a backoff timeout if the data rate has to be tuned down to an unrealistically long send time.

In this way, Boinc can learn for itself how to best utilise limited network bandwidth as demand varies, and all done automatically with no human 'guesswork' required.

(For example, I bet that the present backoff parameters are just a human 'damp finger in the air guesstimate'...)

A big problem is that the effective bandwidth of an overloaded link is very much less than the physical link bandwidth due to all the wasteful resends required when even just a small few data packets get dropped/lost due to overload.

Regards,
Martin
____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

Profile Byron Leigh Hatch @ team Carl SaganProject donor
Volunteer tester
Avatar
Send message
Joined: 5 Jul 99
Posts: 3623
Credit: 11,949,660
RAC: 238
Message 916145 - Posted: 9 Jul 2009, 13:48:31 UTC - in response to Message 915800.




Matt:

I wonder if the University of California - Berkley

would be eligible to apply for some of this NSF Money ?

to run a Fiber Optic Cable up the hill to the Space Sciences Laboratory (SSL) at Berkeley ?

Re:

Inter-Campus and Intra-Campus Cyber Connectivity ?

Message: 1
From: National Science Foundation Update <nsf-update@nsf.gov>
Date: Mon, 6 Jul 2009 16:44:58 -0500 (CDT)
Subject: EPSCoR Research Infrastructure Improvement Program: Inter-Campus and Intra-Campus Cyber Connectivity




Program Title:

Inter-Campus and Intra-Campus Cyber Connectivity (RII C2)

EPSCoR Research Infrastructure Improvement Program: Inter-Campus and Intra-Campus Cyber Connectivity (RII C2)

Synopsis of Program:

The Experimental Program to Stimulate Competitive Research (EPSCoR) is a program designed to fulfill the National Science Foundation's (NSF) mandate to promote scientific progress nationwide. The EPSCoR program is directed at those jurisdictions that have historically received lesser amounts of NSF Research and Development (R&D) funding. Twenty-seven states, the Commonwealth of Puerto Rico and the U. S. Virgin Islands are currently eligible to participate. Through this program, NSF establishes partnerships with government, higher education and industry that are designed to effect lasting improvements in a state's or region's research infrastructure, R&D capacity and hence, its national R&D competitiveness.

The American Recovery and Reinvestment Act of 2009 will enable NSF to invest $20 million in the Research Infrastructure Improvement Program: Inter-Campus and Intra-Campus Cyber Connectivity (RII C2). Awards made under this program will provide up to $1 million for up to 2 years to support the enhancement of inter-campus and intra-campus cyber connectivity within an EPSCoR jurisdiction. These awards are intended to enhance broadband access for academic research and the utilization of cyberinfrastructure consistent with the jurisdiction's Science and Technology (S&T) plan. The inter-campus and intra-campus connectivity targeted by these awards is expected to broaden individual and institutional participation in STEM research and education activities within and among jurisdictions and to facilitate synergy among NSF EPSCoR Research Infrastructure Improvement activities.

More information here:

Inter-Campus and Intra-Campus Cyber Connectivity

http://www.nsf.gov/pubs/2009/nsf09569/nsf09569.htm?govDel=USNSF_25







____________

KWSN Sir Clark
Volunteer tester
Avatar
Send message
Joined: 17 Aug 02
Posts: 131
Credit: 220,652
RAC: 117
United Kingdom
Message 916153 - Posted: 9 Jul 2009, 14:15:01 UTC

How about

Sunday, Monday Tuesday - download only
Wednesday - outage
Thursday, Friday, Saturday - upload only
____________

Dena Wiltsie
Volunteer tester
Send message
Joined: 19 Apr 01
Posts: 1292
Credit: 639,504
RAC: 2,784
United States
Message 916164 - Posted: 9 Jul 2009, 14:46:00 UTC - in response to Message 916153.

How about

Sunday, Monday Tuesday - download only
Wednesday - outage
Thursday, Friday, Saturday - upload only

Everyone keeps forgetting the ability of Boinc to Cache work. As long as user are able to download up to 10 days of work, it's possible ride out any outage that is less than 10 days. I suspect a good solution would be to limit the Cache to one or two days so the data base clutter caused by unprocessed work units is limited but users can survive the several hour long outages that we often see. As for the link speed, not to long ago we were living with a 10 meg link. This is a problem that can be solved by throwing money at it. Would you donate $10 to fix the problem? Thats all it would take if enough people donate. With our numbers, we don't need large donations, we just need many small ones.
____________

Profile James Sotherden
Avatar
Send message
Joined: 16 May 99
Posts: 9121
Credit: 37,556,403
RAC: 33,918
United States
Message 916176 - Posted: 9 Jul 2009, 15:32:10 UTC

If the big boys at UC Berkley say, Ok seti you can have your Gig pipline but you need to pay for it i will more than gladly donate. And it would be a tad more than $10.00.
____________

Old James

CryptokiD
Avatar
Send message
Joined: 2 Dec 00
Posts: 134
Credit: 2,814,936
RAC: 0
United States
Message 916183 - Posted: 9 Jul 2009, 15:53:54 UTC

the fact is, as this or any program grows, it eventually needs a bandwidth and server upgrades. look at how many times the hotmail servers and network links have been upgraded over the years for an example.

seti will never be at a point where you can say there is enough bandwith. as more and more people come online, the pipe slowly gets clogged up again. even if we has a gigabit link to the outside world, this would only be good for a few years. imagine how fast computers will be a few years from now. what currently takes a cuda card 6 minutes might take 6 seconds.

so a few years down the road, we will be running into this same problem and asking ourselves, "what can we do to improve speed over our old gigabit line?" and someone will suggest a 10 gigabit line, and years later it will be a terabit line.

Dena Wiltsie
Volunteer tester
Send message
Joined: 19 Apr 01
Posts: 1292
Credit: 639,504
RAC: 2,784
United States
Message 916187 - Posted: 9 Jul 2009, 16:02:15 UTC - in response to Message 916183.


so a few years down the road, we will be running into this same problem and asking ourselves, "what can we do to improve speed over our old gigabit line?" and someone will suggest a 10 gigabit line, and years later it will be a terabit line.

So when they put the fiber in, it should be the fastest fiber they can afford.
____________

CryptokiD
Avatar
Send message
Joined: 2 Dec 00
Posts: 134
Credit: 2,814,936
RAC: 0
United States
Message 916206 - Posted: 9 Jul 2009, 17:06:58 UTC - in response to Message 916187.

i had something else i forgot to mention to matt. i hope you are reading this. since you post nearly daily updates to the technical news section, (thanks for that btw) you have gotten quite an audience of people. i am starting to think of this as an army of computer techs who can pool ideas to help you out.

since the gigabit link is probably not going to happen in the next month or 2...
why not tap into this army and see if we can come up with new ideas that hopefully don't cost any money.

i noticed the other day the pending credit link was disabled. i thought this was a good idea however, why not take it a step farther and temporarily disable other links, such as account into, statistics, maybe even the forums, etc when bandwidth is an issue? i would rather gain a bit of bandwidth then have a forum to blabber on. and its not as though the forum, account info, etc would be permanently disabled. i would suggest only disabling them during peak hours with a small blurb about "forum disabled due to bandwidth issues please try back later" to people who try to access forum when you disable it.

basically, just start selectively turning off features one by one when we run out of bandwidth. the boinc client shows you user and host totals anyways, so theres very little need to check up on the stats online except for people like me who have a burning curiosity to know how myself and my team are doing.

i know you are a busy guy however i was also going to ask if there is any way you could give us a brief rundown of the top bandwidth hogging processes?

i am going to take an obvious guess that multibeam and astropulse uploads/downloads with all of the seti@home users trying to connect to the project would be the number 1 bandwidth hog.
what would be number 2, 3, 4,5 etc...
maybe we could even vote on what parts of the seti site should be turned off first when need be? even a redesign of the seti@home graphics in the boinc client as well as the seti homepage and various links to smaller KB images could be useful in reducing bandwidth. i know there are hundreds of seti users who would jump at the chance to help create new low KB images.

i have been watching the cricket graph for the last week or 2, and more often then not it is maxed out. the only time it isnt maxed out is when the servers are down for maintenance it seems.


i hope you dont get pissed at me and everyone. else who make suggestions. as a computer engineer, i know how frustrating it can be when outsiders come in and try to tell you what to do. just keep up the good work matt. you have gotten seti@home this far so you are obviously doing things right. (or as best you can given the circumstances of the hardware and internet connection).

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 916246 - Posted: 9 Jul 2009, 19:33:53 UTC - in response to Message 916109.

If it is political, that there isn't better comms to the building, maybe we need to shame the 'powers that be' into action.
A few thousand e-mails from the us users might wake them up. Or ask the students to have a demonstration on our behalf.

I think if we want to have a "demonstration" it'd be best to mail in a $5 bill rather than try to "shame" the administration into spending more money on what is clearly an under-funded project flying under the radar.
____________

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 916248 - Posted: 9 Jul 2009, 19:37:21 UTC - in response to Message 916187.


so a few years down the road, we will be running into this same problem and asking ourselves, "what can we do to improve speed over our old gigabit line?" and someone will suggest a 10 gigabit line, and years later it will be a terabit line.

So when they put the fiber in, it should be the fastest fiber they can afford.

As a general rule, the cost of "pulling fiber" is mostly labor.

So, you don't pull one fiber, you pull a bundle of 6 or 12, you light up the one you need and leave the rest "dark" -- then if you need more, or better, or if one breaks, you have plenty of spares.
____________

Richard HaselgroveProject donor
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8809
Credit: 53,449,463
RAC: 44,374
United Kingdom
Message 916259 - Posted: 9 Jul 2009, 20:25:57 UTC - in response to Message 916248.


so a few years down the road, we will be running into this same problem and asking ourselves, "what can we do to improve speed over our old gigabit line?" and someone will suggest a 10 gigabit line, and years later it will be a terabit line.

So when they put the fiber in, it should be the fastest fiber they can afford.

As a general rule, the cost of "pulling fiber" is mostly labor.

So, you don't pull one fiber, you pull a bundle of 6 or 12, you light up the one you need and leave the rest "dark" -- then if you need more, or better, or if one breaks, you have plenty of spares.

And for what we're talking about, you don't even pull a bundle - you lay one single armoured sheath, with a variable number of cores inside. Multi-core may be slightly heavier to ship and handle, but the labor costs are pretty static.

Last time I looked at this (last time we talked about this!), 24-core cost about double what 4-core cost. 6-fold capacity increase for 2-fold price hike is a no-brainer: go for the biggest multi-fibre your supplier makes.

Fred W
Volunteer tester
Send message
Joined: 13 Jun 99
Posts: 2524
Credit: 11,954,210
RAC: 0
United Kingdom
Message 916287 - Posted: 9 Jul 2009, 20:59:25 UTC - in response to Message 916259.


so a few years down the road, we will be running into this same problem and asking ourselves, "what can we do to improve speed over our old gigabit line?" and someone will suggest a 10 gigabit line, and years later it will be a terabit line.

So when they put the fiber in, it should be the fastest fiber they can afford.

As a general rule, the cost of "pulling fiber" is mostly labor.

So, you don't pull one fiber, you pull a bundle of 6 or 12, you light up the one you need and leave the rest "dark" -- then if you need more, or better, or if one breaks, you have plenty of spares.

And for what we're talking about, you don't even pull a bundle - you lay one single armoured sheath, with a variable number of cores inside. Multi-core may be slightly heavier to ship and handle, but the labor costs are pretty static.

Last time I looked at this (last time we talked about this!), 24-core cost about double what 4-core cost. 6-fold capacity increase for 2-fold price hike is a no-brainer: go for the biggest multi-fibre your supplier makes.

And one of the other beauties about fibre (vs copper) is that you can add wavelengths to each fibre as your requirements grow - and electronics costs continue to fall. So whatever is pulled in now could support tera-bit if/when that becomes necessary.

F.
____________

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 916290 - Posted: 9 Jul 2009, 21:01:00 UTC - in response to Message 916206.

i hope you dont get pissed at me and everyone. else who make suggestions. as a computer engineer, i know how frustrating it can be when outsiders come in and try to tell you what to do.

All of this is observation, or based on my recollection of statements made by SETI@Home staff. Errors are mine, and my thoughts don't represent U.C. Berkeley or SETI@Home.

If you do a traceroute to setiathome.ssl.berkeley.edu and another traceroute to setiboincdata.ssl.berkeley.edu, you'll find very different paths.

The forums run using "campus" bandwidth, so the forums and the statistics links and etc. don't touch the 100 megabit bandwidth.

What they do use is database access, and the database is one of the bottlenecks.

SETI@Home buys bandwidth "off campus" because of what Campus charges them for "campus" bandwidth. The bandwidth through Hurricane Electric is much cheaper, and Campus has been gracious enough to route the off-campus bandwidth to the lab. Here is an article from 2002: http://www.mail-archive.com/setiathome@klx.com/msg01049.html.

U.C. Berkeley owns their own facilities. To bring in outside "wire" is going to cost a lot more than extending campus facilities.

The lab was built in 1966, and I'm sure they pulled in plenty of phone lines based on 1966 standards. Today, they likely would not pull in bundles and bundles of copper, but would pull a healthy bundle of fiber.
____________

Profile Gary CharpentierProject donor
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 13176
Credit: 7,911,294
RAC: 14,409
United States
Message 916291 - Posted: 9 Jul 2009, 21:02:27 UTC - in response to Message 916206.

i had something else i forgot to mention to matt. i hope you are reading this. since you post nearly daily updates to the technical news section, (thanks for that btw) you have gotten quite an audience of people. i am starting to think of this as an army of computer techs who can pool ideas to help you out.

since the gigabit link is probably not going to happen in the next month or 2...
why not tap into this army and see if we can come up with new ideas that hopefully don't cost any money.

i noticed the other day the pending credit link was disabled. i thought this was a good idea however, why not take it a step farther and temporarily disable other links, such as account into, statistics, maybe even the forums, etc when bandwidth is an issue? i would rather gain a bit of bandwidth then have a forum to blabber on. and its not as though the forum, account info, etc would be permanently disabled. i would suggest only disabling them during peak hours with a small blurb about "forum disabled due to bandwidth issues please try back later" to people who try to access forum when you disable it.

basically, just start selectively turning off features one by one when we run out of bandwidth. the boinc client shows you user and host totals anyways, so theres very little need to check up on the stats online except for people like me who have a burning curiosity to know how myself and my team are doing.

i know you are a busy guy however i was also going to ask if there is any way you could give us a brief rundown of the top bandwidth hogging processes?

i am going to take an obvious guess that multibeam and astropulse uploads/downloads with all of the seti@home users trying to connect to the project would be the number 1 bandwidth hog.
what would be number 2, 3, 4,5 etc...
maybe we could even vote on what parts of the seti site should be turned off first when need be? even a redesign of the seti@home graphics in the boinc client as well as the seti homepage and various links to smaller KB images could be useful in reducing bandwidth. i know there are hundreds of seti users who would jump at the chance to help create new low KB images.

i have been watching the cricket graph for the last week or 2, and more often then not it is maxed out. the only time it isnt maxed out is when the servers are down for maintenance it seems.


i hope you dont get pissed at me and everyone. else who make suggestions. as a computer engineer, i know how frustrating it can be when outsiders come in and try to tell you what to do. just keep up the good work matt. you have gotten seti@home this far so you are obviously doing things right. (or as best you can given the circumstances of the hardware and internet connection).

Let me clear something up. The website is setiathome.berkeley.edu and is on the normal campus bandwidth. The upload/downloads are on ssl.berkeley.edu and are on the Huricane Electric bandwidth SETI pays for. Shutting down the forums doesn't get any additional bandwidth.

Now you ask why were they down? Load on the database servers. All those links we have to our work units and the stats are a load on the database server. They are the reason that sometimes it makes sense after the weekly outage to leave them off until the replica DB catches itself back up to sync. The stats are normally served off the replica and not the master.
____________

Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 8601
Credit: 4,257,457
RAC: 1,351
United Kingdom
Message 916319 - Posted: 9 Jul 2009, 21:51:12 UTC - in response to Message 916287.
Last modified: 9 Jul 2009, 21:56:04 UTC

...Multi-core may be slightly heavier to ship and handle, but the labor costs are pretty static.

Last time I looked at this (last time we talked about this!), 24-core cost about double what 4-core cost. 6-fold capacity increase for 2-fold price hike is a no-brainer: go for the biggest multi-fibre your supplier makes.

And one of the other beauties about fibre (vs copper) is that you can add wavelengths to each fibre as your requirements grow - and electronics costs continue to fall. So whatever is pulled in now could support tera-bit if/when that becomes necessary.

And from earlier in the thread:

Using the existing grey fibre looks to be the best bet.

With DWDM, you can get 8 Tb/s down one fibre... Could the Berkeley servers keep up with that?


So... 8 Tb/s x24 = 192 Tb/s


As mentioned previously, I believe it has already been mentioned that the SSL are already using grey fibre, but only at 100 Mb/s.

... OK, so perhaps 192 Tb/s isn't actually needed right now, but there are 1.5 Gb/s fibre modems readily available.

Keep searchin',
Martin
____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 916322 - Posted: 9 Jul 2009, 21:55:51 UTC - in response to Message 916319.
Last modified: 9 Jul 2009, 21:56:05 UTC


As mentioned previously, I believe it has already been mentioned that the SSL are already using grey fibre, but only at 100 Mb/s.

Martin,

If you can find that anywhere, I'd be interested. I think the current wire is copper.

-- Ned
____________

Profile Jon Golding
Avatar
Send message
Joined: 20 Apr 00
Posts: 56
Credit: 365,460
RAC: 1
United Kingdom
Message 916327 - Posted: 9 Jul 2009, 22:01:07 UTC

Currently, the Southern sky is very under-represented in the SETI data. Would there be any benefit in encouraging the Australians (for instance) to bid for money to set up a Southern SETI to distribute and analyse data from the Parkes radio telescope? Thus, if clients were unable to connect to Berkeley, they wouldn't necessarily sit idle, because they'd then automatically check if Parkes had any WUs to crunch.
Obviously, this wouldn't happen overnight because it depends upon funding and peer review. Nevertheless, increasing the amount of sky surveyed and (importantly) increasing the number of distribution hubs for SETI WUs would reduce the pressure on the Berkeley servers.
On a related note, would it be useful to get NASA or ESA involved in distributing radio telescope data? Perhaps a model of multiple distribution servers, but one receiving server might simplify things?
Of course, everything depends on funding; but if no-one tries submitting the grant applications then nothing happens.
____________

Previous · 1 · 2 · 3 · 4 · 5 · Next

Message boards : Technical News : On Bandwidth (Jul 08 2009)

Copyright © 2014 University of California