On Bandwidth (Jul 08 2009)

Message boards : Technical News : On Bandwidth (Jul 08 2009)
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · Next

AuthorMessage
William Roeder
Volunteer tester
Avatar

Send message
Joined: 19 May 99
Posts: 69
Credit: 523,414
RAC: 0
United States
Message 916134 - Posted: 9 Jul 2009, 12:28:30 UTC - in response to Message 916120.  

Improve Boinc so that better traffic management is included in the first place. Automatically throttle the client transfer rate in realtime if packets get dropped/lost?

Boinc has two different methods - the exponential backoff and the project requested delay.

Seti only uses a 7 second delay. So once a CUDA host gets a single WU it will finish it and come back a minute later and try again and again and...

If Seti set the project delay to 15 minutes, say, all the fast hosts would be backing off allowing everyone else to get something.

ID: 916134 · Report as offensive
Profile ML1
Volunteer moderator
Volunteer tester

Send message
Joined: 25 Nov 01
Posts: 20147
Credit: 7,508,002
RAC: 20
United Kingdom
Message 916136 - Posted: 9 Jul 2009, 12:48:51 UTC - in response to Message 916134.  
Last modified: 9 Jul 2009, 12:52:50 UTC

Improve Boinc so that better traffic management is included in the first place. Automatically throttle the client transfer rate in realtime if packets get dropped/lost?

Boinc has two different methods - the exponential backoff and the project requested delay.

Seti only uses a 7 second delay. So once a CUDA host gets a single WU it will finish it and come back a minute later and try again and again and...

If Seti set the project delay to 15 minutes, say, all the fast hosts would be backing off allowing everyone else to get something.

That's only one aspect and works with a similar effect as does the Ned rolling IP 'blackout' idea to reduce the number of simultaneous requests.

What I'm proposing is traffic management of the data rates so that whatever network bottleneck is suffered wherever, the clients reduce their data rates for themselves to a low enough level such that they themselves no longer see packet loss or resend requests at the TCP level. In other words, implement traffic shaping from the client side to avoid a link overload. Hit a backoff timeout if the data rate has to be tuned down to an unrealistically long send time.

In this way, Boinc can learn for itself how to best utilise limited network bandwidth as demand varies, and all done automatically with no human 'guesswork' required.

(For example, I bet that the present backoff parameters are just a human 'damp finger in the air guesstimate'...)

A big problem is that the effective bandwidth of an overloaded link is very much less than the physical link bandwidth due to all the wasteful resends required when even just a small few data packets get dropped/lost due to overload.

Regards,
Martin
See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)
ID: 916136 · Report as offensive
Profile Byron Leigh Hatch @ team Carl Sagan
Volunteer tester
Avatar

Send message
Joined: 5 Jul 99
Posts: 4548
Credit: 35,667,570
RAC: 4
Canada
Message 916145 - Posted: 9 Jul 2009, 13:48:31 UTC - in response to Message 915800.  




Matt:

I wonder if the University of California - Berkley

would be eligible to apply for some of this NSF Money ?

to run a Fiber Optic Cable up the hill to the Space Sciences Laboratory (SSL) at Berkeley ?

Re:

Inter-Campus and Intra-Campus Cyber Connectivity ?

Message: 1
From: National Science Foundation Update <nsf-update@nsf.gov>
Date: Mon, 6 Jul 2009 16:44:58 -0500 (CDT)
Subject: EPSCoR Research Infrastructure Improvement Program: Inter-Campus and Intra-Campus Cyber Connectivity




Program Title:

Inter-Campus and Intra-Campus Cyber Connectivity (RII C2)

EPSCoR Research Infrastructure Improvement Program: Inter-Campus and Intra-Campus Cyber Connectivity (RII C2)

Synopsis of Program:

The Experimental Program to Stimulate Competitive Research (EPSCoR) is a program designed to fulfill the National Science Foundation's (NSF) mandate to promote scientific progress nationwide. The EPSCoR program is directed at those jurisdictions that have historically received lesser amounts of NSF Research and Development (R&D) funding. Twenty-seven states, the Commonwealth of Puerto Rico and the U. S. Virgin Islands are currently eligible to participate. Through this program, NSF establishes partnerships with government, higher education and industry that are designed to effect lasting improvements in a state's or region's research infrastructure, R&D capacity and hence, its national R&D competitiveness.

The American Recovery and Reinvestment Act of 2009 will enable NSF to invest $20 million in the Research Infrastructure Improvement Program: Inter-Campus and Intra-Campus Cyber Connectivity (RII C2). Awards made under this program will provide up to $1 million for up to 2 years to support the enhancement of inter-campus and intra-campus cyber connectivity within an EPSCoR jurisdiction. These awards are intended to enhance broadband access for academic research and the utilization of cyberinfrastructure consistent with the jurisdiction's Science and Technology (S&T) plan. The inter-campus and intra-campus connectivity targeted by these awards is expected to broaden individual and institutional participation in STEM research and education activities within and among jurisdictions and to facilitate synergy among NSF EPSCoR Research Infrastructure Improvement activities.

More information here:

Inter-Campus and Intra-Campus Cyber Connectivity

http://www.nsf.gov/pubs/2009/nsf09569/nsf09569.htm?govDel=USNSF_25







ID: 916145 · Report as offensive
KWSN Sir Clark
Volunteer tester

Send message
Joined: 17 Aug 02
Posts: 139
Credit: 1,002,493
RAC: 8
United Kingdom
Message 916153 - Posted: 9 Jul 2009, 14:15:01 UTC

How about

Sunday, Monday Tuesday - download only
Wednesday - outage
Thursday, Friday, Saturday - upload only
ID: 916153 · Report as offensive
Dena Wiltsie
Volunteer tester

Send message
Joined: 19 Apr 01
Posts: 1628
Credit: 24,230,968
RAC: 26
United States
Message 916164 - Posted: 9 Jul 2009, 14:46:00 UTC - in response to Message 916153.  

How about

Sunday, Monday Tuesday - download only
Wednesday - outage
Thursday, Friday, Saturday - upload only

Everyone keeps forgetting the ability of Boinc to Cache work. As long as user are able to download up to 10 days of work, it's possible ride out any outage that is less than 10 days. I suspect a good solution would be to limit the Cache to one or two days so the data base clutter caused by unprocessed work units is limited but users can survive the several hour long outages that we often see. As for the link speed, not to long ago we were living with a 10 meg link. This is a problem that can be solved by throwing money at it. Would you donate $10 to fix the problem? Thats all it would take if enough people donate. With our numbers, we don't need large donations, we just need many small ones.
ID: 916164 · Report as offensive
Profile James Sotherden
Avatar

Send message
Joined: 16 May 99
Posts: 10436
Credit: 110,373,059
RAC: 54
United States
Message 916176 - Posted: 9 Jul 2009, 15:32:10 UTC

If the big boys at UC Berkley say, Ok seti you can have your Gig pipline but you need to pay for it i will more than gladly donate. And it would be a tad more than $10.00.
[/quote]

Old James
ID: 916176 · Report as offensive
CryptokiD
Avatar

Send message
Joined: 2 Dec 00
Posts: 150
Credit: 3,216,632
RAC: 0
United States
Message 916183 - Posted: 9 Jul 2009, 15:53:54 UTC

the fact is, as this or any program grows, it eventually needs a bandwidth and server upgrades. look at how many times the hotmail servers and network links have been upgraded over the years for an example.

seti will never be at a point where you can say there is enough bandwith. as more and more people come online, the pipe slowly gets clogged up again. even if we has a gigabit link to the outside world, this would only be good for a few years. imagine how fast computers will be a few years from now. what currently takes a cuda card 6 minutes might take 6 seconds.

so a few years down the road, we will be running into this same problem and asking ourselves, "what can we do to improve speed over our old gigabit line?" and someone will suggest a 10 gigabit line, and years later it will be a terabit line.
ID: 916183 · Report as offensive
Dena Wiltsie
Volunteer tester

Send message
Joined: 19 Apr 01
Posts: 1628
Credit: 24,230,968
RAC: 26
United States
Message 916187 - Posted: 9 Jul 2009, 16:02:15 UTC - in response to Message 916183.  


so a few years down the road, we will be running into this same problem and asking ourselves, "what can we do to improve speed over our old gigabit line?" and someone will suggest a 10 gigabit line, and years later it will be a terabit line.

So when they put the fiber in, it should be the fastest fiber they can afford.
ID: 916187 · Report as offensive
CryptokiD
Avatar

Send message
Joined: 2 Dec 00
Posts: 150
Credit: 3,216,632
RAC: 0
United States
Message 916206 - Posted: 9 Jul 2009, 17:06:58 UTC - in response to Message 916187.  

i had something else i forgot to mention to matt. i hope you are reading this. since you post nearly daily updates to the technical news section, (thanks for that btw) you have gotten quite an audience of people. i am starting to think of this as an army of computer techs who can pool ideas to help you out.

since the gigabit link is probably not going to happen in the next month or 2...
why not tap into this army and see if we can come up with new ideas that hopefully don't cost any money.

i noticed the other day the pending credit link was disabled. i thought this was a good idea however, why not take it a step farther and temporarily disable other links, such as account into, statistics, maybe even the forums, etc when bandwidth is an issue? i would rather gain a bit of bandwidth then have a forum to blabber on. and its not as though the forum, account info, etc would be permanently disabled. i would suggest only disabling them during peak hours with a small blurb about "forum disabled due to bandwidth issues please try back later" to people who try to access forum when you disable it.

basically, just start selectively turning off features one by one when we run out of bandwidth. the boinc client shows you user and host totals anyways, so theres very little need to check up on the stats online except for people like me who have a burning curiosity to know how myself and my team are doing.

i know you are a busy guy however i was also going to ask if there is any way you could give us a brief rundown of the top bandwidth hogging processes?

i am going to take an obvious guess that multibeam and astropulse uploads/downloads with all of the seti@home users trying to connect to the project would be the number 1 bandwidth hog.
what would be number 2, 3, 4,5 etc...
maybe we could even vote on what parts of the seti site should be turned off first when need be? even a redesign of the seti@home graphics in the boinc client as well as the seti homepage and various links to smaller KB images could be useful in reducing bandwidth. i know there are hundreds of seti users who would jump at the chance to help create new low KB images.

i have been watching the cricket graph for the last week or 2, and more often then not it is maxed out. the only time it isnt maxed out is when the servers are down for maintenance it seems.


i hope you dont get pissed at me and everyone. else who make suggestions. as a computer engineer, i know how frustrating it can be when outsiders come in and try to tell you what to do. just keep up the good work matt. you have gotten seti@home this far so you are obviously doing things right. (or as best you can given the circumstances of the hardware and internet connection).
ID: 916206 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 916246 - Posted: 9 Jul 2009, 19:33:53 UTC - in response to Message 916109.  

If it is political, that there isn't better comms to the building, maybe we need to shame the 'powers that be' into action.
A few thousand e-mails from the us users might wake them up. Or ask the students to have a demonstration on our behalf.

I think if we want to have a "demonstration" it'd be best to mail in a $5 bill rather than try to "shame" the administration into spending more money on what is clearly an under-funded project flying under the radar.
ID: 916246 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 916248 - Posted: 9 Jul 2009, 19:37:21 UTC - in response to Message 916187.  


so a few years down the road, we will be running into this same problem and asking ourselves, "what can we do to improve speed over our old gigabit line?" and someone will suggest a 10 gigabit line, and years later it will be a terabit line.

So when they put the fiber in, it should be the fastest fiber they can afford.

As a general rule, the cost of "pulling fiber" is mostly labor.

So, you don't pull one fiber, you pull a bundle of 6 or 12, you light up the one you need and leave the rest "dark" -- then if you need more, or better, or if one breaks, you have plenty of spares.
ID: 916248 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 916259 - Posted: 9 Jul 2009, 20:25:57 UTC - in response to Message 916248.  


so a few years down the road, we will be running into this same problem and asking ourselves, "what can we do to improve speed over our old gigabit line?" and someone will suggest a 10 gigabit line, and years later it will be a terabit line.

So when they put the fiber in, it should be the fastest fiber they can afford.

As a general rule, the cost of "pulling fiber" is mostly labor.

So, you don't pull one fiber, you pull a bundle of 6 or 12, you light up the one you need and leave the rest "dark" -- then if you need more, or better, or if one breaks, you have plenty of spares.

And for what we're talking about, you don't even pull a bundle - you lay one single armoured sheath, with a variable number of cores inside. Multi-core may be slightly heavier to ship and handle, but the labor costs are pretty static.

Last time I looked at this (last time we talked about this!), 24-core cost about double what 4-core cost. 6-fold capacity increase for 2-fold price hike is a no-brainer: go for the biggest multi-fibre your supplier makes.
ID: 916259 · Report as offensive
Fred W
Volunteer tester

Send message
Joined: 13 Jun 99
Posts: 2524
Credit: 11,954,210
RAC: 0
United Kingdom
Message 916287 - Posted: 9 Jul 2009, 20:59:25 UTC - in response to Message 916259.  


so a few years down the road, we will be running into this same problem and asking ourselves, "what can we do to improve speed over our old gigabit line?" and someone will suggest a 10 gigabit line, and years later it will be a terabit line.

So when they put the fiber in, it should be the fastest fiber they can afford.

As a general rule, the cost of "pulling fiber" is mostly labor.

So, you don't pull one fiber, you pull a bundle of 6 or 12, you light up the one you need and leave the rest "dark" -- then if you need more, or better, or if one breaks, you have plenty of spares.

And for what we're talking about, you don't even pull a bundle - you lay one single armoured sheath, with a variable number of cores inside. Multi-core may be slightly heavier to ship and handle, but the labor costs are pretty static.

Last time I looked at this (last time we talked about this!), 24-core cost about double what 4-core cost. 6-fold capacity increase for 2-fold price hike is a no-brainer: go for the biggest multi-fibre your supplier makes.

And one of the other beauties about fibre (vs copper) is that you can add wavelengths to each fibre as your requirements grow - and electronics costs continue to fall. So whatever is pulled in now could support tera-bit if/when that becomes necessary.

F.
ID: 916287 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 916290 - Posted: 9 Jul 2009, 21:01:00 UTC - in response to Message 916206.  

i hope you dont get pissed at me and everyone. else who make suggestions. as a computer engineer, i know how frustrating it can be when outsiders come in and try to tell you what to do.

All of this is observation, or based on my recollection of statements made by SETI@Home staff. Errors are mine, and my thoughts don't represent U.C. Berkeley or SETI@Home.

If you do a traceroute to setiathome.ssl.berkeley.edu and another traceroute to setiboincdata.ssl.berkeley.edu, you'll find very different paths.

The forums run using "campus" bandwidth, so the forums and the statistics links and etc. don't touch the 100 megabit bandwidth.

What they do use is database access, and the database is one of the bottlenecks.

SETI@Home buys bandwidth "off campus" because of what Campus charges them for "campus" bandwidth. The bandwidth through Hurricane Electric is much cheaper, and Campus has been gracious enough to route the off-campus bandwidth to the lab. Here is an article from 2002: http://www.mail-archive.com/setiathome@klx.com/msg01049.html.

U.C. Berkeley owns their own facilities. To bring in outside "wire" is going to cost a lot more than extending campus facilities.

The lab was built in 1966, and I'm sure they pulled in plenty of phone lines based on 1966 standards. Today, they likely would not pull in bundles and bundles of copper, but would pull a healthy bundle of fiber.
ID: 916290 · Report as offensive
Profile Gary Charpentier Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 25 Dec 00
Posts: 30608
Credit: 53,134,872
RAC: 32
United States
Message 916291 - Posted: 9 Jul 2009, 21:02:27 UTC - in response to Message 916206.  

i had something else i forgot to mention to matt. i hope you are reading this. since you post nearly daily updates to the technical news section, (thanks for that btw) you have gotten quite an audience of people. i am starting to think of this as an army of computer techs who can pool ideas to help you out.

since the gigabit link is probably not going to happen in the next month or 2...
why not tap into this army and see if we can come up with new ideas that hopefully don't cost any money.

i noticed the other day the pending credit link was disabled. i thought this was a good idea however, why not take it a step farther and temporarily disable other links, such as account into, statistics, maybe even the forums, etc when bandwidth is an issue? i would rather gain a bit of bandwidth then have a forum to blabber on. and its not as though the forum, account info, etc would be permanently disabled. i would suggest only disabling them during peak hours with a small blurb about "forum disabled due to bandwidth issues please try back later" to people who try to access forum when you disable it.

basically, just start selectively turning off features one by one when we run out of bandwidth. the boinc client shows you user and host totals anyways, so theres very little need to check up on the stats online except for people like me who have a burning curiosity to know how myself and my team are doing.

i know you are a busy guy however i was also going to ask if there is any way you could give us a brief rundown of the top bandwidth hogging processes?

i am going to take an obvious guess that multibeam and astropulse uploads/downloads with all of the seti@home users trying to connect to the project would be the number 1 bandwidth hog.
what would be number 2, 3, 4,5 etc...
maybe we could even vote on what parts of the seti site should be turned off first when need be? even a redesign of the seti@home graphics in the boinc client as well as the seti homepage and various links to smaller KB images could be useful in reducing bandwidth. i know there are hundreds of seti users who would jump at the chance to help create new low KB images.

i have been watching the cricket graph for the last week or 2, and more often then not it is maxed out. the only time it isnt maxed out is when the servers are down for maintenance it seems.


i hope you dont get pissed at me and everyone. else who make suggestions. as a computer engineer, i know how frustrating it can be when outsiders come in and try to tell you what to do. just keep up the good work matt. you have gotten seti@home this far so you are obviously doing things right. (or as best you can given the circumstances of the hardware and internet connection).

Let me clear something up. The website is setiathome.berkeley.edu and is on the normal campus bandwidth. The upload/downloads are on ssl.berkeley.edu and are on the Huricane Electric bandwidth SETI pays for. Shutting down the forums doesn't get any additional bandwidth.

Now you ask why were they down? Load on the database servers. All those links we have to our work units and the stats are a load on the database server. They are the reason that sometimes it makes sense after the weekly outage to leave them off until the replica DB catches itself back up to sync. The stats are normally served off the replica and not the master.
ID: 916291 · Report as offensive
Profile ML1
Volunteer moderator
Volunteer tester

Send message
Joined: 25 Nov 01
Posts: 20147
Credit: 7,508,002
RAC: 20
United Kingdom
Message 916319 - Posted: 9 Jul 2009, 21:51:12 UTC - in response to Message 916287.  
Last modified: 9 Jul 2009, 21:56:04 UTC

...Multi-core may be slightly heavier to ship and handle, but the labor costs are pretty static.

Last time I looked at this (last time we talked about this!), 24-core cost about double what 4-core cost. 6-fold capacity increase for 2-fold price hike is a no-brainer: go for the biggest multi-fibre your supplier makes.

And one of the other beauties about fibre (vs copper) is that you can add wavelengths to each fibre as your requirements grow - and electronics costs continue to fall. So whatever is pulled in now could support tera-bit if/when that becomes necessary.

And from earlier in the thread:

Using the existing grey fibre looks to be the best bet.

With DWDM, you can get 8 Tb/s down one fibre... Could the Berkeley servers keep up with that?


So... 8 Tb/s x24 = 192 Tb/s


As mentioned previously, I believe it has already been mentioned that the SSL are already using grey fibre, but only at 100 Mb/s.

... OK, so perhaps 192 Tb/s isn't actually needed right now, but there are 1.5 Gb/s fibre modems readily available.

Keep searchin',
Martin
See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)
ID: 916319 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 916322 - Posted: 9 Jul 2009, 21:55:51 UTC - in response to Message 916319.  
Last modified: 9 Jul 2009, 21:56:05 UTC


As mentioned previously, I believe it has already been mentioned that the SSL are already using grey fibre, but only at 100 Mb/s.

Martin,

If you can find that anywhere, I'd be interested. I think the current wire is copper.

-- Ned
ID: 916322 · Report as offensive
Profile Jon Golding
Avatar

Send message
Joined: 20 Apr 00
Posts: 105
Credit: 841,861
RAC: 0
United Kingdom
Message 916327 - Posted: 9 Jul 2009, 22:01:07 UTC

Currently, the Southern sky is very under-represented in the SETI data. Would there be any benefit in encouraging the Australians (for instance) to bid for money to set up a Southern SETI to distribute and analyse data from the Parkes radio telescope? Thus, if clients were unable to connect to Berkeley, they wouldn't necessarily sit idle, because they'd then automatically check if Parkes had any WUs to crunch.
Obviously, this wouldn't happen overnight because it depends upon funding and peer review. Nevertheless, increasing the amount of sky surveyed and (importantly) increasing the number of distribution hubs for SETI WUs would reduce the pressure on the Berkeley servers.
On a related note, would it be useful to get NASA or ESA involved in distributing radio telescope data? Perhaps a model of multiple distribution servers, but one receiving server might simplify things?
Of course, everything depends on funding; but if no-one tries submitting the grant applications then nothing happens.
ID: 916327 · Report as offensive
1mp0£173
Volunteer tester

Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 916331 - Posted: 9 Jul 2009, 22:05:44 UTC - in response to Message 916327.  

On a related note, would it be useful to get NASA or ESA involved in distributing radio telescope data?

Current U.S. Law severely restricts the use of federal funds for SETI of any sort.

... that's one reason for Astropulse: because it could produce results other than SETI, it can potentially get funding from sources that can't fund SETI.

I'm sure ESA has no such restrictions.
ID: 916331 · Report as offensive
Profile ML1
Volunteer moderator
Volunteer tester

Send message
Joined: 25 Nov 01
Posts: 20147
Credit: 7,508,002
RAC: 20
United Kingdom
Message 916346 - Posted: 9 Jul 2009, 22:39:44 UTC - in response to Message 916322.  
Last modified: 9 Jul 2009, 22:40:30 UTC


As mentioned previously, I believe it has already been mentioned that the SSL are already using grey fibre, but only at 100 Mb/s.

If you can find that anywhere, I'd be interested. I think the current wire is copper.

This is from:

Bit Ceiling (Oct 08 2008) - Richard Haselgrove

Bit Ceiling (Oct 08 2008) - ML1


This is the sort of source post I'm remembering:

The really big problem is us

And from Matt himself:

Tweenday Two (Dec 27 2007)

In reality, we have a 1GBit connection to the world via Hurricane Electric, but alas this is constrained by a 100 Mbit fiber coming out of the lab down to campus - it will take some big $$$ to upgrade that, which may happen sooner or later (as it would not only benefit us).

... We have gigabit all over our server closet (more or less - some older servers are 100Mbit going into the 1Gbit switch).



So, appears there's already fibre in place, but not the high speed connections.

I wonder which department is waiting for which department to provide the $$$?

Regards,
Martin
See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)
ID: 916346 · Report as offensive
Previous · 1 · 2 · 3 · 4 · Next

Message boards : Technical News : On Bandwidth (Jul 08 2009)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.