Tenaya (Feb 24 2009)


log in

Advanced search

Message boards : Technical News : Tenaya (Feb 24 2009)

Previous · 1 · 2 · 3 · 4
Author Message
Profile Virtual Boss*
Volunteer tester
Avatar
Send message
Joined: 4 May 08
Posts: 417
Credit: 6,169,815
RAC: 802
Australia
Message 871297 - Posted: 2 Mar 2009, 12:37:46 UTC - in response to Message 871123.
Last modified: 2 Mar 2009, 12:38:14 UTC

Simply locating the download server downhill with 1 GB access to the world is what would improve the situation. Each WU or application would only go from the SSL lab to that server once, multiple downloads by hosts would not be seen on the 100 MB link. The issue is maintenance and remote management, not any technical difficulty in having that server at a distance.


So if the download server(s) were located at the 1GB connection point -

100 Mb/s going out to clients would only equate to ~45 Mb/s traffic from lab to download server over 100 Mb/s link (plus whatever ??? bw was needed for the download server to interact with the other servers up the hill).(

This would seem to reduce the constriction imposed by the 100 Mb/s link by a a considerable amount = Good effect.

But as you state would cause some servicing difficulties = Bad effect.
And probably the extra latency between servers = Very bad effect?.

Profile RandyC
Avatar
Send message
Joined: 20 Oct 99
Posts: 714
Credit: 1,704,345
RAC: 0
United States
Message 871303 - Posted: 2 Mar 2009, 12:57:45 UTC - in response to Message 871291.

You have yet to propose a solution.

In previous discussion of the subject, he's made a couple of proposals.


Let me rephrase...propose a viable solution.

Moving the download server improves the current situation. Yes there are drawbacks, including increased system complexity and probably some extra cost too.

Throttling the server in its existing location only guarantees more DDOS 'attacks' when the network clogs up.

PhonAcq
Send message
Joined: 14 Apr 01
Posts: 1622
Credit: 21,971,372
RAC: 3,987
United States
Message 871350 - Posted: 2 Mar 2009, 15:39:32 UTC

As I/others have suggested before, a better solution would be to distribute the servers to other sites. Keep the main results database at Berkeley, but send the raw data to one or more sites willing to host splitting and downloading.

Distribution is a solution to the scalability question. It would take coordination and planning, yes. But alternative suggestions, such as the ones discussed here, also take effort to plan and implement.

Why not break the parochial model of everything has to occur at Berekeley? Of course, I don't know who may be stable enough out there to provide a second host for download services, but it is worth asking the question.

I say, lets spend 'our' efforts to make plans and take steps to have 1-3M active hosts, 3-10x increase over today.

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 871372 - Posted: 2 Mar 2009, 16:41:56 UTC - in response to Message 871277.

This is really important:

It is easier to move a problem around than it is to solve it.

There is no free lunch.


You have yet to propose a solution.

A solution needs to solve the problem, not just move it around.

Matt has repeatedly said the server cluster is pretty "atomic" -- that things are all sufficiently interrelated that splitting off one function or another would be a problem.

The only exception I can see is downloading the science application. That is one sliver that could be moved, and since trying to do that with coral cache(?) caused problems last weekend, I'd think about the work involved in moving those files off-site.

It's also a very simple function -- same files for everybody.

I don't think two result files are the same, but I've never seen two different result files for the same WU -- and I doubt anyone else has. Sure, there could be some meta format that modified one "meta-result" into the two result files and that'd do for all but reissues.

... but without that, you don't get the 2:1 leverage that some posts on the thread suggest would be possible.

I don't know enough about where the connection goes from gigabit to 100 megabits, so I don't know how much rack space would be available. I don't know the access issues, and the people who have to do the work keeping all of this running would be concerned about access.

In my opinion (having run servers on-site and off-site) the personal cost of having the servers off-site is kind-of high. I lost a small server over the weekend, and it was a minor hassle because the servers are all right here.

My suggestion is to make the BOINC client less aggressive, so we don't have these big loads right after an outage -- and to add a communications path from the project to BOINC so it can tell everyone to slow down.

But I'm not going to make any demands that this be implemented. If I get time when some of my personal projects are done, I'm going to write the code.
____________

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 871375 - Posted: 2 Mar 2009, 16:55:15 UTC - in response to Message 871303.

Throttling the server in its existing location only guarantees more DDOS 'attacks' when the network clogs up.

I used the word "throttling" in a very narrow way -- to point out that the splitter process, pushing data to an off-site download server must be controlled or that one process could easily dominate the 100 megabit connection.

I pointed out that throttling that one process would be useful to maximize throughput between the splitter and a hypothetical off-site download server.

If we're talking about the client-server connection, it is important to NOT THROTTLE them. If you slow up the bandwidth between "A" and "B" you tie up all the associated resources -- processes and connections last longer, so you likely have more of them running at once, and overhead increases.

For the client/server communications, to maximize throughput, you want to figure out how much bandwidth you can afford for downloads (70 megabits??) and then control the clients so that you have that right around that many clients downloading at any one time.

Let's say for simplicity that the average cruncher has a megabit (some faster, some slower). If you could keep about 70 downloads going at all times, things will go very fast. Doubling that probably doesn't hurt much.

If you have 7,000 attempted downloads, overhead goes up, timeouts and retries go up (because of congestion) and throughput goes way down. The server might not be able to take 7,000 simultaneous connections, so the SYN packets come in and RSTs go back and they use bandwidth, and CPU time.
____________

Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 8270
Credit: 4,071,566
RAC: 333
United Kingdom
Message 871390 - Posted: 2 Mar 2009, 17:58:22 UTC - in response to Message 871375.
Last modified: 2 Mar 2009, 18:01:54 UTC

... "throttling" in a very narrow way ...

The use of linux (*nix) "tc" to give all icmp/ack/syn packets priority and queue up all other packets with greater latency to 'slow' them down would go a good way to gracefully limiting the bandwidth at source, and all before packets get randomly dumped because the link is choked.

Yet more clever "tc" rules can be used to positively favour completing existing connections to again help to clear the deluge with a minimum of lost connections and resend requests.

Can the cisco head be configured so?...


Another trick would be to do the bandwidth management in the boinc server in the first place...


And as for getting 1 Gbit 'down the hill', I'd still expect a uWave link to cost an awful lot less than $80k including labour. There's no trenches to dig! Or are the end points actually more a case of "over" a hill rather than line-of-sight down?

If fibre is to be used, does s@h get payback when capacity is used by others?...

Regards,
Martin
____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

Aurora Borealis
Volunteer tester
Avatar
Send message
Joined: 14 Jan 01
Posts: 2975
Credit: 4,957,753
RAC: 1,460
Canada
Message 871391 - Posted: 2 Mar 2009, 18:07:18 UTC

One of the proposals, I've seen on the Boinc mailing list, was to include torrent capability in the Boinc manager. This would obviously need to be a user selectable option since most companies and some campuses would object to using this option on their computers. This idea does however have a great potential for reducing the bandwidth required to distribute new applications.

I have no idea if any of the volunteer developers has taken up the challenge of pursuing this option or how quickly it could be implemented. Boinc V7 perhaps...
____________
Questions? Answers are in the "Unofficial" BOINC Wiki.

Boinc V7.0.27
Win7 i5 3.33G 4GB, GTX470

benDan
Avatar
Send message
Joined: 28 Nov 04
Posts: 18
Credit: 6,184,119
RAC: 0
United States
Message 871395 - Posted: 2 Mar 2009, 18:15:46 UTC - in response to Message 871375.

Just a newbie question ... please no flames.
does bandwidth include stuff going out and stuff going in?
If both, do you get more if stuff only goes 1 way?
What if you had another site with a 100 whatever line.
if one site sent stuff out and the other site received stuff would that speed things up?

I know this seems very basic to you folks but I'm trying to figgure this stuff out.

Thanks
____________
--
benDan

Profile Mike O
Avatar
Send message
Joined: 1 Sep 07
Posts: 428
Credit: 6,670,998
RAC: 0
United States
Message 871404 - Posted: 2 Mar 2009, 18:31:46 UTC

what about physical transportanion of the data to a 'out of closet' server?
Swapable HDDs?
Scratch that.. They would need.
1. a nother room to put the server in.
2. the hardware to pull this off.
3. someone to make sure the data is transfered.
I dont think its worth the time, effort or expence. WUs would need to be created fast enough for a full 24 hours of downloads (72 for weekends). I dont think LANDO and BAMBI can keep up.

Im wondwering of it would be posible to use off the shelve WI-FI to get from the closet to the remote server. Do they make such a thing as a repeater like CB radio? Im not sure it would handle the data even if they could run 2 units together. My wireless maxes at 58MBS.

Good news.. I had NO problems with getting work over the weekend.




____________
Not Ready Reading BRAIN. Abort/Retry/Fail?

Aurora Borealis
Volunteer tester
Avatar
Send message
Joined: 14 Jan 01
Posts: 2975
Credit: 4,957,753
RAC: 1,460
Canada
Message 871411 - Posted: 2 Mar 2009, 19:04:40 UTC - in response to Message 871395.
Last modified: 2 Mar 2009, 19:11:44 UTC

Just a newbie question ... please no flames.

We were all been newbies at one time.

does bandwidth include stuff going out and stuff going in?
If both, do you get more if stuff only goes 1 way?

I believe that in theory you could have 100 MB in both directions. In reality once you get over 90 MB you can start to have trouble. The problem occurs if one direction gets saturated. Every packet of data requires acknowledgement of receipt in the opposite direction. If, as happened last week, the downloads are saturated then upload have difficulty because there is no download bandwidth left to confirm that the uploaded packet has been received.

What if you had another site with a 100 whatever line.
if one site sent stuff out and the other site received stuff would that speed things up?

The Seti servers have too many interdependencies to split things up into separate locations and too few people to take care of problems when they do occur, which is all the time.

I know this seems very basic to you folks but I'm trying to figgure this stuff out.

Thanks

We are all learning together.

Richard Haselgrove
Volunteer tester
Send message
Joined: 4 Jul 99
Posts: 8375
Credit: 46,745,685
RAC: 21,893
United Kingdom
Message 871416 - Posted: 2 Mar 2009, 19:26:09 UTC

Last time we had a conversation like this (probably after the last big outage, or the last big data-storm - it comes round in cycles like that), about the only thing that seemed to make sense technically was a complete, self-contained, black-box 'SETI on a stick' - or more likely, SETI on a server on a pallet.

If (and it's a very big if) you could put all the elements of the SETI workstream onto one server - BOINC database, science database, splitters, scheduler, validators, assimilators etc. etc. - you could load it up with one or more raw Arecibo recordings and ship it off to a closed community somewhere, then retrieve the science database when all was done. If you could work out a suitable ratio of tapes per college campus per semester, you could even set up a little league - first server back gets a small (non-monetary) prize - I believe US students will happily compete for pennants?

It wouldn't be SETI as we know it - no international message boards (logins would be retricted to campus), no external stats sites - anything which related to user ID, host ID, WU ID or task ID would have to be self-contained without interaction with the main international SETI site - but it seems to be about the only way of overcoming the atomicity problem.

benDan
Avatar
Send message
Joined: 28 Nov 04
Posts: 18
Credit: 6,184,119
RAC: 0
United States
Message 871418 - Posted: 2 Mar 2009, 19:41:19 UTC - in response to Message 871411.

Thanks for the info.
people were talking about mirror sites and I understood that to be a different machine not necessarily a distant machine.

guido.man wrote:
<Right now Seti@Home is coping quite well with a 100 Mb connection to the internet
for the most part.>
So I was thinking add one more line, one for up and one for down. Perhaps mirror is the wrong term to use.


____________
--
benDan

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 871450 - Posted: 2 Mar 2009, 21:38:59 UTC - in response to Message 871390.

And as for getting 1 Gbit 'down the hill', I'd still expect a uWave link to cost an awful lot less than $80k including labour. There's no trenches to dig! Or are the end points actually more a case of "over" a hill rather than line-of-sight down?

They aren't quite on top of the hill from what I can tell, so it's a question of finding something they can "see" from the hilltop that has bandwidth without some other expense.
____________

Zydor
Send message
Joined: 4 Oct 03
Posts: 172
Credit: 491,111
RAC: 0
United Kingdom
Message 871496 - Posted: 2 Mar 2009, 23:01:36 UTC - in response to Message 871450.
Last modified: 2 Mar 2009, 23:05:53 UTC

uWave is a viable solution in a broad sense. A 1Gb link would be in the region of $55,000 and very quick to install. As always the devil is in the detail - and the "extras" that turn out to be practical essentials rolf:) - however in a generalised sense the cost of the main carrier infrastructure is well below the $80K pain point being bandied about for cable. Worth revisiting even if looked at before, as prices are coming down relentlessly, as usual when technology moves onwards and upwards.

Quite a few US Companies do this, and the door could be open to a sponsorship (or part sponsorship) deal that would bring the cost down even more. The demand for short range uWave systems is high, and competition is fierce - a link like this one just may attract a Marketing Director's eye ....
____________

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 871529 - Posted: 3 Mar 2009, 0:01:24 UTC - in response to Message 871496.

uWave is a viable solution in a broad sense. A 1Gb link would be in the region of $55,000 and very quick to install. As always the devil is in the detail - and the "extras" that turn out to be practical essentials rolf:) - however in a generalised sense the cost of the main carrier infrastructure is well below the $80K pain point being bandied about for cable. Worth revisiting even if looked at before, as prices are coming down relentlessly, as usual when technology moves onwards and upwards.

Quite a few US Companies do this, and the door could be open to a sponsorship (or part sponsorship) deal that would bring the cost down even more. The demand for short range uWave systems is high, and competition is fierce - a link like this one just may attract a Marketing Director's eye ....

Remember that SETI@Home can't "bootleg" this in. The on-campus part has to be ordered, provisioned and supported by CNS.

I think that has advantages and disadvantages, but on the whole it probably works out best.

It just eliminates some of the competitive shopping that might be done.
____________

John McLeod VII
Volunteer developer
Volunteer tester
Avatar
Send message
Joined: 15 Jul 99
Posts: 24098
Credit: 517,932
RAC: 146
United States
Message 871539 - Posted: 3 Mar 2009, 0:16:04 UTC - in response to Message 871404.

Im wondwering of it would be posible to use off the shelve WI-FI to get from the closet to the remote server. Do they make such a thing as a repeater like CB radio? Im not sure it would handle the data even if they could run 2 units together. My wireless maxes at 58MBS.

As far as I know CB is not allowed to use repeaters, Amatuer (HAM) radio is.

Off the shelf WI-FI is not suitable as the range is much too limited. The required range is about a mile. The range of standard WI-FI is less than half that in perfect conditions (and when are conditions EVER perfect). As far as I know, repeaters are not done with WI-FI, and each step would need electricity and an enclosure. WI-FI is also not a speed boost over 100Mb.
____________


BOINC WIKI

Previous · 1 · 2 · 3 · 4

Message boards : Technical News : Tenaya (Feb 24 2009)

Copyright © 2014 University of California