More of the Same (Jul 09 2009)


log in

Advanced search

Message boards : Technical News : More of the Same (Jul 09 2009)

1 · 2 · 3 · Next
Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1391
Credit: 74,079
RAC: 10
United States
Message 916334 - Posted: 9 Jul 2009, 22:09:13 UTC

Not much news. Eric, Jeff, and I are still poking and prodding the servers trying to figure out ways to improve the current bandwidth situation. It's all really confusing, to tell you the truth. The process is something like: scratch head, try tuning the obvious parameter, observe the completely opposite effect, scratch head again, try tuning it the other direction just for kicks, it works so we celebrate and get back to work, we check back five minutes later and realize it wasn't actually working after all, scratch head, etc.

Thanks for all the suggestions the past couple of days (actually the past ten years). Bear in mind I'm actually more of a software guy, so I'm firmly aware that there's far more expertise out there regarding the nitty gritty network stuff. That said, like all large ventures of this sort the set of resources and demands are quite random, complicated, and unique - so solutions that seems easy/obvious solution may be impossible to implement for unexpected reasons - or there's some key details that are misunderstood. This doesn't make your suggestions any less helpful/brilliant.

Okay.. back to multitasking..

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Profile Ageless
Avatar
Send message
Joined: 9 Jun 99
Posts: 12461
Credit: 2,689,246
RAC: 1,138
Netherlands
Message 916336 - Posted: 9 Jul 2009, 22:11:28 UTC - in response to Message 916334.

Won't all that scratching make you go bald? Or will a next donation drive be for hairpieces for the crew? ;-)
____________
Jord

Fighting for the correct use of the apostrophe, together with Weird Al Yankovic

zpm
Volunteer tester
Avatar
Send message
Joined: 25 Apr 08
Posts: 284
Credit: 1,616,384
RAC: 401
United States
Message 916337 - Posted: 9 Jul 2009, 22:13:21 UTC - in response to Message 916336.
Last modified: 9 Jul 2009, 22:13:27 UTC

Won't all that scratching make you go bald? Or will a next donation drive be for hairpieces for the crew? ;-)

LOL!

OzzFan
Volunteer tester
Avatar
Send message
Joined: 9 Apr 02
Posts: 13704
Credit: 31,696,386
RAC: 12,498
United States
Message 916341 - Posted: 9 Jul 2009, 22:20:58 UTC - in response to Message 916336.

Won't all that scratching make you go bald?


Dang, I have to stop scratching my head!
____________

Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 8601
Credit: 4,256,543
RAC: 1,326
United Kingdom
Message 916342 - Posted: 9 Jul 2009, 22:22:07 UTC - in response to Message 916334.

OK, one burning question:

Do you have optical fibre up to SSL across campus, or are you all copper cable for the link to HE?

Good luck with the tweakings!

Regards,
Martin

____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

Profile Bill Walker
Avatar
Send message
Joined: 4 Sep 99
Posts: 3455
Credit: 2,211,049
RAC: 870
Canada
Message 916349 - Posted: 9 Jul 2009, 23:12:24 UTC

A question for Matt et al that may be related to your current efforts: what is the Vote Monitor that now shows up on the server status page? Is this new, or have I been missing it until today?
____________

Profile Gary CharpentierProject donor
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 13168
Credit: 7,902,612
RAC: 14,190
United States
Message 916371 - Posted: 10 Jul 2009, 0:38:01 UTC - in response to Message 916342.

OK, one burning question:

Do you have optical fibre up to SSL across campus, or are you all copper cable for the link to HE?

Good luck with the tweakings!

Regards,
Martin

Answered a long time ago.
Tweenday Two (Dec 27 2007)

In reality, we have a 1GBit connection to the world via Hurricane Electric, but alas this is constrained by a 100 Mbit fiber coming out of the lab down to campus - it will take some big $$$ to upgrade that, which may happen sooner or later (as it would not only benefit us).

I mentioned how we seem to have a 60Mbit ceiling...

... We have gigabit all over our server closet (more or less - some older servers are 100Mbit going into the 1Gbit switch).


What the software guy Matt doesn't know and is out of his expertise is if that fiber is capable of giga bit speed, in which case a couple of [approved] boxes at each end is all that is needed. Crosses fingers.

As to the ticket, one wonders if some home made sweets to a worker bee in CNS might find out just what other fiber links there are and what boxes are in use and maybe even if the fiber is capable. Someone knowing that and adding it in the remarks section of a ticket might smooth things through.

Just an out of the box thought.

____________

Zydor
Send message
Joined: 4 Oct 03
Posts: 172
Credit: 491,111
RAC: 0
United Kingdom
Message 916391 - Posted: 10 Jul 2009, 1:43:03 UTC - in response to Message 916371.
Last modified: 10 Jul 2009, 1:44:13 UTC

If that fibre will take 1Gbs traffic, then it could well bring it back into the frame. Current CISCO routers can routinely handle multiple Gb/sec, and a few years ago were not averse to providing some hardware for high profile network tasks that could give some Marketing leverage.

CISCO and IBM Tivoli are long-time business partners. Its not beyond imagination that CISCO could bring some IBM Tivoli technology along with them to stitch together the whole server/database/network management mix - Tivoli would be a sledgehammer to crack a nut for sure (would only need a small Tivoli sub-set), as the SETI volume and complexity would be no issue to Tivoli in Systems Management terms, just needs Marketing Management clout to make it happen. They both have done it before where a marketing leverage gave the pay back on the hardware.

All depends on the reality of the fibre capacity ...... but given that, a phone call in the right place could produce results....

Regards
Zy
____________

NedKelly
Volunteer tester
Send message
Joined: 22 May 99
Posts: 3
Credit: 2,979,802
RAC: 723
United States
Message 916402 - Posted: 10 Jul 2009, 2:42:59 UTC - in response to Message 916391.

One thing to remember is that by solving the bandwidth problem, we probably relocate the choke point. Recall not that far back, disk space was a big issue.

Here is a crazy thought to consider - replicate the project somewhere else. There are now literally dozens of BOINC projects running out there, all running different things. Is there a partner/supporter out there with BOINC ambitions but not quite the same Noble prize aspirations willing to work with the lab, split some tapes, collect the science and ship the results back to the lab? Ideally in a different part of the world with a gigabit connection. Clearly there will be some NRE required to set it up, but the running costs should be less than 2x. I see lots of tangible benefits with bandwidth, storage, support more users, staggered down times etc.

I don't have the where all to run this down, and I imagine there are likely policy/political/practical/financial reasons that make this a long shot.

Want more crazy - once you have done this once, you can do it again.

Really, really crazy - get Google to donate a little spare server time. They have a bazillion servers, acres of disk farms and more bandwidth than most developed countries.

As a long time lurker, I know how much effort Matt and the team have put in to get the project from nothing to where it is today. So if they say this is untenable, I can respect that. I am just trying to look past what a length of cable and some new switches can do to see where the vision of an ideal future lies. I have never seen a flying pig, but some crazy ideas can bear fruit.

Back to lurking.

seti@elrcastor.com
Volunteer tester
Send message
Joined: 30 Jan 00
Posts: 35
Credit: 4,868,442
RAC: 0
United States
Message 916409 - Posted: 10 Jul 2009, 3:05:11 UTC - in response to Message 916371.

OK, one burning question:

Do you have optical fibre up to SSL across campus, or are you all copper cable for the link to HE?

Good luck with the tweakings!

Regards,
Martin

Answered a long time ago.
Tweenday Two (Dec 27 2007)

In reality, we have a 1GBit connection to the world via Hurricane Electric, but alas this is constrained by a 100 Mbit fiber coming out of the lab down to campus - it will take some big $$$ to upgrade that, which may happen sooner or later (as it would not only benefit us).

I mentioned how we seem to have a 60Mbit ceiling...

... We have gigabit all over our server closet (more or less - some older servers are 100Mbit going into the 1Gbit switch).


What the software guy Matt doesn't know and is out of his expertise is if that fiber is capable of giga bit speed, in which case a couple of [approved] boxes at each end is all that is needed. Crosses fingers.

As to the ticket, one wonders if some home made sweets to a worker bee in CNS might find out just what other fiber links there are and what boxes are in use and maybe even if the fiber is capable. Someone knowing that and adding it in the remarks section of a ticket might smooth things through.

Just an out of the box thought.


one problem is it could be multimode fiber with a length, short enough for a 100mb connection to work, less than 2km. But to do gig over multimode with a WS-G5486 LX/LH gbic you are limited to 550m. To get gig to go 2km or more you need single mode fiber, when you use single mode fiber with a LX/LH gbic you get 10km.
____________

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 916413 - Posted: 10 Jul 2009, 3:22:07 UTC - in response to Message 916371.

OK, one burning question:

Do you have optical fibre up to SSL across campus, or are you all copper cable for the link to HE?

Good luck with the tweakings!

Regards,
Martin

Answered a long time ago.
Tweenday Two (Dec 27 2007)

In reality, we have a 1GBit connection to the world via Hurricane Electric, but alas this is constrained by a 100 Mbit fiber coming out of the lab down to campus - it will take some big $$$ to upgrade that, which may happen sooner or later (as it would not only benefit us).

I mentioned how we seem to have a 60Mbit ceiling...

... We have gigabit all over our server closet (more or less - some older servers are 100Mbit going into the 1Gbit switch).


What the software guy Matt doesn't know and is out of his expertise is if that fiber is capable of giga bit speed, in which case a couple of [approved] boxes at each end is all that is needed. Crosses fingers.


Yeah, yeah, yeah, I know:

Question: how many software guys does it take to change a light bulb?

Answer: they can't, light bulbs are hardware.

Seriously, some of us software dudes know a little about hardware. A few of us even have DVMs, oscilloscopes, and soldering irons, and know how to use them.

____________

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 916415 - Posted: 10 Jul 2009, 3:29:40 UTC - in response to Message 916402.


Here is a crazy thought to consider - replicate the project somewhere else. There are now literally dozens of BOINC projects running out there, all running different things. Is there a partner/supporter out there with BOINC ambitions but not quite the same Noble prize aspirations willing to work with the lab, split some tapes, collect the science and ship the results back to the lab? Ideally in a different part of the world with a gigabit connection. Clearly there will be some NRE required to set it up, but the running costs should be less than 2x. I see lots of tangible benefits with bandwidth, storage, support more users, staggered down times etc.

Matt has said many times that the servers project is pretty "atomic" -- by which he means it'd be pretty hard to put some parts of the project here and other parts there. Most recently, in the "On Bandwidth" thread:

Of course, another option is relocating our whole project down the hill (where gigabit links are readily available), or at least the server closet. Since the backend is quite complicated with many essential and nested dependencies it's all or nothing - we can't just move one server or functionality elsewhere - we'd have to move everything (this has been explained by me and others in countless other threads over the years). If we do end up moving (always a possibility) then all the above issues are moot.


Someone else mentioned a SETI@Home based at Parkes or some other "Son of SERENDIP" site, and one could leverage the work at Berkeley and put up a complete second project -- with permission, I'm sure.

____________

Profile HAL9000
Volunteer tester
Avatar
Send message
Joined: 11 Sep 99
Posts: 4663
Credit: 123,461,090
RAC: 99,482
United States
Message 916422 - Posted: 10 Jul 2009, 4:06:02 UTC - in response to Message 916415.


Here is a crazy thought to consider - replicate the project somewhere else. There are now literally dozens of BOINC projects running out there, all running different things. Is there a partner/supporter out there with BOINC ambitions but not quite the same Noble prize aspirations willing to work with the lab, split some tapes, collect the science and ship the results back to the lab? Ideally in a different part of the world with a gigabit connection. Clearly there will be some NRE required to set it up, but the running costs should be less than 2x. I see lots of tangible benefits with bandwidth, storage, support more users, staggered down times etc.

Matt has said many times that the servers project is pretty "atomic" -- by which he means it'd be pretty hard to put some parts of the project here and other parts there. Most recently, in the "On Bandwidth" thread:

Of course, another option is relocating our whole project down the hill (where gigabit links are readily available), or at least the server closet. Since the backend is quite complicated with many essential and nested dependencies it's all or nothing - we can't just move one server or functionality elsewhere - we'd have to move everything (this has been explained by me and others in countless other threads over the years). If we do end up moving (always a possibility) then all the above issues are moot.


Someone else mentioned a SETI@Home based at Parkes or some other "Son of SERENDIP" site, and one could leverage the work at Berkeley and put up a complete second project -- with permission, I'm sure.


OK so the distributed computing part they thought up works. Now to create a distributed server side to keep up with all the clients!

I know that sorta sounds silly and a bit "OMG we can't do that" but I bet people said that when the whole distributed computing thing started.

Even if the answer to the bandwidth issue is just swapping out a few routers and getting the GB connection up at fill tilt. Like it was stated. Drive space and other resources may start to strain. If a companies such as google or ibm are willing to donate some of their datacenters to the projects. Tweaking or redoing some of the backend to allow for this could prove valuable in the future.
____________
SETI@home classic workunits: 93,865 CPU time: 863,447 hours

Join the BP6/VP6 User Group today!

Profile Gary CharpentierProject donor
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 13168
Credit: 7,902,612
RAC: 14,190
United States
Message 916423 - Posted: 10 Jul 2009, 4:08:49 UTC - in response to Message 916413.


Yeah, yeah, yeah, I know:

Question: how many software guys does it take to change a light bulb?

Answer: they can't, light bulbs are hardware.

Seriously, some of us software dudes know a little about hardware. A few of us even have DVMs, oscilloscopes, and soldering irons, and know how to use them.

Not putting him or his abilities down, just acknowledging the pecking order.

Plenty of JOAT, MON's in the world. And they make the world go around.

____________

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 916441 - Posted: 10 Jul 2009, 6:14:30 UTC - in response to Message 916422.

OK so the distributed computing part they thought up works. Now to create a distributed server side to keep up with all the clients!

I know that sorta sounds silly and a bit "OMG we can't do that" but I bet people said that when the whole distributed computing thing started.

If I may offer RFC-1925 for your review.

Specifically, corollary (6a):
It is always possible to add another level of indirection.


That's what we're talking about when we add "distributed downloads" -- instead of the splitter loading up the download server (and telling the database) we have the splitter sending stuff to local storage, a new bit pushing the work down the same pipe out to distributed servers, and then work goes from there.

If those are "volunteer servers" (following the P2P model) then BOINC has to deal with another layer of failures: disappearing servers.

CPDN did something like this, and lost an upload server -- I don't remember how it was ultimately solved, but I do remember it was ugly.

At the end of the (proverbial) day, work has to originate with the project, and end up at the project.

All of the truths apply to BOINC (and everything else).
____________

Profile HAL9000
Volunteer tester
Avatar
Send message
Joined: 11 Sep 99
Posts: 4663
Credit: 123,461,090
RAC: 99,482
United States
Message 916508 - Posted: 10 Jul 2009, 13:12:10 UTC - in response to Message 916441.

OK so the distributed computing part they thought up works. Now to create a distributed server side to keep up with all the clients!

I know that sorta sounds silly and a bit "OMG we can't do that" but I bet people said that when the whole distributed computing thing started.

If I may offer RFC-1925 for your review.

Specifically, corollary (6a):
It is always possible to add another level of indirection.


That's what we're talking about when we add "distributed downloads" -- instead of the splitter loading up the download server (and telling the database) we have the splitter sending stuff to local storage, a new bit pushing the work down the same pipe out to distributed servers, and then work goes from there.

If those are "volunteer servers" (following the P2P model) then BOINC has to deal with another layer of failures: disappearing servers.

CPDN did something like this, and lost an upload server -- I don't remember how it was ultimately solved, but I do remember it was ugly.

At the end of the (proverbial) day, work has to originate with the project, and end up at the project.

All of the truths apply to BOINC (and everything else).



lol, I like RFC-1925

"(3) With sufficient thrust, pigs fly just fine. However, this is not necessarily a good idea. It is hard to be sure where they are going to land, and it could be dangerous sitting under them as they fly overhead."

I have said something simular to this myself.
____________
SETI@home classic workunits: 93,865 CPU time: 863,447 hours

Join the BP6/VP6 User Group today!

Profile ML1
Volunteer tester
Send message
Joined: 25 Nov 01
Posts: 8601
Credit: 4,256,543
RAC: 1,326
United Kingdom
Message 916527 - Posted: 10 Jul 2009, 14:59:18 UTC - in response to Message 916508.
Last modified: 10 Jul 2009, 15:01:04 UTC

If I may offer RFC-1925 for your review.

Specifically, corollary (6a):
It is always possible to add another level of indirection.

... All of the truths apply to BOINC (and everything else).


lol, I like RFC-1925

"(3) With sufficient thrust, pigs fly just fine. However, this is not necessarily a good idea. It is hard to be sure where they are going to land, and it could be dangerous sitting under them as they fly overhead."

No "lol"... That is all Very Serious Stuff and highly and widely applicable.

I will admit that Section 2-(3) is a rather apt humourism...!

(Pink elephants not needed.)


I'm just a little bit worried for when Matt gets flattened by the 2-(3) or just finally gets blown away in the downdraught... :-(

How many spikes can the Berkeley db take?

Happy crunchin',
Martin
____________
See new freedom: Mageia4
Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

Profile KWSN THE Holy Hand Grenade!
Volunteer tester
Avatar
Send message
Joined: 20 Dec 05
Posts: 2003
Credit: 11,187,847
RAC: 13,607
United States
Message 916546 - Posted: 10 Jul 2009, 16:43:44 UTC - in response to Message 916413.



Yeah, yeah, yeah, I know:

Question: how many software guys does it take to change a light bulb?

Answer: they can't, light bulbs are hardware.

Seriously, some of us software dudes know a little about hardware. A few of us even have DVMs, oscilloscopes, and soldering irons, and know how to use them.


Ned, the correct answer is "they don't etc..." - its not that they can't, it's just not in their job description! (so, like government workers everywhere, their attitude is "Not My Job!") ;-) (from a former government employee...)

____________
.

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 916550 - Posted: 10 Jul 2009, 16:57:30 UTC - in response to Message 916508.

OK so the distributed computing part they thought up works. Now to create a distributed server side to keep up with all the clients!

I know that sorta sounds silly and a bit "OMG we can't do that" but I bet people said that when the whole distributed computing thing started.

If I may offer RFC-1925 for your review.

Specifically, corollary (6a):
It is always possible to add another level of indirection.


That's what we're talking about when we add "distributed downloads" -- instead of the splitter loading up the download server (and telling the database) we have the splitter sending stuff to local storage, a new bit pushing the work down the same pipe out to distributed servers, and then work goes from there.

If those are "volunteer servers" (following the P2P model) then BOINC has to deal with another layer of failures: disappearing servers.

CPDN did something like this, and lost an upload server -- I don't remember how it was ultimately solved, but I do remember it was ugly.

At the end of the (proverbial) day, work has to originate with the project, and end up at the project.

All of the truths apply to BOINC (and everything else).



lol, I like RFC-1925

"(3) With sufficient thrust, pigs fly just fine. However, this is not necessarily a good idea. It is hard to be sure where they are going to land, and it could be dangerous sitting under them as they fly overhead."

I have said something simular to this myself.

Those of us here who actually practice these dark and arcane arts don't laugh at RFC-1925. We're all sitting here saying "Whoa, reality."

I know I pointed to 6a) but many of the solutions proposed (P2P, Torrent, offsite upload/download servers) feel like taking a problem and moving it around.

The problem is the 100 megabit pipe, and the servers themselves.

It looks to me like the current infrastructure can handle the current average load. The problem is how the load builds during an outage, and how networking works when the load is near or above 100%.

Increasing the bandwidth (and more/better servers) raise the 100% mark, and are always going to be a good idea.

I just wonder if there are some others that could help, without reshuffling the problem.
____________

1mp0£173
Volunteer tester
Send message
Joined: 3 Apr 99
Posts: 8423
Credit: 356,897
RAC: 0
United States
Message 916554 - Posted: 10 Jul 2009, 17:03:27 UTC - in response to Message 916546.
Last modified: 10 Jul 2009, 17:04:14 UTC



Yeah, yeah, yeah, I know:

Question: how many software guys does it take to change a light bulb?

Answer: they can't, light bulbs are hardware.

Seriously, some of us software dudes know a little about hardware. A few of us even have DVMs, oscilloscopes, and soldering irons, and know how to use them.


Ned, the correct answer is "they don't etc..." - its not that they can't, it's just not in their job description! (so, like government workers everywhere, their attitude is "Not My Job!") ;-) (from a former government employee...)

Actually, this is what is known as a Joke.

It isn't necessarily true, it is meant to be humorous.

In all seriousness, though, I've spent the past four decades primarily as a software-type, but my first serious job was in "Design Automation" -- helping build the Burroughs B6800 and B6900 mainframes through software.

The line between software and hardware has always been a little blurry for me.

I will admit that because I'm a software guy, I prefer a smaller iron when doing PWB work, because things happen a little slower. If I did it all the time, I'd want a hotter iron so I could go fast.
____________

1 · 2 · 3 · Next

Message boards : Technical News : More of the Same (Jul 09 2009)

Copyright © 2014 University of California