QXP-13 Space Modulator (Apr 05 2011)


log in

Advanced search

Message boards : Technical News : QXP-13 Space Modulator (Apr 05 2011)

Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 1093673 - Posted: 5 Apr 2011, 22:43:45 UTC

Happy Tuesday! We had our usual outage today (mysql cleanup/backup). The replica mysql server is still having issues. Over the weekend while it was catching up after being rebuilt it hit some corrupted relay log data. This is a bit troublesome - either the logs were corrputed on carolyn (the master) or they got corrupted during to transfer to the replica, or there are still fibre channel issues on this system causing random storage corruption (even after swapping out the entire disk cage, cable, and gbic). I'm rebuilding the replica yet again (with today's backup) and we'll go from there...

Some good news: The entire lab recently upgraded to a gigabit connection to the rest of the campus (and to the world). Actually that was months ago. We weren't seeing much help from this for some reason. Well today we found the bottleneck (one 100Mbit switch) that was constraining the traffic from our server closet. Yay! So now the web site is seeing 1000Mbit to the world instead of a meager 100Mbit. Does it seem snappier? Even more important is our raw data transfers to the offsite archives are vastly sped up, which means less opportunities for the data pipeline to get jammed (and therefore running low on raw data to split). Note this doesn't change our 100MBit limit through Hurricane Electric, which handles are result uploads/workunit downloads. We need to buy some hardware to make that happen, but we may very well eventually move our traffic onto the SSL LAN - this is a political problem more than a technical one at this point.

Over the weekend we had some web servers croak here and there, affecting the home page, workunit downloads, and scheduling. We think this was all due to removing ptolemy from our server mix (the system is powered off and its name and IP address are back in the general pool). Many machines/scripts still have references to ptolemy (or files on ptolemy). We did our best to clean this up before shutting it off, but we knew there would be some minor aches and pains. In this case, a generic web log rotation script was having fits and not killing/restarting apache very well.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

dan
Avatar
Send message
Joined: 18 Oct 02
Posts: 392
Credit: 17,594,953
RAC: 12,406
United States
Message 1093676 - Posted: 5 Apr 2011, 22:51:10 UTC - in response to Message 1093673.

Thanks for the update and glad the lab is on the larger pipe.
Dan
____________

Profile SciManStev
Volunteer tester
Avatar
Send message
Joined: 20 Jun 99
Posts: 4792
Credit: 79,817,640
RAC: 36,385
United States
Message 1093677 - Posted: 5 Apr 2011, 22:51:30 UTC
Last modified: 5 Apr 2011, 23:44:30 UTC

It seems you are making excellent progress! Thank you for the update, and I share your joy for getting rid of that 100 Mbit bottleneck. The web pages are fast indeed!

Steve
____________
Warning, addicted to SETI crunching!
Crunching as a member of GPU Users Group.
GPUUG Website

Swibby Bear
Send message
Joined: 1 Aug 01
Posts: 236
Credit: 7,257,009
RAC: 1,953
United States
Message 1093684 - Posted: 5 Apr 2011, 23:10:21 UTC - in response to Message 1093673.
Last modified: 5 Apr 2011, 23:12:34 UTC

Note this doesn't change our 100MBit limit through Hurricane Electric, which handles our result uploads/workunit downloads. We need to buy some hardware to make that happen, but we may very well eventually move our traffic onto the SSL LAN - this is a political problem more than a technical one at this point.

Suggestion: As a way to smooth the political problems and to not totally saturate the gigabit line, should you ever get connected to it, consider sending downloads from one download server thru Hurricane, and the downloads from the other download server (and all the uploads) thru the gigabit connection. That way, the other members of the SSL won't have a major complaint.
Whit

Claggy
Volunteer tester
Send message
Joined: 5 Jul 99
Posts: 4048
Credit: 32,693,420
RAC: 508
United Kingdom
Message 1093704 - Posted: 5 Apr 2011, 23:42:11 UTC - in response to Message 1093673.

Thanks for the update Matt,

Claggy

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38335
Credit: 561,359,323
RAC: 651,099
United States
Message 1093713 - Posted: 6 Apr 2011, 0:48:37 UTC
Last modified: 6 Apr 2011, 0:49:46 UTC

Thank you for the good news, Matt.
Swibby Bear's suggestion of splitting the download servers over 2 links is not a bad idea. Both to share the load a bit, and perhaps to provide some redundancy in case one link or the other goes down.
And yup...the forums seem pretty zippy right now!!

Or, if it takes hardware to upgrade the Hurricane link, perhaps you could advise what would be required to make it happen.
The Seti faithful have shown a great response when presented with a specific goal or problem.

Best of luck in sorting the other nagging problems...
I have the best of faith that you will keep playing whack a drive with them until you beat them into submission.

Meow meow!
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

Profile Bill Walker
Avatar
Send message
Joined: 4 Sep 99
Posts: 3330
Credit: 1,973,950
RAC: 2,321
Canada
Message 1093728 - Posted: 6 Apr 2011, 2:02:16 UTC

Thanks for that Matt. I was always big fan of the QXP-13.
____________

Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 1093952 - Posted: 6 Apr 2011, 16:08:07 UTC

Honestly I'm not exactly sure what the remaining red tape is, but historically UC was paying not by the pipe, but by the bit, which is why they forced us to buy our own connection in the first place (when our solitary upload/download server was accounting for 33% of the entire campus internet traffic). In any case this whole project is moving forward, albeit as time/effort allows.

- Matt
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38335
Credit: 561,359,323
RAC: 651,099
United States
Message 1093973 - Posted: 6 Apr 2011, 16:51:40 UTC - in response to Message 1093952.

Honestly I'm not exactly sure what the remaining red tape is, but historically UC was paying not by the pipe, but by the bit, which is why they forced us to buy our own connection in the first place (when our solitary upload/download server was accounting for 33% of the entire campus internet traffic).

- Matt

I would be amazed if that is still the case....
I wouldn't think in this day and age a Gbit link would come with a cap or be paid for 'by the bit'.
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

Profile perryjay
Volunteer tester
Avatar
Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 14,943,164
RAC: 12,269
United States
Message 1093997 - Posted: 6 Apr 2011, 17:26:05 UTC - in response to Message 1093673.

Note this doesn't change our 100MBit limit through Hurricane Electric, which handles are result uploads/workunit downloads. We need to buy some hardware to make that happen, but we may very well eventually move our traffic onto the SSL LAN - this is a political problem more than a technical one at this point.


As Mark already said, just let us know what needs to be bought and about how much you need. I'm sure we will be able to come up with it.
____________


PROUD MEMBER OF Team Starfire World BOINC

Profile HAL9000
Volunteer tester
Avatar
Send message
Joined: 11 Sep 99
Posts: 3873
Credit: 107,239,427
RAC: 100,274
United States
Message 1094106 - Posted: 6 Apr 2011, 22:25:06 UTC - in response to Message 1093973.

Honestly I'm not exactly sure what the remaining red tape is, but historically UC was paying not by the pipe, but by the bit, which is why they forced us to buy our own connection in the first place (when our solitary upload/download server was accounting for 33% of the entire campus internet traffic).

- Matt

I would be amazed if that is still the case....
I wouldn't think in this day and age a Gbit link would come with a cap or be paid for 'by the bit'.

Our 45Gb pipe at work has a fixed monthly amount then we pay by the Gb we go over. Our 10Gb link to our home office in Japan we pay by the Mb. It is done that was as those are the most cost effective solutions for us.
____________
SETI@home classic workunits: 93,865 CPU time: 863,447 hours

Join the BP6/VP6 User Group today!

Jesse Viviano
Send message
Joined: 27 Feb 00
Posts: 95
Credit: 474,230
RAC: 0
United States
Message 1097137 - Posted: 14 Apr 2011, 16:06:48 UTC

I think that dropping Hurricane Electric is a bad idea because it has been one of the few ISPs that are IPv6 ready for a long time. I cannot find information on how ready UC Berkeley is concerning IPv6, so I think there might be problems coming during the big IPv6 switchover if you dump Hurricane Electric. Also, if you do manage to upgrade the Hurricane Electric connection to gigabit, I would bet that IST would want to use the connection as a failover in case other connections to the internet fail or get saturated, or to start making money by routing traffic between Hurricane Electric and other ISPs that UC Berkeley connects to.

Message boards : Technical News : QXP-13 Space Modulator (Apr 05 2011)

Copyright © 2014 University of California