Striped (Dec 22 2010)


log in

Advanced search

Message boards : Technical News : Striped (Dec 22 2010)

1 · 2 · Next
Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1384
Credit: 74,079
RAC: 0
United States
Message 1058854 - Posted: 22 Dec 2010, 21:25:56 UTC

Everything on the oscar-raid-restripe project went along swimmingly, and I was able to take care of several steps at home last night so that we could get the whole project back online in the morning today.

Funny aside #1: at the exact moment I was finally comfortable enough to issue the command that blew away the old raid device from my home terminal, the network connectivity in my house randomly disappeared. Needless to say this incited minor panic as I suddenly couldn't reach any SETI servers and thought I must have somehow locked up the whole server closet. Eventually I figured out it was a local problem, my home router rebooted itself and I was back in business. Phew.

Funny aside #2: turns out when I installed the OS on oscar/carolyn I didn't bother removing the NetworkManager package, which gets installed by default for reasons beyond my power of conception. I've ranted about this before but it seems the only reason NetworkManager exists is to generate completely random, unexpected, obfuscated network problems on systems in order to give network administrators (a) something to do to kill time, and (b) ensure job security. In this case, without cause or prior consent it added funny loopback address statements in /etc/hosts which caused remote informix connections to fail. I immediately removed NetworkManager at that point and added a "exclude=NetworkManager*" line to yum.conf (which I should have done before).

Anyway, right off the bat there seems to be at least some improvement with the smaller stripe size on the raid, but we're collecting i/o stats under load and have some tweaks in mind which may continue to help general performance. All told, this ended up just being a one-day outage well worth taking.

Meanwhile, upon turning everything back on mysql started exhibiting some old, bad habits. I haven't seen this behavior in a long while, but sometimes when it gets hit hard enough it says, "Yo! Back off! In fact, I'm going to block all incoming queries and write to disk for 10-20 minutes. I won't tell you why, and there's nothing you can do about it." Fair enough - but if you were having trouble reaching the website an hour ago, that's the reason.

That's pretty much it for this week. It's officially the Xmas holiday starting tomorrow. Whatever you choose to do, enjoy! I'll be checking in from home over the "time off," but we're hoping it's kind of a "set it and forget it" kind of weekend instead of an "oh well everything will be down until we return on Monday" kind of weekend.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Claggy
Volunteer tester
Send message
Joined: 5 Jul 99
Posts: 3960
Credit: 31,820,484
RAC: 10,215
United Kingdom
Message 1058857 - Posted: 22 Dec 2010, 21:32:36 UTC - in response to Message 1058854.

Thanks for the update Matt, and have a Merry Christmas,

Claggy

Profile ignorance is no excuse
Avatar
Send message
Joined: 4 Oct 00
Posts: 9529
Credit: 44,432,016
RAC: 169
Korea, North
Message 1058858 - Posted: 22 Dec 2010, 21:33:21 UTC - in response to Message 1058854.

Sounds like you've kept busy. Heck you've earned 4 day weekend.
____________
In a rich man's house there is no place to spit but his face.
Diogenes Of Sinope

End terrorism by building a school

Profile Cliff Harding
Volunteer tester
Avatar
Send message
Joined: 18 Aug 99
Posts: 822
Credit: 46,504,882
RAC: 18,685
United States
Message 1058859 - Posted: 22 Dec 2010, 21:34:47 UTC - in response to Message 1058854.

Matt,

Thank for the update!! Oscar is sure proving his worth with the re-striping and re-loading of science DB. That thing sure is fasssst!!. We might have to replace all of the other servers just to keep up with him and Carolyn. Any idea of an ETA of the third server?

Keep of the good work and have a very joyful Christmas!!

Profile soft^spirit
Avatar
Send message
Joined: 18 May 99
Posts: 6374
Credit: 28,216,480
RAC: 183
United States
Message 1058860 - Posted: 22 Dec 2010, 21:35:02 UTC

Aww how nice, we got early servers back for Xmas.. And we did not get you anything :(

Enjoy your holiday!
____________

Janice

Profile Chris S
Volunteer tester
Avatar
Send message
Joined: 19 Nov 00
Posts: 29468
Credit: 8,864,648
RAC: 26,944
United Kingdom
Message 1058874 - Posted: 22 Dec 2010, 21:56:15 UTC

Thanks for the update Matt. Hey, tell you what, that was a very entertaining post to read. You can now add a 4th Career to your CV, Fledgling Journalist !! :-)

Have a happy Christmas Matt, you've earned it.

____________
Damsel Rescuer, Kitty Patron, Uli Fan, Julie Supporter, CAMRA
ES99 Admirer, Raccoon Friend, IFAW, PETA, 5% Badge


Profile Bernie Vine
Volunteer tester
Avatar
Send message
Joined: 26 May 99
Posts: 6601
Credit: 22,368,370
RAC: 14,903
United Kingdom
Message 1058875 - Posted: 22 Dec 2010, 21:58:50 UTC

Merry Christmas Matt, we all do really appreciate the work you all do. Despite sometimes being a bit grumpy :-)

Bernie
____________


Today is life, the only life we're sure of. Make the most of today.

Profile Jim_S
Avatar
Send message
Joined: 23 Feb 00
Posts: 4456
Credit: 17,471,844
RAC: 6,630
United States
Message 1058886 - Posted: 22 Dec 2010, 22:32:26 UTC - in response to Message 1058875.

Merry Christmas Matt, we all do really appreciate the work you all do. Despite sometimes being a bit grumpy :-)

Bernie

What Bernie said!!! ;-]
____________

I Desire Peace and Justice, Jim Scott

Profile Gary Charpentier
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 11732
Credit: 5,969,877
RAC: 0
United States
Message 1058887 - Posted: 22 Dec 2010, 22:32:55 UTC

Merry Christmas Matt and the rest of the staff.

____________

Profile perryjay
Volunteer tester
Avatar
Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 13,656,502
RAC: 12,057
United States
Message 1058888 - Posted: 22 Dec 2010, 22:32:59 UTC - in response to Message 1058875.

Merry Christmas Matt and all the gang there at SETI. Enjoy your holidays and here's hoping we won't have to see you before Monday.
____________


PROUD MEMBER OF Team Starfire World BOINC

Profile Geek@Play
Volunteer tester
Avatar
Send message
Joined: 31 Jul 01
Posts: 2460
Credit: 83,731,956
RAC: 29,525
United States
Message 1058895 - Posted: 22 Dec 2010, 22:51:50 UTC
Last modified: 22 Dec 2010, 22:52:27 UTC

Merry Christmas to the
Entire Staff at Boinc/Seti


You earned it and thanks......................from all of us!
____________
Boinc....Boinc....Boinc....Boinc....

KB7RZF
Volunteer tester
Avatar
Send message
Joined: 15 Aug 99
Posts: 9447
Credit: 2,838,829
RAC: 1,888
United States
Message 1058898 - Posted: 22 Dec 2010, 22:57:45 UTC

Nice work Matt and folks at the lab. From my family to yours, we wish you a very Merry Christmas!
____________

Profile [seti.international] Dirk Sadowski
Volunteer tester
Avatar
Send message
Joined: 6 Apr 07
Posts: 6966
Credit: 57,029,935
RAC: 22,509
Germany
Message 1058935 - Posted: 23 Dec 2010, 0:46:25 UTC - in response to Message 1058854.

Matt, thanks for the news!


Merry Christmas and happy and relaxing holidays to the SETI@home crew, all members, not members and aliens out there!



____________
BR



>Das Deutsche Cafe. The German Cafe.<

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 37286
Credit: 497,794,046
RAC: 491,529
United States
Message 1058939 - Posted: 23 Dec 2010, 0:57:25 UTC
Last modified: 23 Dec 2010, 0:57:55 UTC

Simply ROFLMAO....

Matt, I have not read such a hilarious bit in years.
Excepting the part where you thought you had crashed the whole closet...LOL.
Even sys admins have a good day once in a while....

You and the whole Seti staff have a great holiday.
If it breaks itself while you are celebrating, it'll wait.
The birth of Jesus only comes around once a year.

Merry for Chirstmas...from Mark and the kitties.

Meow.
____________
******************
Crunching Seti, loving all of God's kitties.

I have met a few friends in my life.
Most were cats.

Profile Geek@Play
Volunteer tester
Avatar
Send message
Joined: 31 Jul 01
Posts: 2460
Credit: 83,731,956
RAC: 29,525
United States
Message 1058946 - Posted: 23 Dec 2010, 1:13:36 UTC

Funny aside #1: at the exact moment I was finally comfortable enough to issue the command that blew away the old raid device from my home terminal, the network connectivity in my house randomly disappeared. Needless to say this incited minor panic as I suddenly couldn't reach any SETI servers and thought I must have somehow locked up the whole server closet. Eventually I figured out it was a local problem, my home router rebooted itself and I was back in business. Phew.


This is so cool! Even the best get bitten once in a while.


____________
Boinc....Boinc....Boinc....Boinc....

Profile Bill G
Avatar
Send message
Joined: 1 Jun 01
Posts: 340
Credit: 29,782,232
RAC: 65,828
United States
Message 1058982 - Posted: 23 Dec 2010, 2:58:54 UTC - in response to Message 1058946.

Thanks for the update and enjoy the holiday.
____________

seti@elrcastor.com
Volunteer tester
Send message
Joined: 30 Jan 00
Posts: 35
Credit: 4,860,803
RAC: 352
United States
Message 1059023 - Posted: 23 Dec 2010, 5:56:53 UTC

is the beta project going to be brought back up? it's kicking up a message it's still down for maintenance.
____________

B-Man
Volunteer tester
Send message
Joined: 11 Feb 01
Posts: 253
Credit: 147,366
RAC: 0
United States
Message 1059030 - Posted: 23 Dec 2010, 7:02:17 UTC

I'll second the question about the Beta project. Any plans to bring it back online.

2nd Hope the staff has a wonderful christmas holiday. May it all run smoothly over the weekend with no staff intervention needed.
____________

Profile Mike
Volunteer tester
Avatar
Send message
Joined: 17 Feb 01
Posts: 22357
Credit: 29,269,662
RAC: 23,985
Germany
Message 1059034 - Posted: 23 Dec 2010, 7:45:32 UTC


Merry Christmas Matt, Jeff and all others at the lab.

____________

tbret
Volunteer tester
Avatar
Send message
Joined: 28 May 99
Posts: 2389
Credit: 164,063,373
RAC: 71,078
United States
Message 1059056 - Posted: 23 Dec 2010, 9:26:54 UTC - in response to Message 1058854.

Rest ye merry, gentlemen.
____________

1 · 2 · Next

Message boards : Technical News : Striped (Dec 22 2010)

Copyright © 2014 University of California