Striped (Dec 22 2010)

Everything on the oscar-raid-restripe project went along swimmingly, and I was able to take care of several steps at home last night so that we could get the whole project back online in the morning today.

Funny aside #1: at the exact moment I was finally comfortable enough to issue the command that blew away the old raid device from my home terminal, the network connectivity in my house randomly disappeared. Needless to say this incited minor panic as I suddenly couldn't reach any SETI servers and thought I must have somehow locked up the whole server closet. Eventually I figured out it was a local problem, my home router rebooted itself and I was back in business. Phew.

Funny aside #2: turns out when I installed the OS on oscar/carolyn I didn't bother removing the NetworkManager package, which gets installed by default for reasons beyond my power of conception. I've ranted about this before but it seems the only reason NetworkManager exists is to generate completely random, unexpected, obfuscated network problems on systems in order to give network administrators (a) something to do to kill time, and (b) ensure job security. In this case, without cause or prior consent it added funny loopback address statements in /etc/hosts which caused remote informix connections to fail. I immediately removed NetworkManager at that point and added a "exclude=NetworkManager*" line to yum.conf (which I should have done before).

Anyway, right off the bat there seems to be at least some improvement with the smaller stripe size on the raid, but we're collecting i/o stats under load and have some tweaks in mind which may continue to help general performance. All told, this ended up just being a one-day outage well worth taking.

Meanwhile, upon turning everything back on mysql started exhibiting some old, bad habits. I haven't seen this behavior in a long while, but sometimes when it gets hit hard enough it says, "Yo! Back off! In fact, I'm going to block all incoming queries and write to disk for 10-20 minutes. I won't tell you why, and there's nothing you can do about it." Fair enough - but if you were having trouble reaching the website an hour ago, that's the reason.

That's pretty much it for this week. It's officially the Xmas holiday starting tomorrow. Whatever you choose to do, enjoy! I'll be checking in from home over the "time off," but we're hoping it's kind of a "set it and forget it" kind of weekend instead of an "oh well everything will be down until we return on Monday" kind of weekend.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Thanks for the update Matt, and have a Merry Christmas,

Claggy

Matt,

Thank for the update!! Oscar is sure proving his worth with the re-striping and re-loading of science DB. That thing sure is fasssst!!. We might have to replace all of the other servers just to keep up with him and Carolyn. Any idea of an ETA of the third server?

Keep of the good work and have a very joyful Christmas!!

Thanks for the update Matt. Hey, tell you what, that was a very entertaining post to read. You can now add a 4th Career to your CV, Fledgling Journalist !! :-)

Have a happy Christmas Matt, you've earned it.

____________
Damsel Rescuer, Kitty Patron, Raccoon Friend, Uli Fan,
Julie Supporter, ES99 Admirer, PETA Member, 1st Childhood

Merry Christmas Matt, we all do really appreciate the work you all do. Despite sometimes being a bit grumpy :-)

Bernie
____________

Merry Christmas Matt, we all do really appreciate the work you all do. Despite sometimes being a bit grumpy :-)

Bernie

Merry Christmas Matt and the rest of the staff.

____________

Merry Christmas to the
Entire Staff at Boinc/Seti

You earned it and thanks......................from all of us!
____________
Boinc....Boinc....Boinc....Boinc....

Nice work Matt and folks at the lab. From my family to yours, we wish you a very Merry Christmas!
____________

Matt, thanks for the news!


Merry Christmas and happy and relaxing holidays to the SETI@home crew, all members, not members and aliens out there!



____________


>Das Deutsche Cafe. The German Cafe.<

Simply ROFLMAO....

Matt, I have not read such a hilarious bit in years.
Excepting the part where you thought you had crashed the whole closet...LOL.
Even sys admins have a good day once in a while....

You and the whole Seti staff have a great holiday.
If it breaks itself while you are celebrating, it'll wait.
The birth of Jesus only comes around once a year.

Merry for Chirstmas...from Mark and the kitties.

Meow.
____________
******

"Ask not, what your kitty can do for you. Ask what you can do for your kitty."

As it is kitten, so shall it be done.

Funny aside #1: at the exact moment I was finally comfortable enough to issue the command that blew away the old raid device from my home terminal, the network connectivity in my house randomly disappeared. Needless to say this incited minor panic as I suddenly couldn't reach any SETI servers and thought I must have somehow locked up the whole server closet. Eventually I figured out it was a local problem, my home router rebooted itself and I was back in business. Phew.

is the beta project going to be brought back up? it's kicking up a message it's still down for maintenance.
____________

I'll second the question about the Beta project. Any plans to bring it back online.

2nd Hope the staff has a wonderful christmas holiday. May it all run smoothly over the weekend with no staff intervention needed.
____________

Merry Christmas Matt, Jeff and all others at the lab.

____________