Striped (Dec 22 2010)


log in

Advanced search

Message boards : Technical News : Striped (Dec 22 2010)

1 · 2 · Next
Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1390
Credit: 74,079
RAC: 0
United States
Message 1058854 - Posted: 22 Dec 2010, 21:25:56 UTC

Everything on the oscar-raid-restripe project went along swimmingly, and I was able to take care of several steps at home last night so that we could get the whole project back online in the morning today.

Funny aside #1: at the exact moment I was finally comfortable enough to issue the command that blew away the old raid device from my home terminal, the network connectivity in my house randomly disappeared. Needless to say this incited minor panic as I suddenly couldn't reach any SETI servers and thought I must have somehow locked up the whole server closet. Eventually I figured out it was a local problem, my home router rebooted itself and I was back in business. Phew.

Funny aside #2: turns out when I installed the OS on oscar/carolyn I didn't bother removing the NetworkManager package, which gets installed by default for reasons beyond my power of conception. I've ranted about this before but it seems the only reason NetworkManager exists is to generate completely random, unexpected, obfuscated network problems on systems in order to give network administrators (a) something to do to kill time, and (b) ensure job security. In this case, without cause or prior consent it added funny loopback address statements in /etc/hosts which caused remote informix connections to fail. I immediately removed NetworkManager at that point and added a "exclude=NetworkManager*" line to yum.conf (which I should have done before).

Anyway, right off the bat there seems to be at least some improvement with the smaller stripe size on the raid, but we're collecting i/o stats under load and have some tweaks in mind which may continue to help general performance. All told, this ended up just being a one-day outage well worth taking.

Meanwhile, upon turning everything back on mysql started exhibiting some old, bad habits. I haven't seen this behavior in a long while, but sometimes when it gets hit hard enough it says, "Yo! Back off! In fact, I'm going to block all incoming queries and write to disk for 10-20 minutes. I won't tell you why, and there's nothing you can do about it." Fair enough - but if you were having trouble reaching the website an hour ago, that's the reason.

That's pretty much it for this week. It's officially the Xmas holiday starting tomorrow. Whatever you choose to do, enjoy! I'll be checking in from home over the "time off," but we're hoping it's kind of a "set it and forget it" kind of weekend instead of an "oh well everything will be down until we return on Monday" kind of weekend.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

ClaggyProject donor
Volunteer tester
Send message
Joined: 5 Jul 99
Posts: 4213
Credit: 34,475,120
RAC: 16,093
United Kingdom
Message 1058857 - Posted: 22 Dec 2010, 21:32:36 UTC - in response to Message 1058854.

Thanks for the update Matt, and have a Merry Christmas,

Claggy

Profile ignorance is no excuse
Avatar
Send message
Joined: 4 Oct 00
Posts: 9529
Credit: 44,433,321
RAC: 0
Korea, North
Message 1058858 - Posted: 22 Dec 2010, 21:33:21 UTC - in response to Message 1058854.

Sounds like you've kept busy. Heck you've earned 4 day weekend.
____________
In a rich man's house there is no place to spit but his face.
Diogenes Of Sinope

End terrorism by building a school

Profile Cliff HardingProject donor
Volunteer tester
Avatar
Send message
Joined: 18 Aug 99
Posts: 1026
Credit: 53,859,225
RAC: 14,904
United States
Message 1058859 - Posted: 22 Dec 2010, 21:34:47 UTC - in response to Message 1058854.

Matt,

Thank for the update!! Oscar is sure proving his worth with the re-striping and re-loading of science DB. That thing sure is fasssst!!. We might have to replace all of the other servers just to keep up with him and Carolyn. Any idea of an ETA of the third server?

Keep of the good work and have a very joyful Christmas!!

Profile soft^spirit
Avatar
Send message
Joined: 18 May 99
Posts: 6374
Credit: 28,647,395
RAC: 516
United States
Message 1058860 - Posted: 22 Dec 2010, 21:35:02 UTC

Aww how nice, we got early servers back for Xmas.. And we did not get you anything :(

Enjoy your holiday!
____________

Janice

Profile Chris SProject donor
Volunteer tester
Avatar
Send message
Joined: 19 Nov 00
Posts: 32325
Credit: 14,273,797
RAC: 9,001
United Kingdom
Message 1058874 - Posted: 22 Dec 2010, 21:56:15 UTC

Thanks for the update Matt. Hey, tell you what, that was a very entertaining post to read. You can now add a 4th Career to your CV, Fledgling Journalist !! :-)

Have a happy Christmas Matt, you've earned it.

____________
Damsel Rescuer, Uli Devotee, Julie Supporter, ES99 Admirer,
Raccoon Friend, Anniet fan, Official crusty old fart


Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 26 May 99
Posts: 7129
Credit: 28,512,430
RAC: 20,684
United Kingdom
Message 1058875 - Posted: 22 Dec 2010, 21:58:50 UTC

Merry Christmas Matt, we all do really appreciate the work you all do. Despite sometimes being a bit grumpy :-)

Bernie
____________


Today is life, the only life we're sure of. Make the most of today.

Profile Jim_SProject donor
Avatar
Send message
Joined: 23 Feb 00
Posts: 4534
Credit: 19,061,352
RAC: 5,793
United States
Message 1058886 - Posted: 22 Dec 2010, 22:32:26 UTC - in response to Message 1058875.

Merry Christmas Matt, we all do really appreciate the work you all do. Despite sometimes being a bit grumpy :-)

Bernie

What Bernie said!!! ;-]
____________

I Desire Peace and Justice, Jim Scott (Mod-Ret.)

Profile Gary CharpentierProject donor
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 12988
Credit: 7,663,443
RAC: 7,601
United States
Message 1058887 - Posted: 22 Dec 2010, 22:32:55 UTC

Merry Christmas Matt and the rest of the staff.

____________

Profile perryjay
Volunteer tester
Avatar
Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 16,211,254
RAC: 5,488
United States
Message 1058888 - Posted: 22 Dec 2010, 22:32:59 UTC - in response to Message 1058875.

Merry Christmas Matt and all the gang there at SETI. Enjoy your holidays and here's hoping we won't have to see you before Monday.
____________


PROUD MEMBER OF Team Starfire World BOINC

Profile Geek@PlayProject donor
Volunteer tester
Avatar
Send message
Joined: 31 Jul 01
Posts: 2467
Credit: 86,144,272
RAC: 279
United States
Message 1058895 - Posted: 22 Dec 2010, 22:51:50 UTC
Last modified: 22 Dec 2010, 22:52:27 UTC

Merry Christmas to the
Entire Staff at Boinc/Seti


You earned it and thanks......................from all of us!
____________
Boinc....Boinc....Boinc....Boinc....

KB7RZF
Volunteer tester
Avatar
Send message
Joined: 15 Aug 99
Posts: 9464
Credit: 3,112,738
RAC: 10
United States
Message 1058898 - Posted: 22 Dec 2010, 22:57:45 UTC

Nice work Matt and folks at the lab. From my family to yours, we wish you a very Merry Christmas!
____________

Profile [seti.international] Dirk Sadowski
Volunteer tester
Avatar
Send message
Joined: 6 Apr 07
Posts: 7115
Credit: 61,258,534
RAC: 5,651
Germany
Message 1058935 - Posted: 23 Dec 2010, 0:46:25 UTC - in response to Message 1058854.

Matt, thanks for the news!


Merry Christmas and happy and relaxing holidays to the SETI@home crew, all members, not members and aliens out there!



____________
BR

SETI@home Needs your Help ... $10 & U get a Star!

Team seti.international

Das Deutsche Cafe. The German Cafe.

Profile Geek@PlayProject donor
Volunteer tester
Avatar
Send message
Joined: 31 Jul 01
Posts: 2467
Credit: 86,144,272
RAC: 279
United States
Message 1058946 - Posted: 23 Dec 2010, 1:13:36 UTC

Funny aside #1: at the exact moment I was finally comfortable enough to issue the command that blew away the old raid device from my home terminal, the network connectivity in my house randomly disappeared. Needless to say this incited minor panic as I suddenly couldn't reach any SETI servers and thought I must have somehow locked up the whole server closet. Eventually I figured out it was a local problem, my home router rebooted itself and I was back in business. Phew.


This is so cool! Even the best get bitten once in a while.


____________
Boinc....Boinc....Boinc....Boinc....

Profile Bill GProject donor
Avatar
Send message
Joined: 1 Jun 01
Posts: 349
Credit: 43,955,466
RAC: 16,075
United States
Message 1058982 - Posted: 23 Dec 2010, 2:58:54 UTC - in response to Message 1058946.

Thanks for the update and enjoy the holiday.
____________

seti@elrcastor.com
Volunteer tester
Send message
Joined: 30 Jan 00
Posts: 35
Credit: 4,868,442
RAC: 0
United States
Message 1059023 - Posted: 23 Dec 2010, 5:56:53 UTC

is the beta project going to be brought back up? it's kicking up a message it's still down for maintenance.
____________

B-Man
Volunteer tester
Send message
Joined: 11 Feb 01
Posts: 253
Credit: 147,366
RAC: 0
United States
Message 1059030 - Posted: 23 Dec 2010, 7:02:17 UTC

I'll second the question about the Beta project. Any plans to bring it back online.

2nd Hope the staff has a wonderful christmas holiday. May it all run smoothly over the weekend with no staff intervention needed.
____________

Profile MikeProject donor
Volunteer tester
Avatar
Send message
Joined: 17 Feb 01
Posts: 24908
Credit: 34,407,431
RAC: 9,788
Germany
Message 1059034 - Posted: 23 Dec 2010, 7:45:32 UTC


Merry Christmas Matt, Jeff and all others at the lab.

____________

tbretProject donor
Volunteer tester
Avatar
Send message
Joined: 28 May 99
Posts: 2897
Credit: 218,381,475
RAC: 29,826
United States
Message 1059056 - Posted: 23 Dec 2010, 9:26:54 UTC - in response to Message 1058854.

Rest ye merry, gentlemen.
____________

1 · 2 · Next

Message boards : Technical News : Striped (Dec 22 2010)

Copyright © 2014 University of California