Tuesday Edition (Aug 26 2008)


log in

Advanced search

Message boards : Technical News : Tuesday Edition (Aug 26 2008)

Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1389
Credit: 74,079
RAC: 0
United States
Message 802336 - Posted: 26 Aug 2008, 22:53:45 UTC

Ah, yes - here we go again - the regular Tuesday outage for mysql database backup/compression and other tasks better suited to happen during "quiescent" time.

For example, this week we replaced the failed drive in the workunit storage server with a new drive. That was painless. We also spent a bunch of time experimenting with the new-ish RAID server. I say "new-ish" as it's new to us, but it is an old system. For example, it can't handle logical volumes greater than 2TB. We however today confirmed (a) it can handle physical single drives at least 750GB in size, and (b) physical volumes greater than 2TB (i.e. put three 750GB drive together to make a 1.5TB RAID5).

We also tested that this system is keeping up pretty well doing a continual backup of our upload directory. That is, we're doing a constant rsync with the upload directory to keep a "hot backup" around on a separate system. We didn't have the bandwidth/storage capacity to do this ourselves before (and daily backups to tape were too expensive).

Anyway.. the extended length of the outage today was mostly due to revamping the way we're doing the backups. We're working to include better query blocking (to ensure the database is totally update-free) and figure out the best way to maximize our time, thus ultimately shortening these outages.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Profile Dr. C.E.T.I.
Avatar
Send message
Joined: 29 Feb 00
Posts: 15993
Credit: 690,597
RAC: 10
United States
Message 802339 - Posted: 26 Aug 2008, 22:57:34 UTC
Last modified: 26 Aug 2008, 22:58:09 UTC

. . . nice work Matt - and that goes out to each of you @ Berkeley as well
____________
BOINC Wiki . . .

Science Status Page . . .

Profile Edward Lee Michau
Avatar
Send message
Joined: 31 Jul 06
Posts: 138
Credit: 9,640,846
RAC: 0
United States
Message 802348 - Posted: 26 Aug 2008, 23:22:21 UTC

Great work Matt. Glad to see you back. Hope your vacation was super.

Ed

BarryAZ
Send message
Joined: 1 Apr 01
Posts: 2580
Credit: 11,896,315
RAC: 3,604
United States
Message 802351 - Posted: 26 Aug 2008, 23:42:42 UTC - in response to Message 802336.

Matt, thanks for the update. A suggestion,update the weekly Tuesday notice --instead of 3 to 4 hours, have it be either 5 to 6 hours (which seems to be more accurate), or 3 to 6 hours (which allows for the relatively rare situation where nothing extra is scheduled and everything runs smoothly). It is all about setting expectations.

Ah, yes - here we go again - the regular Tuesday outage for mysql database backup/compression and other tasks better suited to happen during "quiescent" time.

For example, this week we replaced the failed drive in the workunit storage server with a new drive. That was painless. We also spent a bunch of time experimenting with the new-ish RAID server. I say "new-ish" as it's new to us, but it is an old system. For example, it can't handle logical volumes greater than 2TB. We however today confirmed (a) it can handle physical single drives at least 750GB in size, and (b) physical volumes greater than 2TB (i.e. put three 750GB drive together to make a 1.5TB RAID5).

We also tested that this system is keeping up pretty well doing a continual backup of our upload directory. That is, we're doing a constant rsync with the upload directory to keep a "hot backup" around on a separate system. We didn't have the bandwidth/storage capacity to do this ourselves before (and daily backups to tape were too expensive).

Anyway.. the extended length of the outage today was mostly due to revamping the way we're doing the backups. We're working to include better query blocking (to ensure the database is totally update-free) and figure out the best way to maximize our time, thus ultimately shortening these outages.

- Matt


____________

DJStarfox
Send message
Joined: 23 May 01
Posts: 1040
Credit: 539,987
RAC: 567
United States
Message 802394 - Posted: 27 Aug 2008, 1:37:36 UTC - in response to Message 802336.
Last modified: 27 Aug 2008, 1:38:51 UTC

No one made this comment yet... I just wanted to say great work getting the assimilation queue down to less than 100k. From what I've seen, it's been a backlog for a while until now. That should help storage space a little too, I'd imagine.

PS And I hope donations from Bluff's drive last week make some kind of difference we can see.

Profile [B^S] madmac
Volunteer tester
Avatar
Send message
Joined: 9 Feb 04
Posts: 1138
Credit: 3,510,901
RAC: 4,028
United Kingdom
Message 802466 - Posted: 27 Aug 2008, 7:00:41 UTC

Thank you for this information, now I have a problem I have a small knowledge of computers. So I was wondering if you we can have this with laymans language, so that people like myself can truly understand what a great job you are doing and what you are doing every week to improve this wonderful system.
____________

Profile Keith T.
Volunteer tester
Avatar
Send message
Joined: 23 Aug 99
Posts: 738
Credit: 231,168
RAC: 0
United Kingdom
Message 802681 - Posted: 28 Aug 2008, 0:32:32 UTC - in response to Message 802466.

Thank you for this information, now I have a problem I have a small knowledge of computers. So I was wondering if you we can have this with laymans language, so that people like myself can truly understand what a great job you are doing and what you are doing every week to improve this wonderful system.


@ madmac

I think Matt does already post mostly in understandable language, after all this is the "Technical News" forum.

May I suggest that if you don't understand a particular technical term, you could try looking it up on a site like http://www.whatis.com or http://www.wikipedia.org or Google etc.

Regards, Keith
____________
Sir Arthur C Clarke 1917-2008

Message boards : Technical News : Tuesday Edition (Aug 26 2008)

Copyright © 2014 University of California