Back to the Grind (Jan 03 2011)


log in

Advanced search

Message boards : Technical News : Back to the Grind (Jan 03 2011)

Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1384
Credit: 74,079
RAC: 0
United States
Message 1063163 - Posted: 3 Jan 2011, 22:12:30 UTC

Happy New Year!

We seem to have been running rather smoothly since last I wrote. And the servers were nice enough to wait until we were all back in the lab before going crazy. Well, it's not that bad - just lots of tiny problems. The astropulse science database server got stuck in some deadlock and needed a hard power cycle. Then I tried fixing this nagging db_dump problem (it hasn't regularly generating daily stats dumps) by moving the process from bruno to lando, and lando couldn't handle it - and I ended up needing to hard power cycle lando as well. Having lando reboot caused bruno to freak out a little bit as it was in the middle of a package update when lando disappeared out from underneath it. So I had some rpm database cleanup to deal with to get bruno to start upload results again. Oy!

Meanwhile we were bringing services up and down in a controlled manner to make some more science database tweaks on oscar. We're still deep into trying to figure out how to improve its throughput in general. We're finding the disks are still a bottleneck (even after the RAID restripe), but if we can get informix to cache more efficiently then disk i/o is less important. Or we'll add more disk spindles. In the meantime we have many knobs to turn.

I got the tracking # for new donated server "synergy" - should be here on Friday afternoon, so we'll start playing with it next week!

Regarding the weekly outages: we're sticking to the general idea which I mentioned recently - generally have our standard Tuesday half-day outage, and perhaps other planned one or two day outages as needed (with some effort to provide ample advance warning), but otherwise leave all public facing data services up and running as much as possible. However in the first sign of trouble, these services may be taken down for extended periods. The bottom line is you probably won't notice any difference between current operations and those from six months ago (before the weekly extended outages). The goal is to aim for 24/7 uptime while maintaining staff sanity.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Profile Chris S
Volunteer tester
Avatar
Send message
Joined: 19 Nov 00
Posts: 29559
Credit: 8,988,698
RAC: 27,601
United Kingdom
Message 1063178 - Posted: 3 Jan 2011, 22:35:14 UTC

Happy New Year to you too Matt! And thanks for the update.

The goal is to aim for 24/7 uptime while maintaining staff sanity.


I think we could all raise a glass to that one, sounds like an excellent game plan!

____________
Damsel Rescuer, Kitty Patron, Uli Fan, Julie Supporter, CAMRA
ES99 Admirer, Raccoon Friend, IFAW, PETA, 5% Badge


Profile Gary Charpentier
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 11732
Credit: 5,969,877
RAC: 0
United States
Message 1063196 - Posted: 3 Jan 2011, 22:56:49 UTC

Happy New Year and thanks for the updates and please keep sane!

____________

Pete Church
Volunteer tester
Avatar
Send message
Joined: 1 Sep 08
Posts: 7
Credit: 6,827,500
RAC: 0
United States
Message 1063349 - Posted: 4 Jan 2011, 6:18:17 UTC - in response to Message 1063163.

Yes sanity==good and you guys are bending over backwards to keep things going. Hang in there!
____________

Profile Brant
Send message
Joined: 10 Dec 99
Posts: 11
Credit: 1,192,978
RAC: 135
United States
Message 1063371 - Posted: 4 Jan 2011, 9:33:52 UTC

Happy new year to the whole crew!!!!

You are still down for maintenance far less than I am, so its all good here. Science first the rest is just that //the rest//.




Brant
____________

rob smith
Volunteer moderator
Send message
Joined: 7 Mar 03
Posts: 7680
Credit: 44,947,970
RAC: 76,535
United Kingdom
Message 1063373 - Posted: 4 Jan 2011, 9:45:40 UTC

Staff sanity - good luck.

All the best for the New Year.
____________
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

Dave
Avatar
Send message
Joined: 29 Mar 02
Posts: 774
Credit: 23,193,139
RAC: 0
United Kingdom
Message 1063728 - Posted: 5 Jan 2011, 18:53:36 UTC

Worth remembering some of that for the subsequent moaners we will encounter as time goes on.

Profile perryjay
Volunteer tester
Avatar
Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 13,712,543
RAC: 12,357
United States
Message 1063733 - Posted: 5 Jan 2011, 18:59:42 UTC

Sanity is so overrated!! :-)
____________


PROUD MEMBER OF Team Starfire World BOINC

Olli
Send message
Joined: 25 Apr 07
Posts: 143
Credit: 2,089,162
RAC: 0
Finland
Message 1063822 - Posted: 6 Jan 2011, 1:34:28 UTC - in response to Message 1063163.

To remind me, what task will Synergy be assigned to?

Sten-Arne
Volunteer tester
Send message
Joined: 1 Nov 08
Posts: 3307
Credit: 16,350,021
RAC: 15,431
Sweden
Message 1063824 - Posted: 6 Jan 2011, 1:46:33 UTC - in response to Message 1063822.

To remind me, what task will Synergy be assigned to?


Afaik, that is yet to be decided.

____________

Profile Ozmoses
Volunteer tester
Avatar
Send message
Joined: 9 Jun 03
Posts: 16
Credit: 21,814,697
RAC: 8,709
Australia
Message 1063935 - Posted: 6 Jan 2011, 9:52:35 UTC

Good luck with Sanity and 24/7 up time.

Cheers,

Chris

____________

Profile WHOSIT
Avatar
Send message
Joined: 12 Nov 10
Posts: 29
Credit: 162,958
RAC: 0
United States
Message 1064815 - Posted: 9 Jan 2011, 0:54:23 UTC

I think everyone aims for 24/7 up time. Never down time. No matter how hard the team or anyone strides for it, at this point in technological time, it's just going to happen sooner or later, so lots o' luck with keeping everyone sane all the time as well.

Besides. . . All that constant sanity is sooo boring anyway. People should try inserting some unharmful insanity once in a while into their always serious lives. It helps unclog the old brain's drain. ;-)
{Just keep the water balloon & soaker fights confined to safe areas while inside buildings. :-)}
____________
Wishing you continued science "fun" and at least, always valid results,
Laters,
Rick "WHOSIT" W.
Participating in: Einstein/MilkyWay/Rosetta/SETI

Profile KWSN THE Holy Hand Grenade!
Volunteer tester
Avatar
Send message
Joined: 20 Dec 05
Posts: 1831
Credit: 7,563,067
RAC: 20,999
United States
Message 1065012 - Posted: 9 Jan 2011, 19:02:00 UTC - in response to Message 1064815.


Besides. . . All that constant sanity is sooo boring anyway. People should try inserting some unharmful insanity once in a while into their always serious lives. It helps unclog the old brain's drain. ;-)
{Just keep the water balloon & soaker fights confined to safe areas while inside buildings. :-)}


...And keep all that water away from the server closet!
____________
.

Profile WHOSIT
Avatar
Send message
Joined: 12 Nov 10
Posts: 29
Credit: 162,958
RAC: 0
United States
Message 1065201 - Posted: 10 Jan 2011, 8:57:41 UTC - in response to Message 1065012.


Besides. . . All that constant sanity is sooo boring anyway. People should try inserting some unharmful insanity once in a while into their always serious lives. It helps unclog the old brain's drain. ;-)
{Just keep the water balloon & soaker fights confined to safe areas while inside buildings. :-)}


...And keep all that water away from the server closet!


Oh heck yeah! Exactly the type of area which wouldn't be safe and would most likely cause permanent insanity.

"The horror. The horror." :-)

____________
Wishing you continued science "fun" and at least, always valid results,
Laters,
Rick "WHOSIT" W.
Participating in: Einstein/MilkyWay/Rosetta/SETI

Message boards : Technical News : Back to the Grind (Jan 03 2011)

Copyright © 2014 University of California