Back to the Grind (Jan 03 2011)

Message boards : Technical News : Back to the Grind (Jan 03 2011)
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 1063163 - Posted: 3 Jan 2011, 22:12:30 UTC

Happy New Year!

We seem to have been running rather smoothly since last I wrote. And the servers were nice enough to wait until we were all back in the lab before going crazy. Well, it's not that bad - just lots of tiny problems. The astropulse science database server got stuck in some deadlock and needed a hard power cycle. Then I tried fixing this nagging db_dump problem (it hasn't regularly generating daily stats dumps) by moving the process from bruno to lando, and lando couldn't handle it - and I ended up needing to hard power cycle lando as well. Having lando reboot caused bruno to freak out a little bit as it was in the middle of a package update when lando disappeared out from underneath it. So I had some rpm database cleanup to deal with to get bruno to start upload results again. Oy!

Meanwhile we were bringing services up and down in a controlled manner to make some more science database tweaks on oscar. We're still deep into trying to figure out how to improve its throughput in general. We're finding the disks are still a bottleneck (even after the RAID restripe), but if we can get informix to cache more efficiently then disk i/o is less important. Or we'll add more disk spindles. In the meantime we have many knobs to turn.

I got the tracking # for new donated server "synergy" - should be here on Friday afternoon, so we'll start playing with it next week!

Regarding the weekly outages: we're sticking to the general idea which I mentioned recently - generally have our standard Tuesday half-day outage, and perhaps other planned one or two day outages as needed (with some effort to provide ample advance warning), but otherwise leave all public facing data services up and running as much as possible. However in the first sign of trouble, these services may be taken down for extended periods. The bottom line is you probably won't notice any difference between current operations and those from six months ago (before the weekly extended outages). The goal is to aim for 24/7 uptime while maintaining staff sanity.

- Matt

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 1063163 · Report as offensive
Profile Gary Charpentier Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 25 Dec 00
Posts: 30981
Credit: 53,134,872
RAC: 32
United States
Message 1063196 - Posted: 3 Jan 2011, 22:56:49 UTC

Happy New Year and thanks for the updates and please keep sane!

ID: 1063196 · Report as offensive
Pete Church
Volunteer tester
Avatar

Send message
Joined: 1 Sep 08
Posts: 7
Credit: 6,827,500
RAC: 0
United States
Message 1063349 - Posted: 4 Jan 2011, 6:18:17 UTC - in response to Message 1063163.  

Yes sanity==good and you guys are bending over backwards to keep things going. Hang in there!
ID: 1063349 · Report as offensive
Profile Brant

Send message
Joined: 10 Dec 99
Posts: 13
Credit: 3,529,140
RAC: 36
United States
Message 1063371 - Posted: 4 Jan 2011, 9:33:52 UTC

Happy new year to the whole crew!!!!

You are still down for maintenance far less than I am, so its all good here. Science first the rest is just that //the rest//.




Brant
ID: 1063371 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22504
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1063373 - Posted: 4 Jan 2011, 9:45:40 UTC

Staff sanity - good luck.

All the best for the New Year.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1063373 · Report as offensive
Dave

Send message
Joined: 29 Mar 02
Posts: 778
Credit: 25,001,396
RAC: 0
United Kingdom
Message 1063728 - Posted: 5 Jan 2011, 18:53:36 UTC

Worth remembering some of that for the subsequent moaners we will encounter as time goes on.
ID: 1063728 · Report as offensive
Profile perryjay
Volunteer tester
Avatar

Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 20,676,751
RAC: 0
United States
Message 1063733 - Posted: 5 Jan 2011, 18:59:42 UTC

Sanity is so overrated!! :-)


PROUD MEMBER OF Team Starfire World BOINC
ID: 1063733 · Report as offensive
Olli

Send message
Joined: 25 Apr 07
Posts: 143
Credit: 2,089,162
RAC: 0
Finland
Message 1063822 - Posted: 6 Jan 2011, 1:34:28 UTC - in response to Message 1063163.  

To remind me, what task will Synergy be assigned to?
ID: 1063822 · Report as offensive
Profile Ozmoses
Volunteer tester
Avatar

Send message
Joined: 9 Jun 03
Posts: 20
Credit: 31,926,513
RAC: 4
Australia
Message 1063935 - Posted: 6 Jan 2011, 9:52:35 UTC

Good luck with Sanity and 24/7 up time.

Cheers,

Chris

ID: 1063935 · Report as offensive
Profile WHOSIT
Avatar

Send message
Joined: 12 Nov 10
Posts: 29
Credit: 162,958
RAC: 0
United States
Message 1064815 - Posted: 9 Jan 2011, 0:54:23 UTC

I think everyone aims for 24/7 up time. Never down time. No matter how hard the team or anyone strides for it, at this point in technological time, it's just going to happen sooner or later, so lots o' luck with keeping everyone sane all the time as well.

Besides. . . All that constant sanity is sooo boring anyway. People should try inserting some unharmful insanity once in a while into their always serious lives. It helps unclog the old brain's drain. ;-)
{Just keep the water balloon & soaker fights confined to safe areas while inside buildings. :-)}
Wishing you continued science "fun" and at least, always valid results,
Laters,
Rick "WHOSIT" W.
Participating in: Einstein/MilkyWay/Rosetta/SETI
ID: 1064815 · Report as offensive
Profile KWSN THE Holy Hand Grenade!
Volunteer tester
Avatar

Send message
Joined: 20 Dec 05
Posts: 3187
Credit: 57,163,290
RAC: 0
United States
Message 1065012 - Posted: 9 Jan 2011, 19:02:00 UTC - in response to Message 1064815.  


Besides. . . All that constant sanity is sooo boring anyway. People should try inserting some unharmful insanity once in a while into their always serious lives. It helps unclog the old brain's drain. ;-)
{Just keep the water balloon & soaker fights confined to safe areas while inside buildings. :-)}


...And keep all that water away from the server closet!
.

Hello, from Albany, CA!...
ID: 1065012 · Report as offensive
Profile WHOSIT
Avatar

Send message
Joined: 12 Nov 10
Posts: 29
Credit: 162,958
RAC: 0
United States
Message 1065201 - Posted: 10 Jan 2011, 8:57:41 UTC - in response to Message 1065012.  


Besides. . . All that constant sanity is sooo boring anyway. People should try inserting some unharmful insanity once in a while into their always serious lives. It helps unclog the old brain's drain. ;-)
{Just keep the water balloon & soaker fights confined to safe areas while inside buildings. :-)}


...And keep all that water away from the server closet!


Oh heck yeah! Exactly the type of area which wouldn't be safe and would most likely cause permanent insanity.

"The horror. The horror." :-)

Wishing you continued science "fun" and at least, always valid results,
Laters,
Rick "WHOSIT" W.
Participating in: Einstein/MilkyWay/Rosetta/SETI
ID: 1065201 · Report as offensive

Message boards : Technical News : Back to the Grind (Jan 03 2011)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.