The Reenactment (Mar 06 2012)


log in

Advanced search

Message boards : Technical News : The Reenactment (Mar 06 2012)

Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1390
Credit: 74,079
RAC: 0
United States
Message 1203152 - Posted: 6 Mar 2012, 22:39:37 UTC

Yesterday (Monday) there were an emergency generator test which affected the whole lab. Even though this test was mostly for the benefit of another project here at SSL, this still meant we had to power everything down, wait for the test to complete, then power everything back up. For the most part it went okay, but we had a few casualties on the way back up. A small subset of outlets on the back of one of our UPSes failed (a broken internal breaker?) - not a big deal. Dan's cheap desktop system also mysterious died, and won't power on anymore. That's a worse problem, but not a showstopper. However our scheduling server, bane, failed to boot. This seemed like an OS install problem, even though we had successfully rebooted it before after upgrading it the other day.

Luckily we have synergy in our racks, and it took me less than 10 minutes to configure it as a replacement scheduling server. But before I take any pride in that feat I admit that some internal server errors were getting lost in the noise upon bringing the projects back on line. Turns out a max request size setting for mod_fcgid was high by default in the older OS on bane, but not as high by default in the newer OS on synergy, so we needed to set that explicitly by hand. Fair enough, but all evening a set of crunchers were finding it impossible to connect and get work. I fixed that this morning before the standard weekly outage.

Also it should be noted a bookkeeping cronjob running on bane (now missing with bane out of commission) caused the splitters to run out of work overnight as well. This also was fixed this morning. We should be more or less back to normal after we catch up for a bit. Sorry about all the workflow hiccups.

Meanwhile, what's up with bane?! I spent half the day today installing and resintalling the OS, thinking I'm getting on top of the problem each time, but nope. Seems like the Fedora 16 installer has some issues in general, compared to previous versions. Yeah, I know, we should be using . I'll keep kicking it - though we'll probably keep the scheduler on synergy, and hopefully use bane to replace a much larger, less efficient administrative system in the closet.

New server-wise, I just checked the tracking info. Looks like they will arrive on Thursday.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Profile Jim_SProject donor
Avatar
Send message
Joined: 23 Feb 00
Posts: 4534
Credit: 19,068,475
RAC: 5,118
United States
Message 1203154 - Posted: 6 Mar 2012, 22:43:59 UTC

Thanks for the News Matt...Both the Good and Bad of it.
____________

I Desire Peace and Justice, Jim Scott (Mod-Ret.)

Profile Michel448a
Volunteer tester
Avatar
Send message
Joined: 27 Oct 00
Posts: 1201
Credit: 2,891,635
RAC: 0
Canada
Message 1203164 - Posted: 6 Mar 2012, 23:54:59 UTC
Last modified: 6 Mar 2012, 23:55:43 UTC

thanks you Matt for the info.

too bad for Bane :( but i understood, it s not his fault. but a failed OS that doesnt like how Bane's personality.
and cause that, Bane will lost his job and will be laid-off :(

thanks Matt for all your work.
____________

Message boards : Technical News : The Reenactment (Mar 06 2012)

Copyright © 2014 University of California