The Reenactment (Mar 06 2012)

log in

Advanced search

Message boards : Technical News : The Reenactment (Mar 06 2012)

Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 1 Mar 99
Posts: 1441
Credit: 210,886
RAC: 1
United States
Message 1203152 - Posted: 6 Mar 2012, 22:39:37 UTC

Yesterday (Monday) there were an emergency generator test which affected the whole lab. Even though this test was mostly for the benefit of another project here at SSL, this still meant we had to power everything down, wait for the test to complete, then power everything back up. For the most part it went okay, but we had a few casualties on the way back up. A small subset of outlets on the back of one of our UPSes failed (a broken internal breaker?) - not a big deal. Dan's cheap desktop system also mysterious died, and won't power on anymore. That's a worse problem, but not a showstopper. However our scheduling server, bane, failed to boot. This seemed like an OS install problem, even though we had successfully rebooted it before after upgrading it the other day.

Luckily we have synergy in our racks, and it took me less than 10 minutes to configure it as a replacement scheduling server. But before I take any pride in that feat I admit that some internal server errors were getting lost in the noise upon bringing the projects back on line. Turns out a max request size setting for mod_fcgid was high by default in the older OS on bane, but not as high by default in the newer OS on synergy, so we needed to set that explicitly by hand. Fair enough, but all evening a set of crunchers were finding it impossible to connect and get work. I fixed that this morning before the standard weekly outage.

Also it should be noted a bookkeeping cronjob running on bane (now missing with bane out of commission) caused the splitters to run out of work overnight as well. This also was fixed this morning. We should be more or less back to normal after we catch up for a bit. Sorry about all the workflow hiccups.

Meanwhile, what's up with bane?! I spent half the day today installing and resintalling the OS, thinking I'm getting on top of the problem each time, but nope. Seems like the Fedora 16 installer has some issues in general, compared to previous versions. Yeah, I know, we should be using . I'll keep kicking it - though we'll probably keep the scheduler on synergy, and hopefully use bane to replace a much larger, less efficient administrative system in the closet.

New server-wise, I just checked the tracking info. Looks like they will arrive on Thursday.

- Matt

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Profile Jim_S
Send message
Joined: 23 Feb 00
Posts: 4627
Credit: 33,476,938
RAC: 23,508
United States
Message 1203154 - Posted: 6 Mar 2012, 22:43:59 UTC

Thanks for the News Matt...Both the Good and Bad of it.

I Desire Peace and Justice, Jim Scott (Mod-Ret.)

Profile Michel448a
Volunteer tester
Send message
Joined: 27 Oct 00
Posts: 1201
Credit: 2,891,635
RAC: 0
Message 1203164 - Posted: 6 Mar 2012, 23:54:59 UTC
Last modified: 6 Mar 2012, 23:55:43 UTC

thanks you Matt for the info.

too bad for Bane :( but i understood, it s not his fault. but a failed OS that doesnt like how Bane's personality.
and cause that, Bane will lost his job and will be laid-off :(

thanks Matt for all your work.

msattler - (meow)Project Donor
Volunteer tester
Send message
Joined: 9 Jul 00
Posts: 44562
Credit: 789,150,200
RAC: 216,935
United States
Message 1203290 - Posted: 7 Mar 2012, 9:09:01 UTC
Last modified: 7 Mar 2012, 9:10:34 UTC

I hope you have been informed by now that things are NOT flowing nicely.
Lot of crickets available, not as much work going out as should be.

Perhaps Synergy is being taxed by so many tasks allocated to it?
Or some more settings needing tweaking.

Whatever the case, work is not flowing as it should be.

Thank you if you can find the answers.
And hopefully the new hardware inbound will ease your duties, once properly commissioned.

And meow from the kitties for your hard work, my friend. Not unappreciated.

In memory of my beloved, faithful kitty companion, Tigger II.
9/17/01 - 3/25/16

Have made a few friends in life.
Most were cats.

Message boards : Technical News : The Reenactment (Mar 06 2012)

Copyright © 2016 University of California