The Reenactment (Mar 06 2012)

Message boards : Technical News : The Reenactment (Mar 06 2012)
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 1203152 - Posted: 6 Mar 2012, 22:39:37 UTC

Yesterday (Monday) there were an emergency generator test which affected the whole lab. Even though this test was mostly for the benefit of another project here at SSL, this still meant we had to power everything down, wait for the test to complete, then power everything back up. For the most part it went okay, but we had a few casualties on the way back up. A small subset of outlets on the back of one of our UPSes failed (a broken internal breaker?) - not a big deal. Dan's cheap desktop system also mysterious died, and won't power on anymore. That's a worse problem, but not a showstopper. However our scheduling server, bane, failed to boot. This seemed like an OS install problem, even though we had successfully rebooted it before after upgrading it the other day.

Luckily we have synergy in our racks, and it took me less than 10 minutes to configure it as a replacement scheduling server. But before I take any pride in that feat I admit that some internal server errors were getting lost in the noise upon bringing the projects back on line. Turns out a max request size setting for mod_fcgid was high by default in the older OS on bane, but not as high by default in the newer OS on synergy, so we needed to set that explicitly by hand. Fair enough, but all evening a set of crunchers were finding it impossible to connect and get work. I fixed that this morning before the standard weekly outage.

Also it should be noted a bookkeeping cronjob running on bane (now missing with bane out of commission) caused the splitters to run out of work overnight as well. This also was fixed this morning. We should be more or less back to normal after we catch up for a bit. Sorry about all the workflow hiccups.

Meanwhile, what's up with bane?! I spent half the day today installing and resintalling the OS, thinking I'm getting on top of the problem each time, but nope. Seems like the Fedora 16 installer has some issues in general, compared to previous versions. Yeah, I know, we should be using . I'll keep kicking it - though we'll probably keep the scheduler on synergy, and hopefully use bane to replace a much larger, less efficient administrative system in the closet.

New server-wise, I just checked the tracking info. Looks like they will arrive on Thursday.

- Matt

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 1203152 · Report as offensive
Profile Jim_S
Avatar

Send message
Joined: 23 Feb 00
Posts: 4705
Credit: 64,560,357
RAC: 31
United States
Message 1203154 - Posted: 6 Mar 2012, 22:43:59 UTC

Thanks for the News Matt...Both the Good and Bad of it.

I Desire Peace and Justice, Jim Scott (Mod-Ret.)
ID: 1203154 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1203290 - Posted: 7 Mar 2012, 9:09:01 UTC
Last modified: 7 Mar 2012, 9:10:34 UTC

Matt......
I hope you have been informed by now that things are NOT flowing nicely.
Lot of crickets available, not as much work going out as should be.

Perhaps Synergy is being taxed by so many tasks allocated to it?
Or some more settings needing tweaking.

Whatever the case, work is not flowing as it should be.

Thank you if you can find the answers.
And hopefully the new hardware inbound will ease your duties, once properly commissioned.

And meow from the kitties for your hard work, my friend. Not unappreciated.

Mark
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1203290 · Report as offensive

Message boards : Technical News : The Reenactment (Mar 06 2012)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.