Rainy Day (Jan 19 2010)

Long holiday weekend (it was Martin Luther King Day yesterday) during which no major snags, but a couple minor ones. The data pipeline ran dry, but Jeff got on top of that before most people noticed. The mysql replica also lost touch with the master and threw itself offline. This is an old problem that went away for a while, but is apparently back to annoy us. Not a big deal, except the alert e-mails kinda got "lost in the noise" of the holiday weekend and we didn't kick the thing back to life until this morning.

Meanwhile, we're having the usual weekly mysql maintenance outage. The replica caught up a bunch while we were offline today, but still may take a day or two to fully get back in sync. Until then, any queries made to that database will be slightly out of date. Fine.

In better news, this last iteration of the secondary science database recovery project seems to have worked, or at least working. It took 6 days as expected to fully back up and restore the secondary from the primary, which was expected, but this time we had enough logical logs on line so that they didn't "wrap around" during this process and we were forced to try to recover from continuous logical log backups. We tried recovery from the continuous logs last week - The bothersome thing is that should have worked. Anyway... the secondary up and doing the final stages of recovery now.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

That's good news for you guys.

Hopefully you had a relaxing day yesterday, which you so deserve.

Thank you as always for the updates.

As you mentioned in a previous thread, you guys had a tidy up of the lab, any chance of some new pictures of the server closet and your handy work :-)?

This will be my last post from this machine, it is about to move from a E6400 to a Q8200 (Dual to Quad Core), shame the GTX 260 that should have come with it, is not with the order and is on back order for 2 weeks. Nice new 750w Kingwin PSU as well.
____________