Loss Leader (Dec 16 2008)

First and foremost, it's snowing outside. This doesn't happen very often around here.

So today was an outage day - with one unexpected surprise: a visit from Court, systems administrator extraordinaire here in our lab a couple years back. Nice to see him again and catch up.

The standard outage stuff was, well, standard. Allow me to remind our new readers: Weekly we "compress" the mysql databases (which bloat from continual inserts/deletes all week, much like disk fragmentation) and back them up. These databases contain all the user/host/team info, and who is working on which workunits - basically all the generic volunteer computing stuff. The science is all kept in a separate database (using an Informix engine) on a different server altogether. The latter doesn't suffer from the same bloat, so we can do simple no-frills backups to disk while the database is live, without much ado. In theory we could do the mysql dumps live as well, but we choose to take things down to ensure the master/replica databases are in sync, and allow us some regular downtime to take care of pending server tasks. For example...

Today we finally turned off the old Network Appliance - a NAS server which worked fast and wonderfully, but (a) was only 3 Terabytes raw storage, (b) took up one third of our server closet, and (c) the individual disks have been failing at an increasing rate. We moved all of its functionality elsewhere already, so it was time to say goodbye. Jeff and I tore it apart shelf by shelf. Any sadness was lost in the joy of now having a completely empty rack full of completely useful shelves (we've had ridiculous problems finding racks/shelves that matched in the past). It's kind of funny the most useful part of that system at this point was its racks/shelves. We put all the recently donated Overland Storage servers into this now-empty rack (containing 10 Terabytes worth of storage), as well as anakin (the scheduling server), and there's still room for a lot more stuff. We still have to configure/employ all this new storage, but it's all plugged in and on line at least.

Recovery from the outage is usually painful. Today seems a little worse. Part of that is our work-to-send queue is at zero and the splitters are waiting for some space to free up before creating new work. I also think server "bruno" is having result storage issues slowing things down (people are connecting okay, which they can push through the usual traffic jam). We might need to reconfigure/rebuild that RAID array sooner than later.

I brought the mini video camera to make a quick video tour of our server closet, but the noise of all the fans is so loud it's basically worthless. I did take some low-quality still photos though - I'll get those up on the web someday.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Keep up the good work. I may be seeing that snow in a few hours down here in LA.

____________

Hi Matt, thanks for your update and work .It is nice that more raw storage is available.
It's not snowing, only -2C and it's gonna rain, temps will rise.

____________


Knight Who Says Ni N!, OUT numbered.................

Good work as always and Thank you for at least spending some of your time recording the video / images even though the video is worthless.

Saw on the news that you guys were getting snow, probably goes with the very very cold temperatures with windchill of -50 that the western Canadian provinces are currently getting.

Here on the East coast of Canada, Montreal area, we are just waiting on 15+ cm's of snow coming tomorrow.

Thank you for your information and glad you managed to get a lot done today.
____________

Thanks for the quick background info on what happens on outage day. I had known that things ‘went down’ for a few hours on every Tuesday but never really understood why. (I probably still have no real idea, but your disk-defrag simile is something that even us less technical crunchers can understand.)

Also, on behalf of many others, we appreciate your occasional posting of pictures in the about/picture albums section. I check it out from time to time and find that it helps put a human face on things. :)
____________

True, my apology. Still it is going to snow and much needed to help the huge ice rink I spent a good part of today making.
____________

I brought the mini video camera to make a quick video tour of our server closet, but the noise of all the fans is so loud it's basically worthless.

I brought the mini video camera to make a quick video tour of our server closet, but the noise of all the fans is so loud it's basically worthless.

As long as your lips aren't on camera who is to know?

Make a call over to your TV people on campus and ask them if you can borrow a "Lav" short for lavaliere.

I am British with Canadian Permanent Residency so technical it doesn't really matter which part of Canada it is in :-)

Only thing that matters, is that Quebec is part of Canada (albeit for the foreseeable future at least).

Really is quite annoying though doing computing jobs and having to switch languages all the time, you get used to it though over time. I have one client who insists on sending messages to me and my colleagues in french, when it quite clearly states that all correspondents to the server admins be in ENGLISH. He pays well though so can't complain. Someone attempting to run a DB with no knowledge of IBM Dynamic Servers (Informix based) is really annoying, worst of all the guy gets paid to do it and yet sub-contracts the work out!
____________

It's snowing in Vegas, and sticking! Global weirding, here we come...