Get Out of My House (Jan 18 2011)

Nothing like coming back from a long holiday weekend and having one of your main production servers croak as soon as you arrive. It's a sunny day outside and I was stuck wearing my fleece jacket and fingerless gloves inside a well air-conditioned server closet.

So what happened? Not sure exactly, but bruno (the upload server, as well as the main boincadm administrative server) was all hung up as soon as we started the normal Tuesday outage. I had to reboot it, and that was that - it wouldn't come up properly again.

It seems to be a multiple-part problem. There was a disk failure, and the 3ware card in this system has always given us trouble. What kind of trouble? Well, if you reboot the system (without a full power cycle) random drives go missing. That's kind of a problem, no? I don't think this is a single broken card - a labmate has similar problems with the same model in his system (I forget the model #, but it's 24-channels). Anyway, the big RAID10 holding all the results was tagged as degraded and rebuilding now.

That's fine, except the OS (which is on separate partitions and not under the jurisdiction of the 3ware card) isn't booting either. Jeez! The good news is I can boot of a Fedora live CD and see both the root and upload storage drives, so there's no data loss. It just won't boot!

The other good news is that, if we need it, we have a backup system already: synergy! It might be getting pulled into prime time sooner than expected. It doesn't have nearly the large number of disk spindles as on bruno, but this might not be an issue - there's still plenty of disk space on it. And a lot of memory for potential file system caching. It's still undecided if we're going to make synergy the new bruno, but I'm at least copying everything there now just to be safe.

I might still be able to get bruno up this afternoon, but if not, looks like we're down for the evening (it'll take that long to copy everything over to synergy).

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Thanks for the update Matt. And I'll second that good luck!!!
____________

I Desire Peace and Justice, Jim Scott

Never been a big fan of the 3Ware cards of days past - they were bought by LSI to capture market share and are partnered with Intel so there is plenty of development budget behind them now.

It is good that there is a "spare" around that would be up to the task to take over where bruno left off.

I sense another fundrasier in our future - I still have access to another barebones server from Intel at a discounted price.

Todd
____________

Matt , Todd ;-) , thanks for the news!

____________


>Das Deutsche Cafe. The German Cafe.<

Thanks for the update Matt, good luck with Bruno,

Claggy

Good luck on Bruno and thanks for keeping us informed.

____________

That's some trouble, good luck, hope you can get Bruno going again.

By the way, no dice on bruno. I tried a bunch of things, it still won't boot - even though I can see all the drives/data when I boot from CD. And the RAID card refuses to NOT tag the RAID as degraded.

We're probably going to be down for at least another day or two. I'll post something to the front page shortly.

- Matt
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Time for the spring cleaning. Reseat all the connectors from head to tail. I know that will take a couple hours, especially with the the connectors in the drives. Don't forget the ground strap.

It that fails, I guess Synergy is Bruno.

In any case it sounds like you have the data and that is the important thing.


Or is Bruno jealous of Synergy?
____________