Continuous (Jan 12 2010)

We had an unexpected short outage early this morning. One of our internal file servers crashed, hanging everything. Jeff noticed it upon arrival at the lab this morning and kicked it (and the projects) back to life. Of course an hour later we had to bring everything back down again for the usual weekly maintenance (for mysql database compression and backup). The first outage caused a bit of delay, hence the extended length of what was otherwise a rather vanilla outage.

Jeff and I have been on a binge cleaning up the lab a bit, which has gotten overrun with cables, hardware, compact disc cases, etc. that we'll never need or use. Get it all out of here! Last week we uncovered an unlabeled box containing a motherboard and some RAM which would fit perfectly in this one tower Intel donated a year ago but never worked. So I spent a chunk of yesterday replacing this motherboard, only drawing blood a few times during the course of handling or maneuvering around all the unfortunately sharp heat sinks/solder joints/inner edges of the case. I also managed, while forcing the main power supply plug into the new board, to jam my right index finger down full force onto a set of exposed pins, one of which plunged a good half centimeter underneath my fingernail.

Did I ever mention how much I hate dealing with hardware?

Anyway... the new board works, but for some reason installing the OS on it has been a pain. I'm currently at attempt number five - all the problems stemming from the disk partition layout. This OS install worked perfectly on other systems, including the easy disk formatting GUI. Not sure why on this system I'm only able to create three primary partitions via the GUI, not four. I ultimately had to partition it myself in rescue mode to get it to behave how I wanted. Weird and frustrating. On top of that, the installer was logically swapping sdb and sdc, so when I placed a RAID on what I thought was sda/b it came up "missing" a drive and failed. Whatever. It's sort of working now. Not sure exactly what we'll do with it - probably just replace the slightly less powerful (and crashy) BOINC web server. Two CPUs, 8 GB of memory...

Meanwhile, some more bad news: we're having to backtrack a few steps in the secondary science database recovery project (on bambi). We were able to recover from the backup (a process that took a week or so) but the logical logs have since wrapped around. So we could recover from the continuous logical log backup, right? I mean, that's why we do the continuous logical log backup, no? Well, apparently we can't. Not sure why. So we're going to try to do the whole recovery/rebuild again in a manner that will hopefully take less than the time for those logical logs to wrap around (about 4 days). We'll see. Let me remind you this has zero effect on the public part of the project - well, except that astropulse is still kind of on hold until we're done. Yes, that's much greater than zero.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Thanks for the update, and all the hard work Matt and Jeff.

Claggy

Jeff and I have been on a binge cleaning up the lab a bit, which has gotten overrun with cables, hardware, compact disc cases, etc. that we'll never need or use. Get it all out of here! Last week we uncovered an unlabeled box containing a motherboard and some RAM which would fit perfectly in this one tower Intel donated a year ago but never worked. So I spent a chunk of yesterday replacing this motherboard, only drawing blood a few times during the course of handling or maneuvering around all the unfortunately sharp heat sinks/solder joints/inner edges of the case. I also managed, while forcing the main power supply plug into the new board, to jam my right index finger down full force onto a set of exposed pins, one of which plunged a good half centimeter underneath my fingernail.

Matt wrote: "Did I ever mention how much I hate dealing with hardware?"

As one from the hardware side, I recall the ancient mantra "Beware of programmers carrying screwdrivers."

PS: title: Continuous (Jun 12 2009) ehmm a small belated y2k hickup?

Time to get rid of your iPhone, central A/C, car (unless you are good with cars).... hell, I don't even know how to make ceramic coffee cups. My world would fall apart with coffee! LOL

When all humanity's technology breaks down, we start over with a farm and a hoe.

Actually, you would have to invent the hoe first. And metal working, unless it is a wooden hoe. Then you just need to invent stone tools and figure out how to use them for wood working.

We live in an age of specialization (it started a few thousand years ago). You need to learn a speciality of your own, then find other specialists you can trust, and that need your services.

To start from scratch you must first create the universe :D

But where would you get what the universe started from?

The "Tasks" list from the account page is two days old... the latest tasks on it were sent to me on Jan 17 (Sunday) (posted on the 19th, Tuesday...) (and I've gotten tasks on the 18th and 19th...)
____________
.

The Replica Database has been OFFLINE since about 01:00 on 18 Jan. I posted this yesterday in the Panic Mode (28) thread in NC. http://setiathome.berkeley.edu/forum_thread.php?id=58415&nowrap=true#964348

Yesterday was a Public Holiday in the USA.

[edit]
For those that did not know, Stats exports, User Data pages, and various other pages rely on the Replica Database.
[/edit]

Stats exports are back in business again as of ~1.5 hours ago (23:00 UTC) though as Matt noted in his TechNews post, the replica database is behind the power curve and playing catch up.
____________
Best wishes:)