Nested Problems (Feb 28 2007)

A day of nested problems, starting with Eric's desktop computer going on the fritz. Normally he would deal with it himself but he was out of the office. Usually when systems just suddenly crash without warning, my initial gut reaction is insufficient cooling. I checked out his system and sure enough found the hard drives inside (a pair, mounted in the front of the case away from all fans) were hot to the touch. Nobody had time to transplant any hardware - we just needed the system up and running. So I kept the case open and searched the lab for a table-top fan to blow air inside the system. We had two of them back up in my lab. One simply didn't work. Great. The other worked, but due to previous wear and tear immediately the blade flew off (not enough tension). I had to perform surgery to jury rig the thing back together. An uncommon use of bubble wrap was employed during this procedure. I brought the fan back down to Eric's office followed by a few minutes of following and yanking out dusty unused power cables (to free up an outlet) and placing the right upside-down garbage can on the right box to perfectly perch the fan to blow air right on the hot drives.

Oh yeah.. I'm supposed to be getting isaac back on line. The OS portion was more or less done yesterday, but the initial yum update took all night, pushing the remaining configuration to this morning. It was like pulling teeth getting mysql and httpd working. It's bad enough hammering out configuration problems. But at one point suddenly the ethernet stopped working and we had to figure that out. And then another point we suddenly couldn't mount the system which held the database backup. Oy. By mid afternoon I cobbled together enough functionality to turn the web site back on.

Meanwhile (there's always a meanwhile) there's a bunch of testing going on to figure out what's causing blips in our data (there are threads in the staff blog about all this). Jeff's been collecting test data the past day or so, but then the data recorder crashed last night. Turns out multiple instances of the data recorder process were running which caused the system to panic. My script (350 lines of perl) controls all the wacky logic to keep the thing running smoothly 24/7 without intervention. So I was called in to fix the damn thing. This was a simple tweak - it wasn't a bug as much as a new special case requiring different logic.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Happy SETI Birthday Matt, 8 years today, and nobody, 7 years.