Quick note as I gotta catch a bus..

Wow - what a mess. I think we're in the middle of our biggest outage recovery to date, and it's breaking everything. The good news is we're coming into some newer hardware which we'll get on line to help somehow.

See Eric's thread in the Staff Blog. He's been working overtime getting a new frankenstein machine together to act as another upload/download server and reduce the load on bruno. The scheduling server (galileo) has been choking - I just now moved all that over to bruno as well. So we may retire galileo soon, too. Jeff has been going nuts trying to track down errors in validator/assimilator code so we can get those on line as well. And our old friend "slow feeder query" is back, probably just being aggravated by the heavy load.

Gotta go.. Thank you for keeping us informed. Hopefully things will be up and running soon Hugs QM

- Matt

Thanks for the continuing patience and encouraging sentiments since the science database server crashed over a week ago. Still waiting on the server replacement. I think we're all anxious for it to arrive already, but we originally expected it no earlier than late-in-the-day today.

We had the usual database backup outage, in case anybody noticed. Outside of the usual backup/compression of the BOINC database, I fixed the replica server, so that's back up and running again. I also rebooted our Network Appliance which has been complaining about "misconfigurations" as of late, but that didn't seem to help or hurt. We think a bad drive in the system is causing these errors. I then replaced a bad drive in the Snap Appliance so that's back to having two working hot spares (phew). Jeff, Eric, and I also cleaned up the lab. Entropy reigns supreme around here. The table which we sit around and eat lunch was full of miscellaneous screws, heat sinks, empty drive trays, shredded bubble wrap, etc. but not anymore.

- Matt