Harvey (Mar 24 2009)

The good news is that our regular Tuesday maintenance outage today chugged along quickly, and without incident. The not-so-great news is that we are still fighting with thumper to get it running properly again.

Jeff, Eric, and I whipped up a cookbook yesterday of the 7 or 8 steps to get thumper's root drive mirrored. As of this morning we had only one working drive with root/boot on it, but it's the spare drive sitting in the /dev/sda slot. According to the BIOS, the root/boot drives have to be in slots #0 and #1, but thanks to non-linear disk controller labels on the backplane these drives show up in linux-land as /dev/sdy and /dev/sdac. Of course, you can only install grub on /dev/sd[a-d] which means lots of disk swapping and rebooting and resyncing.

However, we're still on step #2 right now, and it won't finish until later tonight. The three of us were huddled over thumper for almost three hours - a frustrating period of time starting with us rebooting thumper "just to make sure everything is working" and then it wouldn't mount the root drive because of underlying issues with the metadevice. This was all mysterious, and after poking this and that it got worse - we could only boot in recovery mode off of DVD, and we had to hack partition tables and change disk identifiers before we could see root again. That's where it's at now: we're syncing the one working drive with a new spare, a process that we thought would take less than an hour but will take five, apparently.

To add insult to injury our pulse table in the science database on thumper ran out of extents last night, which basically means the tables are full even though we have disk space available. So as if the above ordeal wasn't enough, we'll need an additional day or two to recreate (or at least hack at) the pulse table to add more extents. Long story short, don't expect SETI@home to be generating any new work or assimilating anything for a week (unless we're lucky). We'll at least try to keep Astropulse working during this time, so computers that can run Astropulse will be kept busy.

When it rains it pours, but we'll be back to normal again soon enough.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

OK, wild guess #1 for the root drives problem...

Can you specify in grub to reorder the IO ports for the disks to get sda, sdb, sdc, sdd to map into grub in sequence?

Using disk labels, you can then let Linux sort out the mount mess automagically later.

More of a question is where (which disks) to put the MBR (and redundant copies) for the BIOS to boot into...

Good luck!

(Anyone else ever juggled so many drives?)

Or... Set up a dedicated isoboot CD? Memory stick??

Regards,
Martin
____________
Mandriva Linux A user friendly OS!
See new freedom Mageia2
The Future is what We make IT (GPLv3)

If your RAID resync is going slow (check /sys/block/mdX/md/sync_speed, speed in KB/sec). You could try increasing it by tweaking
/sys/block/mdX/md/sync_speed_{min,max}

If you are only doing MB on a host, set to NNW instead.
____________

I have got it set to MB and Astropulse and have not got any of either work unit.
(Temporarily turned off MB to ease the burden and only set to AP)

Get the 3/25/2009 11:00:22 AM|SETI@home|Message from server: (Project has no jobs available) message when trying to request work.

Says there are units available just doesn't seem to be any being sent out. (Then again the server status is very lagged behind so could be the queue is empty)
____________

Not me, haven't got anything since yesterday. Giving the dual cores a nice earned rest until work becomes available again.
____________

the server status page is out of date, the numbers given there for available work are 25h old.

has someone actualy gotten ap's today?

the server status page is out of date, the numbers given there for available work are 25h old.

has someone actualy gotten ap's today?

yep, 7 APs - all resends - nothing newly split.

the server status page is out of date, the numbers given there for available work are 25h old.

has someone actualy gotten ap's today?

yep, 7 APs - all resends - nothing newly split.

The status page is now up to date. Creation rates for MB and AP_v5 reflect the amount of resends being created. Demand is much higher than that, of course, so it takes luck to get work.

Joe