Phew (Oct 05 2009)

Okay that was an ugly weekend. On Saturday morning I came to realize that our master mysql database server (mork) had crashed. I was the only one available at the time so I came up to the lab and rebooted the thing. We really need to improve our remote kvm/power cycle situation. I babysat the reboot long enough to see that mysql was recovering, knowing though that the replica would be out of sync (and need to be regenerated from scratch during the next weekly backup).

But then everything else crashed, and also hard enough to require human intervention. This time Eric eventually came up on Sunday to try to reboot a series of servers, but to no avail - they kept locking up shortly after reboot.

So Monday morning (today) we came into the lab and started cleaning up the server situation. Eric finally found the cause of the latter, if not all, of our problems. We have a pseudo user account is the "user" that runs a lot of stuff, apache processes, cron jobs, some of the BOINC back end servers, etc. For some reason the .history file had grown to 8GB in size, and it was full of garbage. Not sure why just yet, but that meant every time one of the above processes started, the shell tried to read in this impossibly large history file. Oops. Once Eric deleted this file all these dams broke free and we were able to safely recover all the databases/etc. throughout our long morning.

- Matt
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

We're using a mix of xfs and ext3. The filesystem in question where the .history file was located was xfs, but is always read/written over nfs (which was probably the main problem).

- Matt
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Hi!

Thank you for this info matt, but i want to change my CPU and my mainboard today, so i want load up my finished workunits.

Do you know, if its possible today, to load up the finished WU´s. I don`t want loose the WU`s through installing the operatingsystem new.

Best regards
____________

My answer from user-side:

Upload is working again, but very clogged (everyone wants to upload his/her 200 gpu-wu's)

Reporting them: same.
____________

Everything is working - upload, download, reporting. Just slowly. Everyone is trying to report two days worth of work and get more, all at the same time.

Matt, you might want to kill off the history for the pseudo-user if you don't need it:

export HISTFILE=

Either way, good detective work there and hope the rest of the recovery goes smoothly.
____________
Stats site - http://www.teamocuk.co.uk - still alive and (just about) kicking.

Hug Bear for Master Matt & Master Eric + crew *smile*
____________


N = R x fp x ne x fl x fi x fc x L

We're using a mix of xfs and ext3. The filesystem in question where the .history file was located was xfs, but is always read/written over nfs (which was probably the main problem).

Matt & Eric (and respective families). Thanks for your efforts over the weekend.
____________
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

I just upgraded to 6.6.38 why does it not show pending credits anymore?[/b] And a lot of the other functions are no longer working.
____________