Okay that was an ugly weekend. On Saturday morning I came to realize that our master mysql database server (mork) had crashed. I was the only one available at the time so I came up to the lab and rebooted the thing. We really need to improve our remote kvm/power cycle situation. I babysat the reboot long enough to see that mysql was recovering, knowing though that the replica would be out of sync (and need to be regenerated from scratch during the next weekly backup).

But then everything else crashed, and also hard enough to require human intervention. This time Eric eventually came up on Sunday to try to reboot a series of servers, but to no avail - they kept locking up shortly after reboot.

So Monday morning (today) we came into the lab and started cleaning up the server situation. Eric finally found the cause of the latter, if not all, of our problems. We have a pseudo user account is the "user" that runs a lot of stuff, apache processes, cron jobs, some of the BOINC back end servers, etc. For some reason the .history file had grown to 8GB in size, and it was full of garbage. Not sure why just yet, but that meant every time one of the above processes started, the shell tried to read in this impossibly large history file. Oops. Once Eric deleted this file all these dams broke free and we were able to safely recover all the databases/etc. throughout our long morning.

- Matt

---

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

That's just crazy annoying of a problem. Make sure HISTFILESIZE and HISTSIZE are set in all users' environments. Probably in the .bashrc or .bash_profile files, or whatever file is appropriate for the user's shell.

BTW, what shell are you using?

Anyway, glad it wasn't something more serious.

It looks like a fs corruption.

I'm curious which fs are you using on those machines?

---

We're using a mix of xfs and ext3. The filesystem in question where the .history file was located was xfs, but is always read/written over nfs (which was probably the main problem).

- Matt

---

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Thanks for answer. I suspect that. I use xfs for years and never have similar problem with it. But with nfs shares there was a lot strange problems from corrupted files to locks on nonexistent ones.

---

Hi!

Thank you for this info matt, but i want to change my CPU and my mainboard today, so i want load up my finished workunits.

Do you know, if its possible today, to load up the finished WU´s. I don`t want loose the WU`s through installing the operatingsystem new.

Best regards

---

My answer from user-side:

Upload is working again, but very clogged (everyone wants to upload his/her 200 gpu-wu's)

Reporting them: same.

---

Everything is working - upload, download, reporting. Just slowly. Everyone is trying to report two days worth of work and get more, all at the same time.

Matt, you might want to kill off the history for the pseudo-user if you don't need it:

export HISTFILE=

Either way, good detective work there and hope the rest of the recovery goes smoothly.

---

Stats site - http://www.teamocuk.co.uk - still alive and (just about) kicking.

Hug Bear for Master Matt & Master Eric + crew *smile*

---

N = R x fp x ne x fl x fi x fc x L

Thanks for the update Matt, you guys in Berkeley are great!!

---

We're using a mix of xfs and ext3. The filesystem in question where the .history file was located was xfs, but is always read/written over nfs (which was probably the main problem).

I'm always surprised that the system works as well as it does given your myriad cross mounts and dependence on nfs.

I've found nfs reads are usually fine but nfs writes on very busy systems can be fragile...

Would nfs over tcp help or would there be too much overhead? Network switch overload?...

Good luck,
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)

Would nfs over tcp help or would there be too much overhead? Network switch overload?...

I'm 80% sure they are running NFS v4 over tcp. Seems NFS sparked a long discussion in this forum a long while ago.

Matt & Eric (and respective families). Thanks for your efforts over the weekend.

---

Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

I just upgraded to 6.6.38 why does it not show pending credits anymore?[/b] And a lot of the other functions are no longer working.

---

it has been my experience that anytime the replica database is Offline - there is a lot of loss of system functionality. Currently it is Offline and has been so for days.As for when it will go Online - they say in another thread in this forum that it MAY be tomorrow.

---

Classic WU= 7,237 Classic Hours= 42,079