We discovered that the sheer number of files in the upload directory
was killing NFS performance for the file_upload_handler. We started
with a new empty upload directory and the file_upload_handler performed
much better. The rate of connection drops on the upload server dropped
to zero (most of the time). Of course we need to get all those older
results back into play somehow.
We are preparing our new SnapAppliance 18000 to host the database. We
need to fit it into the power load in our already fairly full data
closet, configure the DB volume, run some tests, and then copy over
the DB files. We will post the progress as we go.