The SnapApp folks were here for most of the afternoon. They replaced
the processors with ones that are faster and have larger caches. More
importantly, we talked about how to stabilize the system. The plan is
to simplify both the HW and service mix as much as possible in the hopes
of removing the variable that is causing the hangs (and of course they will
be going for the root cause at their lab). We altered the startup
so that a bare minimal set of services are running. And we are going to
only use the 18000 head unit for now, keeping no data on the sd30 expansion.
This means moving some data around. We first have to move the upload/download
data to a temporary volume on the sd30, then reconfigure the space on the
18000 head, then move the uldl to the head. The move to the temp volume
is happening now. Tomorrow morning we will do the reconfiguration and
start final move. With luck we will have the project up by EOD tomorrow.