The good news is that the scheduler is now running as a fast cgi
which means it maintains a persistent connection to the DB. The
load on the scheduling server has gone way down which is a good
thing and the mysql server is using more CPU which is also a good
thing. The bad news is that the transitioner is still way behind
and so the project will probably run out of send-able work as
early as 04:00 UT tomorrow. The transitioner is slow because the
mysql server is IO bound. We are having Sun look at our IO system
to see if it can be tuned to better performance. At the same time
we are planning on moving the physical DB to a high performance Snap
Appliance box in the near future. So we are hopeful that these IO
problems can be solved.