| Lengthy battle plans were drawn up. The day was upon us. The servers
were all shut down and we snapped into action. Behold the "Big Reorg." |
 |
The afternoon before the big event, Jeff and Matt did some
advance prep work, including removing the (unused) power supply
at the bottom of the rack of new Sun equipment to replace it
with a UPS. This simple procedure
turned ugly when one feisty screw (pictured here)
resisted removal until its core was completely drilled out.
Rendered weak and near death, it willingly surrendered. That'll learn it. |
 |
In the morning, bright and early, the re-org began. After shutting
down all the SETI@home services, we
moved the most important servers out of the closet and
into room 329. Here are Eric H and Eric K moving the NetApp filer. |
 |
Meanwhile, Matt got behind all the equipment and began disconnecting
anything and everything. What he's doing here is tracing the countless
SCSI cables connected to the servers so that he could
plug the peripherals (that will survive the re-org)
into the correct jacks later. |
 |
Here are the beginnings of the temporary server room. First
things first, we got
the web server up and running on the desk on the right
so users can at least see the web page and read the news
item regarding the current outage. Then we set
up the user database and workunit servers on the floor
with the Wyse terminal on top. |
 |
And here's the NetApp filer on the left, which houses
all the workunits to be shipped out.
Now all the servers were set to be powered up so we can ship workunits to our
data-starved users. Eric K and Dan are trying to
find out which outlets are on which breakers. Their effort was
for naught; the outlets were mislabeled and we blew a breaker
causing the NetApp to lose power. It survived the ungraceful shutdown. |
 |
This is Jeff fixing the Wyse terminal connection so we can see
the boot messages on the servers as we started them up.
They both came up without a glitch.
But we encountered another snafu:
All the ethernet jacks in the room were live, but
only one (of eight) connected to a 100 megabits/sec pipe (the other seven
were only 10 megabits/sec).
So for a short while the data server was choking on a mere
10 megabit bandwidth restriction. |
 |
Meanwhile, Matt was still in the closet taking everything apart,
and preparing the science database server to be completely gutted. |
 |
Matt opened up the science database server and vacuumed its interior which
was quite dusty. And with great vim, he plucked out these six
UltraSCSI PCI cards, no longer to be needed since we were getting
rid of those numerous, faulty, ugly external drive packs. Alas,
in his excitement, he removed all six cards, but should have only
removed four. They were easily replaced later. No harm, no foul. |
 |
These are the guts of the science database server, or at least some
of the guts. That's 4 CPUs on the upper right, and 10 PCI slots towards
the bottom. Towards the right (not pictured here) are 2.5 GB of RAM. |
 |
After getting the incredibly heavy science database server into the
rack, Steve worked on sliding in some Compaq disk arrays.
They have captive rack screws built in that are way thicker than
the standard size. We didn't realize this (having never tried
to put them in a rack before). Oh, well. |
 |
Eric H vacuumed the last remaining dust bunnies in the now-empty
server closet. Before that there was enough grit on the floor it
felt like you were stepping into a litterbox. Actually, it wasn't that bad. |
 |
Ah, the empty server closet - completely empty and totally silent.
Even the air conditioner was shut off. Time for lunch! |
 |
But first, Matt and Jeff, who spend the most time in this
normally cramped closet, took a moment to
enjoy the sudden influx of elbow room. |
 |
Back to work. We had to install a UPS in the bottom of the rack
of new Sun equipment. Here you can see the final result, which
doesn't seem like much, but that
UPS weighed 150 pounds and required three people (Jeff, Eric H,
and Matt) to get it in. That was a bear. |
 |
The equipment began making its way back into the closet. In the
corner is the master science server stack. The rest of the stuff
of the floor are UPS's of varying shapes (which we referred to
as blade, tissue box, and shoe box). Dan spent a long time mapping
out the power distribution of all the equipment in this closet,
and Jeff and Matt spent an equally long time
cautiously adhering to this plan - one false move
means a blown breaker or an overloaded UPS! |
 |
All the main servers were shut off again for the final push.
Since no users could see the web site or get any data the
pressure was on. First the data server slid into the closet,
then the new Sun rack, and then the NetApp filer.
Eric H and Steve, after getting the heavy and unwieldy
user database server into the top shelf, prepared to guide this
last big item in. |
 |
Jeff looked over the game plan to make sure everything was kosher.
Many notes were taken before, during, and after the whole ordeal.
A lot of these will be transformed into xfigs later and posted on
the closet doors. We like xfigs. |
 |
The final piece in position, Eric H and Jeff checked how easy it was
to squeeze back behind the racks. This was unlike the closet setup in
days of yore, where one misaimed footfall would cause you to topple over,
landing in a nasty pile of wires. Heavy computer equipment would then
be tugged off the table and onto your unprotected frame. Not so anymore!
Now several people, as evidenced here,
can slink back behind the rack with room to work without
resorting to contorting.
The white line in the foreground is an ethernet cable. |
 |
Here it is in all its glory!
But there was still work to be done.
Note the many cables cascading down the equipment on the left.
Another switch needed to be installed and more cables routed in and around
and through the racks. Almost there... |
 |
While Jeff was administering the final touches and turning machines
on one by one, Eric H, Paul, Dan, and Eric K discussed plans and
operations out in the hallway. Matt was busy cleaning up and taking
this picture. Very soon the whole project will be back on line! |
 |
Now that everything was up and LEDs were blinking, we turned out the
lights and stood in awe at the incredible spectacle before us.
A stunned silence fell upon all those who beheld this
newly renovated closet. One last photo was taken and the doors were
closed. Another chapter in the book of SETI@home complete. |
