The "Big Reorg" - May 22, 2002


log in
Lengthy battle plans were drawn up. The day was upon us. The servers were all shut down and we snapped into action. Behold the "Big Reorg."

The afternoon before the big event, Jeff and Matt did some advance prep work, including removing the (unused) power supply at the bottom of the rack of new Sun equipment to replace it with a UPS. This simple procedure turned ugly when one feisty screw (pictured here) resisted removal until its core was completely drilled out. Rendered weak and near death, it willingly surrendered. That'll learn it.
In the morning, bright and early, the re-org began. After shutting down all the SETI@home services, we moved the most important servers out of the closet and into room 329. Here are Eric H and Eric K moving the NetApp filer.
Meanwhile, Matt got behind all the equipment and began disconnecting anything and everything. What he's doing here is tracing the countless SCSI cables connected to the servers so that he could plug the peripherals (that will survive the re-org) into the correct jacks later.
Here are the beginnings of the temporary server room. First things first, we got the web server up and running on the desk on the right so users can at least see the web page and read the news item regarding the current outage. Then we set up the user database and workunit servers on the floor with the Wyse terminal on top.
And here's the NetApp filer on the left, which houses all the workunits to be shipped out. Now all the servers were set to be powered up so we can ship workunits to our data-starved users. Eric K and Dan are trying to find out which outlets are on which breakers. Their effort was for naught; the outlets were mislabeled and we blew a breaker causing the NetApp to lose power. It survived the ungraceful shutdown.
This is Jeff fixing the Wyse terminal connection so we can see the boot messages on the servers as we started them up. They both came up without a glitch.

But we encountered another snafu: All the ethernet jacks in the room were live, but only one (of eight) connected to a 100 megabits/sec pipe (the other seven were only 10 megabits/sec). So for a short while the data server was choking on a mere 10 megabit bandwidth restriction.

Meanwhile, Matt was still in the closet taking everything apart, and preparing the science database server to be completely gutted.
Matt opened up the science database server and vacuumed its interior which was quite dusty. And with great vim, he plucked out these six UltraSCSI PCI cards, no longer to be needed since we were getting rid of those numerous, faulty, ugly external drive packs. Alas, in his excitement, he removed all six cards, but should have only removed four. They were easily replaced later. No harm, no foul.
These are the guts of the science database server, or at least some of the guts. That's 4 CPUs on the upper right, and 10 PCI slots towards the bottom. Towards the right (not pictured here) are 2.5 GB of RAM.
After getting the incredibly heavy science database server into the rack, Steve worked on sliding in some Compaq disk arrays. They have captive rack screws built in that are way thicker than the standard size. We didn't realize this (having never tried to put them in a rack before). Oh, well.
Eric H vacuumed the last remaining dust bunnies in the now-empty server closet. Before that there was enough grit on the floor it felt like you were stepping into a litterbox. Actually, it wasn't that bad.
Ah, the empty server closet - completely empty and totally silent. Even the air conditioner was shut off. Time for lunch!
But first, Matt and Jeff, who spend the most time in this normally cramped closet, took a moment to enjoy the sudden influx of elbow room.
Back to work. We had to install a UPS in the bottom of the rack of new Sun equipment. Here you can see the final result, which doesn't seem like much, but that UPS weighed 150 pounds and required three people (Jeff, Eric H, and Matt) to get it in. That was a bear.
The equipment began making its way back into the closet. In the corner is the master science server stack. The rest of the stuff of the floor are UPS's of varying shapes (which we referred to as blade, tissue box, and shoe box). Dan spent a long time mapping out the power distribution of all the equipment in this closet, and Jeff and Matt spent an equally long time cautiously adhering to this plan - one false move means a blown breaker or an overloaded UPS!
All the main servers were shut off again for the final push. Since no users could see the web site or get any data the pressure was on. First the data server slid into the closet, then the new Sun rack, and then the NetApp filer. Eric H and Steve, after getting the heavy and unwieldy user database server into the top shelf, prepared to guide this last big item in.
Jeff looked over the game plan to make sure everything was kosher. Many notes were taken before, during, and after the whole ordeal. A lot of these will be transformed into xfigs later and posted on the closet doors. We like xfigs.
The final piece in position, Eric H and Jeff checked how easy it was to squeeze back behind the racks. This was unlike the closet setup in days of yore, where one misaimed footfall would cause you to topple over, landing in a nasty pile of wires. Heavy computer equipment would then be tugged off the table and onto your unprotected frame. Not so anymore! Now several people, as evidenced here, can slink back behind the rack with room to work without resorting to contorting. The white line in the foreground is an ethernet cable.
Here it is in all its glory! But there was still work to be done. Note the many cables cascading down the equipment on the left. Another switch needed to be installed and more cables routed in and around and through the racks. Almost there...
While Jeff was administering the final touches and turning machines on one by one, Eric H, Paul, Dan, and Eric K discussed plans and operations out in the hallway. Matt was busy cleaning up and taking this picture. Very soon the whole project will be back on line!
Now that everything was up and LEDs were blinking, we turned out the lights and stood in awe at the incredible spectacle before us. A stunned silence fell upon all those who beheld this newly renovated closet. One last photo was taken and the doors were closed. Another chapter in the book of SETI@home complete.

Previous (Earlier) Album  -  Album Menu  -  Next (Later) Album

Copyright © 2014 University of California