Carolyn = Master (Nov 23 2010)

Okay then - after some extreme DBA this morning carolyn is now the master mysql database server and jocelyn is the replica. So that project is officially DONE! Actually, there's a lot of low-priority cleanup to deal with, but all the main plumbing is working and the projects are back up such as they are.

Now all server side focus is on oscar. By far the most important thing to fix during this major long outage was our science database - getting a new mysql database rolling was just icing on the cake. But I guess we still need to finish making the cake. Most of our i/o bottlenecks over the past few years have been somehow linked to thumper (both as a database and file server) so getting this done is essential before we get back on line.

Jeff found a comprehensive list of missing spikes (which I mentioned yesterday) and will begin inserting those. We'll then eat some turkey, then have an all-hands-on-deck week next week to get oscar going. We simply cannot get back on line before then, and so we're still looking at new workunits being generated a couple weeks from today at the earliest. I guess if we're really lucky it'll be sooner, but highly doubtful. I know we're anxious to get rolling again, but remember that when you're dealing with billions of rows of data (in the form of a terabyte of raw files), each step takes many hours no matter how clever you are or how fast you type. It's also easy to get lost in theoretical maximum speeds, which never take into account (a) the dizzying array of initial preparations before even starting, (b) actual speeds, (c) the many extra steps necessary when being careful (like backing up a database one last time before dropping a table containing a billion rows), and (d) unpredictable software/hardware behavior requiring us to go back to N steps in the cookbook and try again.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

May I congratulate you all on a job well done and on going! We are all proud to have helped you get new servers, and appreciate you giving us these valuable updates. You are all truely experts in your fields, and we will be here when the gates are opened. Thank you very much for your hard work and dedication!

Steve
____________
Warning, addicted to SETI crunching!
Crunching as a member of GPU Users Group.
GPUUG Website

Thanks for the update Matt. Take the time needed to get it right the first time. Maybe then the staff can all take a complete Christmas New Years break and not have to run in one day to reboot.


____________

Keep up the good work, you guys may as well try and launch the shuttle with all the trouble... it must seem like it.
____________

I recommend Secunia PSI: http://secunia.com/vulnerability_scanning/personal/
Go Georgia Tech.

Thanks for the update Matt, keep up the good work,

Claggy
____________

Matt, thanks for the news!

____________


>Das Deutsche Cafe. The German Cafe.<

Extreme Database Administration
____________

Extreme Database Administration

I´m just wondering.

____________

"Dropping a table with a billion rows" - it really doesn't matter if its a toy US billion, or a real UK billion, that's a big step. Lots of fingers crossed and rabbits' feet grasped, plus another backup.....
When you see these figures it makes one realise why this job has taken so long. Hopefully in a couple of weeks w will see the new server twins in action.
____________
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

I'm hoping any ventilation issues would be resolved before the servers come fully online. I'd keep it safe and make both servers very well ventilated and not on top of each other like in the final picture you took a few posts back.
____________
What if Fiction was Fact and Fact was Fiction and vice versa?

Full AirConditioning is the only way to keep a SERVER RACK cooled.
Especially when it has to be funtional 24 x7, with over a million hosts
connected to them, constantly Reporting and asking for different types of work!

____________


Knight Who Says Ni N!, OUT numbered.................