One Last Note... (Dec 20 2012)

One more quick update before the apocalypse. Or holiday week off. Or whatever.

We seem to be still having minor headaches due to fallout from the power failures of a couple weeks ago. The various back end queues aren't draining as fast as we'd like. We mostly see that in the assimilator queue size. We recently realized that the backlog is such that one of the four assimilators is dealing with over 99% of the backlog - so effictively we're only 25% as efficient dealing with this particular queue. We're letting this clear itself out "naturally" as opposed to adding more complexity to solve a temporary problem.

I did cause a couple more headaches this morning moving archives from one full partition on one server to a less full partition on another. This caused all the queues to expand, and all network traffic to slow down. This is a bit of a clue as to our general woes. Maybe there's some faulty internal network wiring or switching or configuration...?

On a positive note we have carolyn (which is now the mysql replica server) on UPS and tested to safely shut down as soon as it's on battery power. So this will hopefully prevent the perfect storm type corruption we had during the last outage. At least we'll have one mysql server synced up and gracefully shut down.

Okay. See you on the other side...

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Have a happy and safe end of world.

____________

Thanks for the news, Matt.
It really is good to have you around to try to dig into some of the 'hidden' gremlins that some of us have always suspected were throwing a wrench into things behind the obvious scenes.

Wish you all success in finding and sorting them.

In whatever kind of world we have tomorrow...LOL.
____________
******

"Ask not, what your kitty can do for you. Ask what you can do for your kitty."

As it is kitten, so shall it be done.

Thanks for the update Matt, have a merry christmas and a happy new year,

Claggy

Thanks for the update, Matt. A Merry Christmas and a Happy New Year to you, all the other personnel at the Lab, and to all my fellow crunchers. It's been a good year for science and I was lucky enough to play a small part in it. Now we've found a potentially-habitable planet just 12 light-years away -- can anyone invent instantaneous teleportation over that distance? (No, without proving Einstein wrong... :-( )
____________

Thanks for the news Matt, to you and everyone in the lab, have a very Merry Christmas and a Happy New Year!
____________
In an alternate universe, it was a ZX81 that asked for clothes, boots and motorcycle.

Beer/wine o'clock, the best of the o'clocks.

Humpty dumpty sat on a wall, along came a giant, and cooked him for breakfast.

As usual Matt, very many thanks for the update, it is appreciated. But I did catch your UPS comment.

On a positive note we have carolyn (which is now the mysql replica server) on UPS and tested to safely shut down as soon as it's on battery power. So this will hopefully prevent the perfect storm type corruption we had during the last outage. At least we'll have one mysql server synced up and gracefully shut down.

...
Looks like I'm not getting as many units total in my local buffers which is fine since replacements are coming quickly after uploading.

Is limiting the number of units allowed to each machine part of the fix for the database overload that was going?
...

As usual Matt, very many thanks for the update, it is appreciated. But I did catch your UPS comment.

On a positive note we have carolyn (which is now the mysql replica server) on UPS and tested to safely shut down as soon as it's on battery power. So this will hopefully prevent the perfect storm type corruption we had during the last outage. At least we'll have one mysql server synced up and gracefully shut down.

I truly think that ALL the Seti servers should be on a similar UPS system. A New Year fundraiser for the GPUUG seems to be beckoning .....

In the meantime, may I wish you and the other guys in the lab, a very happy Christmas and a peaceful New Year. You've earned it!

Everything is on a UPS. However, as has been explained, it's not that easy. Different processes, running on different machines, have to be stopped in a specific order to avoid all the corruption that occurred last time. That requires either someone to be there to do it, or (if it's even possible) a very complex script overseeing all the shutdowns.

The issue was the mains at the lab are a bit flaky. So it was shutting down all the time on momentary brownout conditions. To restart after a shutdown someone has to actually be there.

1. You have a 'Server Status' page with a lot of very good information. I suggest you change it to a 'Systems Status' page and include some networking throughput details as well as the server status and splitter status sections. You already have 'Results received in last hour', but it appears to me your network issues would be better spelled out in Kb/s in and out, or something like that, maybe separated into different types of data.....

This raises another question. Why so much more data in than out? One would think the downloads from the servers would be higher than the uploads, since the download package sizes are so much larger than the uploaded results.

So, why not include the summary data, not the graph, on the Status page?

This raises another question. Why so much more data in than out? One would think the downloads from the servers would be higher than the uploads, since the download package sizes are so much larger than the uploaded results. Update queries?