Happy March to one and all. Haven't have much to write about lately, but here's a round up.

We had our usual weekly maintenance outage today during which we took care of all kinds of stuff besides the usual mysql database compression/backup. Early this morning I noticed the replica mysql server had some broken tables, which led me to discover a drive had failed on that system last night - a 73GB fibre channel drive. Not a big deal, as we have tons of these kicking around from older servers at this point. This was easy enough to hot swap, though I got lost in some internal closet networking updates as this disk array is only accessible via telnet. And then the mysql daemon on the replica freaked out a little bit when the new drive was introduced, so I had to reboot the system, re-fix broken tables, etc. etc. etc. The replica is still catching up (will be for a while).

Today we also moved synergy off the probably-flakey UPS. Yeah, I know we should have done this earlier, but just haven't gotten around to it yet. If anything this gave us one more data point in the form of yet another automatic biweekly reboot at Sunday around 3pm (a couple days ago). Now the UPS is out of the equation, we have to wait 2 weeks to see if this was indeed the problem.

What else... we moved a lot more bits from ptolemy onto thumper. You may notice some general speedups on the website or elsewhere. We hope. And Jeff and I tackled a ton of timing tests for the science database on oscar. We're finding all the bottlenecks and finding ways around them. The good news is the database select throughput has gone from 100 spikes/second to 17,000 spikes/second. However these are under optimal conditions. In reality we'll have to deal with many of the aforementioned bottlenecks. Also: gowron is back to being the main workunit server (the full transition is far from complete, though).

That's been my day so far. How's your day?

- Matt

---

-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Keep in mind UPS's do need new batteries from time to time. ;)

---

Janice

Thanks for the update Matt,

Claggy

Thank you as always for the update Matt. Man Alive you all have a lot on your plate. It is always fascinating to read the stuff you do to keep us happily crunching away.

I do want to point out something on the Server status page (like you don't have enough things on the to do list.)

On the the bottom of the page there are definitions/explanations for Tasks ready to send, Tasks in progress, etc. but under the Data Distribution State at the top they are referred to as Results ready to send, etc. I think the Tasks terminology is spot on but the Results reference muddies the waters.

Thanks again to you and everyone at the Lab.

Kind Regards,

Tom

---

Thanks for the update and please insist Eric get the Beta Status page fixed before V7 work hits the masses.

---

Thanks for the update, and good job keeping everything in order over there! Since the big issues a few weeks ago, things have been looking pretty good. The weekly outages have been short, which is much appreciated by this hobbyist...

Thanks for the update Matt, Me I just have to pack for a move, Which is being covered in My thread in My sig.

---

Savoir-Faire is everywhere!
The T1 Trust, T1 Class 4-4-4-4 #5550, America's First HST

Thanks for the update Matt,

Claggy

Just replaced 2 of mine over the weekend - the UPS software was sensing something initiating shutdowns. randomly.

Just replaced 2 of mine over the weekend - the UPS software was sensing something initiating shutdowns. randomly.

I once had a problem where I was trying to communicate with some other device altogether via a serial cable and somehow the computer kept interpreting this as a shutdown command coming from the UPS. I think I disabled the UPS software until I was done with the other thing.

Thanks for the update and all your hard work, Matt. As for me, SSDD. I do notice, however, that my computer hasn't communicated with the project in about 30 hours now. This seems unusual, but I'm sitting on 20 WUs (and no Einstein WUs), so I won't worry about it for another day.

David

---

David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.

Keep in mind UPS's do need new batteries from time to time. ;)

Around every 2 years.

Also keep in mind that at least one well known UPS manufacturer sets it's default self test to "14 days". I know from experience (my company has 150+ sites in the UK all with one or more UPS) that sometimes the self test can cause the UPS to fail, without any actual error in the log.

---

Keep in mind UPS's do need new batteries from time to time. ;)

Around every 2 years.

Also keep in mind that at least one well known UPS manufacturer sets it's default self test to "14 days". I know from experience (my company has 150+ sites in the UK all with one or more UPS) that sometimes the self test can cause the UPS to fail, without any actual error in the log.

Indeed. Assuming you have machines with redundant power supplies, I like to split machines over multiple UPS's. You can still set up the UPS software to send shutdown notices for "real" power failure events, but when you have a self-test induced power off, it is instant and no shutdown messages are sent (in my experience). It's not fool proof, but it has saved me many times when I've had a UPS fail on our distributed file servers.

---

Keep in mind UPS's do need new batteries from time to time. ;)

And they really need to be exercised about once a month with a fairly deep discharge about twice a year.

---

Keep in mind UPS's do need new batteries from time to time. ;)

Around every 2 years.

Also keep in mind that at least one well known UPS manufacturer sets it's default self test to "14 days". I know from experience (my company has 150+ sites in the UK all with one or more UPS) that sometimes the self test can cause the UPS to fail, without any actual error in the log.

That's very interesting. I've never had one "burp" the attached equipment due to a test cycle. Even when it is a deep (80~90%) test cycle. All our UPS are APC.

---

Keep in mind UPS's do need new batteries from time to time. ;)

Around every 2 years.

Also keep in mind that at least one well known UPS manufacturer sets it's default self test to "14 days". I know from experience (my company has 150+ sites in the UK all with one or more UPS) that sometimes the self test can cause the UPS to fail, without any actual error in the log.

That's very interesting. I've never had one "burp" the attached equipment due to a test cycle. Even when it is a deep (80~90%) test cycle. All our UPS are APC.

I recently had to replace the battery in an APC Smart-UPS 720. It was failing its 14-day self-test...

Function: Automatic Self-test
Factory Default: Every 14 days (336 hours)
User Selectable Choices: Every 7 days(168 hours), On Startup Only, No Self test
Description: Set the interval at which the UPS will execute a self-test.

"During the self-test, the UPS briefly operates the connected equipment on battery."

Wow! I am frequently amazed at the interesting stuff posted on these forums. Thanks for all the helpful info.

Whit

Assuming you have machines with redundant power supplies, I like to split machines over multiple UPS's.

Where I work we were told to not put the redundant power supplies on different phases - something about having 415 volts of potential energy if something goes wrong. With one phase you have 240 volts that will give you a nasty kick. When you have two phases interacting you get 415 volts and that is a lot more likely to kill you. I have no idea if it's the same with multiple UPSs - but in theory they are changing the phase and therefore you could get more of a jolt out of two of them. You'd still have to be pretty unlucky to have something go wrong with two power supplies at once.

I've never had one "burp" the attached equipment due to a test cycle. Even when it is a deep (80~90%) test cycle. All our UPS are APC.

I deal with a few dozen APC UPSs at work and I've seen a faulty battery drop the load during a self test a few times... It seemed to have more to do with the batteries - the ones in some of our hotter cupboards had "dried out" (or at least expanded and cracked the plastic battery case) and were not working at all...

---

Assuming you have machines with redundant power supplies, I like to split machines over multiple UPS's.

Where I work we were told to not put the redundant power supplies on different phases - something about having 415 volts of potential energy if something goes wrong. With one phase you have 240 volts that will give you a nasty kick. When you have two phases interacting you get 415 volts and that is a lot more likely to kill you. I have no idea if it's the same with multiple UPSs - but in theory they are changing the phase and therefore you could get more of a jolt out of two of them. You'd still have to be pretty unlucky to have something go wrong with two power supplies at once.

I've never had one "burp" the attached equipment due to a test cycle. Even when it is a deep (80~90%) test cycle. All our UPS are APC.

I deal with a few dozen APC UPSs at work and I've seen a faulty battery drop the load during a self test a few times... It seemed to have more to do with the batteries - the ones in some of our hotter cupboards had "dried out" (or at least expanded and cracked the plastic battery case) and were not working at all...

All the best practice information I've read suggests putting redundant power supplies on separate UPS and even separate power grids. I think if multiple power supplies failed in such a fashion that there was 415V flowing into the system, your bigger concern would be putting out the fire rather than server maintenance =) But now I'm intrigued, as I've never heard that warning before. Off to Google!

---

I've certainly seen equipment killed by a three-phase grounding fault generating 415v. Fortunately, the main victim was a sacrificial surge protector - the telephone PBX behind it was saved. And that was just equipment plugged into a standard UK 13A ring main - in a medium-sized office block, with, I guess, different phases on different floors. Somebody working on the installation connected, or more likely disconnected, the wrong wire.

When I had a couple of redundant PSU servers to look after, knowing that they only need one to run (and in an environment where if the power went out, nobody would need to access the servers anyway), I plugged one PSU into a UPS, and the other direct into the mains. Didn't seem to do any harm.

Thanks Matt, for the update, last power outage, I witnessed,
was 26 years ago, when a 10KV/500V/380V-3 fase,transformer
exploded! Not a big one, though.
After this, alot has been changed. Only 400KV 3 fase is above
ground, every 10KV line, has been put underground.
I remember using an 'antenna' to feed a few Fluorecent lights,
close to the 1000KWatt TV transmitter, which now is out off use,
since atleast 15 years.

Power-outages are also very rare and noone I know, uses an UPS.
But the Netherlands are becomming one big city, atleast the west
part of it, close to the sea.
They already call it the 'Randstad', from Rotterdam to Amsterdam,
is already a city with big green (houses) in between.

And it's a beautifull day, lots of sunshine and about 7C.
(But it still freezes, at night)

---

Keep in mind UPS's do need new batteries from time to time. ;)

And they really need to be exercised about once a month with a fairly deep discharge about twice a year.

Lead acid should not be cycled if possible because it shortens it's life. Some designs are better able to withstand cycling than others but they all age when discharged. Most other battery types do last longer if you cycle them.
Battery University
Deep cycle batteries

---