Major Power Outage at SSL

Am I reading this correctly that when the power went out the servers all
insta-shut?

I am shocked to learn that a facility like this has no backup...even to do a clean shut down.

Great job getting everything back up guys!

So say We all?
____________
BSG Anthem
My Facebook page

Claggy,

That's fair...I guess you and I just read certain things differently...

like when I read... A mixture of "all hands on deck" and incredible luck that nothing really got corrupted/fried when the power suddenly disappeared. There are some RAID resyncs happening at the moment, but looking good thus far...

for example, I do not get a warm fuzzy feeling that servers shut down 'gracefully'...lol

A UPS should allow a system to make a regular shutdown.I had a power failure just now and my system went down not gracefully, UPS notwithstanding. So I restarted only the router, not the system, when power restarted, since I know that power outages are repetitive here. So when the power failed another time the router stayed alive. Evidently the battery in my UPS is not capable of securing my system, only the router. Anyway I restarted the system which made a full filesystem check (Linux) and is working again.
Tollio
____________

They do have UPSes and remote control power switches (or those are part of a UPS or two.. it allows them to SSH in and power cycle a machine). I remember reading a few years ago that all the UPSes were basically just power strips at one point as the battery capacity in them was basically zero.

Even with a large 3000VA UPS, having 4-6 servers on it, some of which have 30+ HDDs, you're talking 5 minutes on brand new batteries for a graceful shutdown, and you can't just tell them all to shut down all at once. Some have to go down before others.
____________

Linux laptop uptime: 1484d 22h 42m
Ended due to UPS failure, found 14 hours after the fact

Tis nice to see this back online, thanks for all the hard work guys.
Also thanks to all at Lunatics at keeping us informed.

My uploads are working, no reporting as of yet.
But I know everything will be back to normal shortly.

Thanks guys for all the work in getting it back up and running again. However, it is a shame the Berkley Uni web site could not find space in it's news to say what had happened to seti@home!
Does not boad well for inter-stellar contact if we can't get the communications right here does it?

Thanks guys for all the work in getting it back up and running again. However, it is a shame the Berkley Uni web site could not find space in it's news to say what had happened to seti@home!
Does not boad well for inter-stellar contact if we can't get the communications right here does it?

...being a retired telecom I know you folks had you hands full getting things back up and running! Kudos to everyone for a fine, fine job!... :)

Well, after some initial problems uploading finished WUs (ok, only 5 of them), they are now gone and reported too, just sitting in PV jail now, as usual.

Got one new one as well, so it looks from here are if things are back to normal... ;-)

Ralf

They do have UPSes and remote control power switches (or those are part of a UPS or two.. it allows them to SSH in and power cycle a machine). I remember reading a few years ago that all the UPSes were basically just power strips at one point as the battery capacity in them was basically zero.

Even with a large 3000VA UPS, having 4-6 servers on it, some of which have 30+ HDDs, you're talking 5 minutes on brand new batteries for a graceful shutdown, and you can't just tell them all to shut down all at once. Some have to go down before others.

Attn Matt
There are thousands of WUs that have "Too any errors (may have bug) WU cancelled" from late on the 15th, early on the 16th of May as a result of download errors, but of those that haven't been cancelled many have been downloaded OK today.
We suspect it's a result of the power failure- the Scheduler & download servers were still up, but the WU storage wasn't (or at least wasn't accessable).
____________
Grant
Darwin NT.

Most UPS's of this size allow you to daisy chain multiple batteries together to allow more time to shut things down before they run out of juice. This makes sense to me, what do you guys there in the thick of it think?

im still tring to get all the results uploaded to you guys and i need new work as well as my pc finished all that was on my pc.

____________

Lol kind of bush, but very effective! :D I have a couple APC 1100 UPS's that I have been looking at replacing batteries in, for a hundred + a crack, and you could get a decent deep cycle battery for that kind of ching. If it wasn't in my house, like in my workshop or something, I might just consider it. But back on topic, I doubt they'd do that, even being quite cost effective. Maybe Matt will chime in and let us know if he feels the upgrade is a good use of funds at this time.
____________

[very long UPS-related thoughts]

You can go with larger AH batteries, but more modern units have a limit to how much they'll charge, and for how long they'll charge. For example, the APC 1400 that I'm using had two 6AH batteries in it, and I replaced them with 9AH batteries that I had left over from ordering the wrong kit for two Tripp-Lite 1400s a few years ago. 50% more run time.

I have heard of people using car batteries, and it does work most of the time (some of those batteries are 60-150AH), but the problem with those is unless they are sealed/maintenance free, you are very very strongly advised against using them indoors, due to the acid fumes during discharge and charge cycles. Also, you can't keep rack-mount stuff neat and tidy if you are using batteries that won't fit within the chassis of the particular unit.

Regarding being able to remotely shut down in that situation.. I don't think they could anyway. Network connectivity went down as well, not just the servers themselves. Network went down and there was no way to tell the servers to shut down unless someone was already there in the lab when it happened.

However, most OSes have the ability to plug into a UPS and monitor the condition of it. For example, Windows 7 sees mine just fine and I have it set to shut down when the battery reaches 50%. That gives me about 10 minutes from the power going out to the OS doing a graceful shutdown. For servers, I would set them to somewhere around 30 seconds from when AC is lost to beginning the shutdown routine. Only problem with that is if you have 5 servers hooked up to one unit, they can't all plug into the status port. You could, though, hook the server up that is last to go down and set up a script on it that when AC is lost, ssh to the other machines and tell them 'init 0' and whatever else you have to do ('umount /all/remote/mounts', etc).

Even then, there's still two things to find out. How long does that particular unit last when all the servers are on and consuming power, and how long does it take them to do a graceful shutdown? As you start shutting them down, the run time will increase, but then you need to find out by how much.

[/very long UPS-related thoughts]
____________

Linux laptop uptime: 1484d 22h 42m
Ended due to UPS failure, found 14 hours after the fact

The basic premise of a commercial UPS system is to hold the equipment online long enough for backup generators to come online. They are surge/spike/brownout resistant as well.

In smaller or home use, "long enough to turn things off" is the primary selling point.

Side note: I see the AP data is showing pretty much non-existant, is this a residual problem or just a side effect of Jocelyn apparently taking some time off?
____________

Janice