Major Power Outage at SSL


log in

Advanced search

Message boards : News : Major Power Outage at SSL

Previous · 1 · 2 · 3 · 4 · 5 · 6 . . . 9 · Next
Author Message
Profile celttooth
Avatar
Send message
Joined: 21 Nov 99
Posts: 16233
Credit: 10,988,796
RAC: 18,286
Canada
Message 1233194 - Posted: 19 May 2012, 2:47:09 UTC

Someone must be working like a Trojan to get everything back on, good work and thanks for the effort. My boxes still have lots of work unites to slave over.

Profile Boomer
Volunteer tester
Send message
Joined: 1 Jun 09
Posts: 41
Credit: 9,314,882
RAC: 14,882
United States
Message 1233229 - Posted: 19 May 2012, 4:46:09 UTC

Hey Matt,

I was worried for a while there that someone pulled the plug on this project. Not that anyone would ever do that; right?

I am really happy to see SETI@HOME come back online. Thanks for doing a great job and for keeping us informed. Most technical folks do not communicate very well. But, you have great technical skills AND communication skills which is a very rare combination.

Kudos and kudos.

Regards, Boomer
____________

Profile ivan
Volunteer tester
Avatar
Send message
Joined: 5 Mar 01
Posts: 628
Credit: 144,366,558
RAC: 153,252
United Kingdom
Message 1233342 - Posted: 19 May 2012, 10:09:56 UTC - in response to Message 1233065.

They do have UPSes and remote control power switches (or those are part of a UPS or two.. it allows them to SSH in and power cycle a machine). I remember reading a few years ago that all the UPSes were basically just power strips at one point as the battery capacity in them was basically zero.

Even with a large 3000VA UPS, having 4-6 servers on it, some of which have 30+ HDDs, you're talking 5 minutes on brand new batteries for a graceful shutdown, and you can't just tell them all to shut down all at once. Some have to go down before others.
I know that this was kind of a freak occurrence, but it does appear to show a weakness in the systems. Would it possibly make sense to direct some of our fund raising contributions towards an even more robust UPS setup? Most UPS's of this size allow you to daisy chain multiple batteries together to allow more time to shut things down before they run out of juice. This makes sense to me, what do you guys there in the thick of it think?

Actually, my thoughts wandered to providing a backup diesel generator! :-) Probably too expensive I know... In fact, their being on the wet coast of the US leads to other problems -- I heard recently about such a backup system kicking in after an earthquake (might have been last year's Japanese disaster) and maintaining systems perfectly for a half-hour or so, until the years of sludge at the bottom of the fuel tanks, stirred up by the shaking, clogged the fuel filters!
____________

Terror Australis
Volunteer tester
Send message
Joined: 14 Feb 04
Posts: 1723
Credit: 205,988,458
RAC: 28,585
Australia
Message 1233484 - Posted: 19 May 2012, 16:12:41 UTC - in response to Message 1233088.

Run time at full load went from a few minutes to about 6 hours.

Six hours? you must be on the Casuarina sub-station ;-)

T.A.

Larry Coolidge
Avatar
Send message
Joined: 16 May 99
Posts: 19
Credit: 14,319,177
RAC: 11,626
United States
Message 1233603 - Posted: 19 May 2012, 21:14:28 UTC - in response to Message 1233342.

A gas turbine would resolve that issue and you have less polution generated as well [/quote]
____________

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5872
Credit: 60,855,471
RAC: 47,518
Australia
Message 1233643 - Posted: 19 May 2012, 22:00:51 UTC - in response to Message 1233484.

Six hours? you must be on the Casuarina sub-station ;-)

Nah, but even without those episodes the power goes of more frequently & for longer periods of time than it used to.
Best way to save money according to the bean counters- cut back on maintenance. If it's working why spend money on it?

____________
Grant
Darwin NT.

NickProject donor
Avatar
Send message
Joined: 11 Oct 11
Posts: 3983
Credit: 2,038,097
RAC: 755
United Kingdom
Message 1233683 - Posted: 19 May 2012, 23:21:22 UTC - in response to Message 1232936.

Thanks guys for all the work in getting it back up and running again. However, it is a shame the Berkley Uni web site could not find space in it's news to say what had happened to seti@home!
Does not boad well for inter-stellar contact if we can't get the communications right here does it?


It did report the power fault, Rt'; you just had to fish around to find it
though.

____________
The Kite Fliers

--------------------
Kite fliers: An imaginary club of solo members, those who don't yet
belong to a formal team so "fly their own kites" - as the saying goes.

Profile Nomad
Send message
Joined: 24 Apr 12
Posts: 2
Credit: 279,123
RAC: 0
United States
Message 1234970 - Posted: 22 May 2012, 9:56:37 UTC

When will the Astro Pulse Tasks be ready to download? Have not received any since the outage.
____________
NOMAD

N9JFE David SProject donor
Volunteer tester
Avatar
Send message
Joined: 4 Oct 99
Posts: 12074
Credit: 14,707,440
RAC: 10,993
United States
Message 1235038 - Posted: 22 May 2012, 13:17:05 UTC - in response to Message 1234970.

When will the Astro Pulse Tasks be ready to download? Have not received any since the outage.

It looks like there are some now.

(Currently, you can't go by what it says about AP tasks in the field, etc., on the Server Status Page. But the page also shows some of the AP splitters running, which is good info.)

By the way, you would do better to ask questions such as this in the Number Crunching forum.

____________
David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.


Profile ShadowStalker
Send message
Joined: 22 Dec 99
Posts: 4
Credit: 70,334
RAC: 0
United States
Message 1238961 - Posted: 31 May 2012, 17:04:01 UTC - in response to Message 1232771.

need to have a independent website outside of seti for when power goes out again where can get updates of whats going on or send out mass email
____________

Dave
Avatar
Send message
Joined: 29 Mar 02
Posts: 774
Credit: 23,193,139
RAC: 0
United Kingdom
Message 1238985 - Posted: 31 May 2012, 17:45:50 UTC

Updates were posted in the Technical News section & the front page.

Profile Chris SProject donor
Volunteer tester
Avatar
Send message
Joined: 19 Nov 00
Posts: 32146
Credit: 13,923,503
RAC: 24,878
United Kingdom
Message 1239023 - Posted: 31 May 2012, 18:33:57 UTC

I would suggest that Boincstats could be used for this, they have a project news update section. Alternatively messages could be posted on other Boinc projects, and the word spread that way.

We need to keep a sensible balance here. It is NOT the end of the world if Seti goes down without prior knowledge for a couple of days. To those few who feel that it is, there are other Boinc projects to support in the meantime. There will be a good reason for the downtime and it will be back ASAP.


____________
Damsel Rescuer, Uli Devotee, Julie Supporter, ES99 Admirer,
Raccoon Friend, Anniet fan, Dominant Culture Member


Profile Cliff HardingProject donor
Volunteer tester
Avatar
Send message
Joined: 18 Aug 99
Posts: 1012
Credit: 53,262,012
RAC: 42,445
United States
Message 1239031 - Posted: 31 May 2012, 18:43:36 UTC - in response to Message 1239023.
Last modified: 31 May 2012, 18:46:22 UTC

I saw via the Cricket Graphs page that SETI was back up after the normal scheduled outage on Tuesday afternoon. When I checked the progress on my task reporting a few minutes later, it had taken a sudden nose dive. I COULD NOT GET TO THE HOME PAGE. I had to check the Lunatics site to get any information on this outage. If this was a scheduled outage there must be a better way of notifying people, how about a broadcast notice that will appear on the NOTICE tab.
____________


I don't buy computers, I build them!!

LetiCern
Send message
Joined: 3 Oct 99
Posts: 11
Credit: 9,745,853
RAC: 2,404
United States
Message 1239036 - Posted: 31 May 2012, 18:46:34 UTC

My 2 cents: Seti and Bionic apps are hands-off on my systems so I hardly ever look at 'em. I may check Bionic twice a month. So when I saw uploads stacking up I could not locate any news. At first I checked my ISP connection finally I checked connection to berkeley.edu so when that failed I stopped checking.

It would have been nice to have BIONIC post a notice 'bout outage.

Thanx guys for all your work (never mind the little SNAFU).
____________

Profile Gary CharpentierProject donor
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 12810
Credit: 7,382,210
RAC: 18,157
United States
Message 1239053 - Posted: 31 May 2012, 19:19:03 UTC - in response to Message 1239031.

I saw via the Cricket Graphs page that SETI was back up after the normal scheduled outage on Tuesday afternoon. When I checked the progress on my task reporting a few minutes later, it had taken a sudden nose dive. I COULD NOT GET TO THE HOME PAGE. I had to check the Lunatics site to get any information on this outage. If this was a scheduled outage there must be a better way of notifying people, how about a broadcast notice that will appear on the NOTICE tab.

Perhaps, but the notice about the planned outage was, and still is with time stamp, on the home page before the normal maintenance outage.
____________

Profile Enzeder
Avatar
Send message
Joined: 30 Jan 01
Posts: 6
Credit: 33,697,307
RAC: 49,194
New Zealand
Message 1239055 - Posted: 31 May 2012, 19:20:58 UTC - in response to Message 1238985.

Updates were posted in the Technical News section & the front page.


Can you point us rookies to the updates you're referring to? I don't see any updates since the power went out (29th), and most of the time I tried to look, the website seemed to be down.

It would be nice if SETI/BOINC could simply mirror their homepage to another location. Those of us that missed that 90 min window could then satisfy our curiosity and know the problem isn't on our end. Perhaps another UC Berkeley server (on the other side of campus?), but in an ideal world something offsite would be nice.

I'm a little surprised that there weren't even any announcements on the news section of the UC Berkeley homepage (that I could find).
____________

Profile Ex
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 12 Mar 12
Posts: 2895
Credit: 1,797,699
RAC: 796
United States
Message 1239072 - Posted: 31 May 2012, 19:43:50 UTC
Last modified: 31 May 2012, 19:48:27 UTC

David Anderson made a post in the Boinc boards, but not until a day before the outage. I too had no idea we were to be down for a couple days. Luckily I asked on the Boinc boards what was going on, just before they too went lights-out.

Some more notice here would have been cool I think.

(And, I still don't quite understand why the forum's servers have anything to do with the scientific servers....... I think it's kind of- erm - dinosaurish that the forums have to go down for scientific maintenance)
____________
-Dave #2

3.2.0-33

Profile Gary CharpentierProject donor
Volunteer tester
Avatar
Send message
Joined: 25 Dec 00
Posts: 12810
Credit: 7,382,210
RAC: 18,157
United States
Message 1239075 - Posted: 31 May 2012, 19:58:36 UTC - in response to Message 1239055.

Updates were posted in the Technical News section & the front page.


Can you point us rookies to the updates you're referring to? I don't see any updates since the power went out (29th), and most of the time I tried to look, the website seemed to be down.

It would be nice if SETI/BOINC could simply mirror their homepage to another location. Those of us that missed that 90 min window could then satisfy our curiosity and know the problem isn't on our end. Perhaps another UC Berkeley server (on the other side of campus?), but in an ideal world something offsite would be nice.

I'm a little surprised that there weren't even any announcements on the news section of the UC Berkeley homepage (that I could find).

See the word "HOME" in the upper left. Press it. It will take you to the notice posted at 17:11:12 UTC that there was an outage in the works. That notice was up from then until the system was taken down. And it was up all the time the maintenance outage was going on.

I suspect the lab didn't get much more notice than this.
____________

Previous · 1 · 2 · 3 · 4 · 5 · 6 . . . 9 · Next

Message boards : News : Major Power Outage at SSL

Copyright © 2014 University of California