Major Power Outage at SSL

Message boards : News : Major Power Outage at SSL
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 . . . 9 · Next

AuthorMessage
Profile celttooth
Avatar

Send message
Joined: 21 Nov 99
Posts: 26503
Credit: 28,583,098
RAC: 0
Canada
Message 1233194 - Posted: 19 May 2012, 2:47:09 UTC

Someone must be working like a Trojan to get everything back on, good work and thanks for the effort. My boxes still have lots of work unites to slave over.
ID: 1233194 · Report as offensive
Profile Boomer
Volunteer tester

Send message
Joined: 1 Jun 09
Posts: 43
Credit: 21,892,319
RAC: 10
United States
Message 1233229 - Posted: 19 May 2012, 4:46:09 UTC

Hey Matt,

I was worried for a while there that someone pulled the plug on this project. Not that anyone would ever do that; right?

I am really happy to see SETI@HOME come back online. Thanks for doing a great job and for keeping us informed. Most technical folks do not communicate very well. But, you have great technical skills AND communication skills which is a very rare combination.

Kudos and kudos.

Regards, Boomer
ID: 1233229 · Report as offensive
Profile ivan
Volunteer tester
Avatar

Send message
Joined: 5 Mar 01
Posts: 783
Credit: 348,560,338
RAC: 223
United Kingdom
Message 1233342 - Posted: 19 May 2012, 10:09:56 UTC - in response to Message 1233065.  

They do have UPSes and remote control power switches (or those are part of a UPS or two.. it allows them to SSH in and power cycle a machine). I remember reading a few years ago that all the UPSes were basically just power strips at one point as the battery capacity in them was basically zero.

Even with a large 3000VA UPS, having 4-6 servers on it, some of which have 30+ HDDs, you're talking 5 minutes on brand new batteries for a graceful shutdown, and you can't just tell them all to shut down all at once. Some have to go down before others.
I know that this was kind of a freak occurrence, but it does appear to show a weakness in the systems. Would it possibly make sense to direct some of our fund raising contributions towards an even more robust UPS setup? Most UPS's of this size allow you to daisy chain multiple batteries together to allow more time to shut things down before they run out of juice. This makes sense to me, what do you guys there in the thick of it think?

Actually, my thoughts wandered to providing a backup diesel generator! :-) Probably too expensive I know... In fact, their being on the wet coast of the US leads to other problems -- I heard recently about such a backup system kicking in after an earthquake (might have been last year's Japanese disaster) and maintaining systems perfectly for a half-hour or so, until the years of sludge at the bottom of the fuel tanks, stirred up by the shaking, clogged the fuel filters!
ID: 1233342 · Report as offensive
Terror Australis
Volunteer tester

Send message
Joined: 14 Feb 04
Posts: 1817
Credit: 262,693,308
RAC: 44
Australia
Message 1233484 - Posted: 19 May 2012, 16:12:41 UTC - in response to Message 1233088.  

Run time at full load went from a few minutes to about 6 hours.

Six hours? you must be on the Casuarina sub-station ;-)

T.A.
ID: 1233484 · Report as offensive
Larry Coolidge
Avatar

Send message
Joined: 16 May 99
Posts: 19
Credit: 63,791,393
RAC: 0
United States
Message 1233603 - Posted: 19 May 2012, 21:14:28 UTC - in response to Message 1233342.  

A gas turbine would resolve that issue and you have less polution generated as well [/quote]
ID: 1233603 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13854
Credit: 208,696,464
RAC: 304
Australia
Message 1233643 - Posted: 19 May 2012, 22:00:51 UTC - in response to Message 1233484.  

Six hours? you must be on the Casuarina sub-station ;-)

Nah, but even without those episodes the power goes of more frequently & for longer periods of time than it used to.
Best way to save money according to the bean counters- cut back on maintenance. If it's working why spend money on it?

Grant
Darwin NT
ID: 1233643 · Report as offensive
Nick
Avatar

Send message
Joined: 11 Oct 11
Posts: 4344
Credit: 3,313,107
RAC: 0
United Kingdom
Message 1233683 - Posted: 19 May 2012, 23:21:22 UTC - in response to Message 1232936.  

Thanks guys for all the work in getting it back up and running again. However, it is a shame the Berkley Uni web site could not find space in it's news to say what had happened to seti@home!
Does not boad well for inter-stellar contact if we can't get the communications right here does it?


It did report the power fault, Rt'; you just had to fish around to find it
though.

The Kite Fliers

--------------------
Kite fliers: An imaginary club of solo members, those who don't yet
belong to a formal team so "fly their own kites" - as the saying goes.
ID: 1233683 · Report as offensive
Profile Nomad

Send message
Joined: 24 Apr 12
Posts: 2
Credit: 279,123
RAC: 0
United States
Message 1234970 - Posted: 22 May 2012, 9:56:37 UTC

When will the Astro Pulse Tasks be ready to download? Have not received any since the outage.
NOMAD
ID: 1234970 · Report as offensive
David S
Volunteer tester
Avatar

Send message
Joined: 4 Oct 99
Posts: 18352
Credit: 27,761,924
RAC: 12
United States
Message 1235038 - Posted: 22 May 2012, 13:17:05 UTC - in response to Message 1234970.  

When will the Astro Pulse Tasks be ready to download? Have not received any since the outage.

It looks like there are some now.

(Currently, you can't go by what it says about AP tasks in the field, etc., on the Server Status Page. But the page also shows some of the AP splitters running, which is good info.)

By the way, you would do better to ask questions such as this in the Number Crunching forum.

David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.

ID: 1235038 · Report as offensive
Profile ShadowStalker

Send message
Joined: 22 Dec 99
Posts: 4
Credit: 88,680
RAC: 1
United States
Message 1238961 - Posted: 31 May 2012, 17:04:01 UTC - in response to Message 1232771.  

need to have a independent website outside of seti for when power goes out again where can get updates of whats going on or send out mass email
ID: 1238961 · Report as offensive
Dave

Send message
Joined: 29 Mar 02
Posts: 778
Credit: 25,001,396
RAC: 0
United Kingdom
Message 1238985 - Posted: 31 May 2012, 17:45:50 UTC

Updates were posted in the Technical News section & the front page.
ID: 1238985 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51478
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1238988 - Posted: 31 May 2012, 17:49:05 UTC - in response to Message 1238985.  

Updates were posted in the Technical News section & the front page.

I can see his point....
If one has not happened to have read the news before the outage.
Once things are down, there is no source of news direct from the Seti project with all the main servers offline.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1238988 · Report as offensive
Profile Cliff Harding
Volunteer tester
Avatar

Send message
Joined: 18 Aug 99
Posts: 1432
Credit: 110,967,840
RAC: 67
United States
Message 1239031 - Posted: 31 May 2012, 18:43:36 UTC - in response to Message 1239023.  
Last modified: 31 May 2012, 18:46:22 UTC

I saw via the Cricket Graphs page that SETI was back up after the normal scheduled outage on Tuesday afternoon. When I checked the progress on my task reporting a few minutes later, it had taken a sudden nose dive. I COULD NOT GET TO THE HOME PAGE. I had to check the Lunatics site to get any information on this outage. If this was a scheduled outage there must be a better way of notifying people, how about a broadcast notice that will appear on the NOTICE tab.


I don't buy computers, I build them!!
ID: 1239031 · Report as offensive
LetiCern

Send message
Joined: 3 Oct 99
Posts: 15
Credit: 12,314,846
RAC: 0
United States
Message 1239036 - Posted: 31 May 2012, 18:46:34 UTC

My 2 cents: Seti and Bionic apps are hands-off on my systems so I hardly ever look at 'em. I may check Bionic twice a month. So when I saw uploads stacking up I could not locate any news. At first I checked my ISP connection finally I checked connection to berkeley.edu so when that failed I stopped checking.

It would have been nice to have BIONIC post a notice 'bout outage.

Thanx guys for all your work (never mind the little SNAFU).
ID: 1239036 · Report as offensive
Profile Gary Charpentier Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 25 Dec 00
Posts: 31006
Credit: 53,134,872
RAC: 32
United States
Message 1239053 - Posted: 31 May 2012, 19:19:03 UTC - in response to Message 1239031.  

I saw via the Cricket Graphs page that SETI was back up after the normal scheduled outage on Tuesday afternoon. When I checked the progress on my task reporting a few minutes later, it had taken a sudden nose dive. I COULD NOT GET TO THE HOME PAGE. I had to check the Lunatics site to get any information on this outage. If this was a scheduled outage there must be a better way of notifying people, how about a broadcast notice that will appear on the NOTICE tab.

Perhaps, but the notice about the planned outage was, and still is with time stamp, on the home page before the normal maintenance outage.
ID: 1239053 · Report as offensive
Profile Enzeder
Avatar

Send message
Joined: 30 Jan 01
Posts: 6
Credit: 56,862,055
RAC: 18
New Zealand
Message 1239055 - Posted: 31 May 2012, 19:20:58 UTC - in response to Message 1238985.  

Updates were posted in the Technical News section & the front page.


Can you point us rookies to the updates you're referring to? I don't see any updates since the power went out (29th), and most of the time I tried to look, the website seemed to be down.

It would be nice if SETI/BOINC could simply mirror their homepage to another location. Those of us that missed that 90 min window could then satisfy our curiosity and know the problem isn't on our end. Perhaps another UC Berkeley server (on the other side of campus?), but in an ideal world something offsite would be nice.

I'm a little surprised that there weren't even any announcements on the news section of the UC Berkeley homepage (that I could find).
ID: 1239055 · Report as offensive
Profile Ex: "Socialist"
Volunteer tester
Avatar

Send message
Joined: 12 Mar 12
Posts: 3433
Credit: 2,616,158
RAC: 2
United States
Message 1239072 - Posted: 31 May 2012, 19:43:50 UTC
Last modified: 31 May 2012, 19:48:27 UTC

David Anderson made a post in the Boinc boards, but not until a day before the outage. I too had no idea we were to be down for a couple days. Luckily I asked on the Boinc boards what was going on, just before they too went lights-out.

Some more notice here would have been cool I think.

(And, I still don't quite understand why the forum's servers have anything to do with the scientific servers....... I think it's kind of- erm - dinosaurish that the forums have to go down for scientific maintenance)
#resist
ID: 1239072 · Report as offensive
Profile Gary Charpentier Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 25 Dec 00
Posts: 31006
Credit: 53,134,872
RAC: 32
United States
Message 1239075 - Posted: 31 May 2012, 19:58:36 UTC - in response to Message 1239055.  

Updates were posted in the Technical News section & the front page.


Can you point us rookies to the updates you're referring to? I don't see any updates since the power went out (29th), and most of the time I tried to look, the website seemed to be down.

It would be nice if SETI/BOINC could simply mirror their homepage to another location. Those of us that missed that 90 min window could then satisfy our curiosity and know the problem isn't on our end. Perhaps another UC Berkeley server (on the other side of campus?), but in an ideal world something offsite would be nice.

I'm a little surprised that there weren't even any announcements on the news section of the UC Berkeley homepage (that I could find).

See the word "HOME" in the upper left. Press it. It will take you to the notice posted at 17:11:12 UTC that there was an outage in the works. That notice was up from then until the system was taken down. And it was up all the time the maintenance outage was going on.

I suspect the lab didn't get much more notice than this.
ID: 1239075 · Report as offensive
Profile Ex: "Socialist"
Volunteer tester
Avatar

Send message
Joined: 12 Mar 12
Posts: 3433
Credit: 2,616,158
RAC: 2
United States
Message 1239077 - Posted: 31 May 2012, 20:01:08 UTC

If they didn't have more notice, than I can understand that. Because how it happened was more like "oh by the way, in an hour we'll be down for days..."

;-)
#resist
ID: 1239077 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22528
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1239079 - Posted: 31 May 2012, 20:02:42 UTC

There was certainly a hint of panic in Matt's announcement - "Going down about 90 minutes after I post this". The vague description he gave suggests to me that this was the campus maintenance men knocking on the door, saying things like "Turn it off NOW"...
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1239079 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 · 6 . . . 9 · Next

Message boards : News : Major Power Outage at SSL


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.