Major Power Outage at SSL


log in

Advanced search

Message boards : News : Major Power Outage at SSL

1 · 2 · 3 · 4 . . . 9 · Next
Author Message
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1388
Credit: 74,079
RAC: 0
United States
Message 1232771 - Posted: 18 May 2012, 15:57:18 UTC

There was a major power outage on Tuesday evening that affected several buildings here on campus including the entire Space Sciences Laboratory. Power has been restored this morning, and we are slowly getting the project back on line.
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1388
Credit: 74,079
RAC: 0
United States
Message 1232776 - Posted: 18 May 2012, 15:59:18 UTC

More detail (from Jon - another systems administrator here at the lab):

Power failed when there was a short in the power distribution lines that feed the hill buildings [including Space Sciences Laboratory, Math Science Research Institute, Lawrence Hall of Science]. The short occurred because of deterioration of the insulation on these power lines, which are buried in a conduit in the ground. The power distribution lines are thousands of feet long, originating in a substation at LBL below LHS. Most of this conduit only contain one power line, but we were lucky in that the section that failed had two power lines in it, so the electricians could just utilize the second unused line to restore electrical service. It was an arduous process finding just where the short was, since they had to disconnect one segment at a time and test it. This is why it took so long to find and repair.

- Matt
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Profile Ronald R CODNEY
Avatar
Send message
Joined: 19 Nov 11
Posts: 87
Credit: 420,497
RAC: 0
United States
Message 1232792 - Posted: 18 May 2012, 16:10:04 UTC

Heh Matt:
Is there gonna b an ISP change for uploads/downloads, or a reset requirement by the masses?

Thanks for filling us in also.

Profile Chris S
Volunteer tester
Avatar
Send message
Joined: 19 Nov 00
Posts: 30980
Credit: 11,154,391
RAC: 19,595
United Kingdom
Message 1232797 - Posted: 18 May 2012, 16:12:56 UTC

To get it all up and back on line from scratch in two hours was a major feat of teamwork, I take my hat off to you and the lads, well done!

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38149
Credit: 555,266,322
RAC: 620,560
United States
Message 1232809 - Posted: 18 May 2012, 16:23:26 UTC
Last modified: 18 May 2012, 16:23:44 UTC

Thanks for the news, Matt.
And thanks to you and Eric and Jeff for getting out of bed so early today to get the servers back online.
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar
Send message
Joined: 1 Mar 99
Posts: 1388
Credit: 74,079
RAC: 0
United States
Message 1232810 - Posted: 18 May 2012, 16:24:08 UTC - in response to Message 1232797.

To get it all up and back on line from scratch in two hours was a major feat of teamwork, I take my hat off to you and the lads, well done!


Thanks. A mixture of "all hands on deck" and incredible luck that nothing really got corrupted/fried when the power suddenly disappeared. There are some RAID resyncs happening at the moment, but looking good thus far...

- Matt
____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

rob smith
Volunteer tester
Send message
Joined: 7 Mar 03
Posts: 8117
Credit: 52,183,217
RAC: 78,301
United Kingdom
Message 1232815 - Posted: 18 May 2012, 16:34:33 UTC

Glad to hear nothing got fried.
Frying leading to corruption is one of the biggest hazards of a big mains cable shorting, especially one that is of any decent length.
____________
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

Profile Keith T.
Volunteer tester
Avatar
Send message
Joined: 23 Aug 99
Posts: 738
Credit: 231,168
RAC: 0
United Kingdom
Message 1232817 - Posted: 18 May 2012, 16:36:27 UTC

Well done to all.

Uploads, reporting and Downloads are all working here. Even SETI Beta is working !

I noticed that SSL is over 50 years old now, are the cables a similar age ?

Profile perryjay
Volunteer tester
Avatar
Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 14,829,657
RAC: 11,909
United States
Message 1232818 - Posted: 18 May 2012, 16:36:41 UTC - in response to Message 1232810.

Glad to have you guys back up and running. Thanks for all you do.
____________


PROUD MEMBER OF Team Starfire World BOINC

Claggy
Volunteer tester
Send message
Joined: 5 Jul 99
Posts: 4039
Credit: 32,691,076
RAC: 800
United Kingdom
Message 1232819 - Posted: 18 May 2012, 16:36:44 UTC - in response to Message 1232810.

To get it all up and back on line from scratch in two hours was a major feat of teamwork, I take my hat off to you and the lads, well done!


Thanks. A mixture of "all hands on deck" and incredible luck that nothing really got corrupted/fried when the power suddenly disappeared. There are some RAID resyncs happening at the moment, but looking good thus far...

- Matt

Thanks for the update, and well done for getting everything up so quick too,

Claggy

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38149
Credit: 555,266,322
RAC: 620,560
United States
Message 1232821 - Posted: 18 May 2012, 16:37:57 UTC
Last modified: 18 May 2012, 16:38:40 UTC

Matt....
Could you possibly check on one thing for the kitties?
I have one rig trying to report about 390 completed tasks.
I have others that are going to have much higher numbers to report than that.

Getting HTTP server errors.
I know this might be due to heavy traffic, or the servers not quite up to speed yet.

But at one time, there were some server settings that were causing problems with reporting large numbers of completed tasks, and they were adjusted at that time.

Could you have a look and make sure that the server end is OK please?

Meow.
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

Lynn Curtis
Send message
Joined: 20 Feb 12
Posts: 1
Credit: 5,353
RAC: 0
United States
Message 1232825 - Posted: 18 May 2012, 16:45:31 UTC - in response to Message 1232771.

Haven't been able to Upload to SETI via BOINC. Are there still issues with connecting to the server?

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38149
Credit: 555,266,322
RAC: 620,560
United States
Message 1232827 - Posted: 18 May 2012, 16:49:58 UTC - in response to Message 1232825.

Haven't been able to Upload to SETI via BOINC. Are there still issues with connecting to the server?

Just came back up after a two day outage and the servers are getting hammered.
Let Boinc keep trying and it should get through after a while, but it might take a bit to settle down.
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

Steven Proveaux
Send message
Joined: 10 Jun 99
Posts: 1
Credit: 691,952
RAC: 268
United States
Message 1232829 - Posted: 18 May 2012, 16:50:22 UTC

I just uploaded the latest batch of work units since the power has been restored (Friday morning) but there has been no new downloads. has anyone had a similar problem?
____________

Claggy
Volunteer tester
Send message
Joined: 5 Jul 99
Posts: 4039
Credit: 32,691,076
RAC: 800
United Kingdom
Message 1232833 - Posted: 18 May 2012, 16:52:42 UTC - in response to Message 1232829.

I just uploaded the latest batch of work units since the power has been restored (Friday morning) but there has been no new downloads. has anyone had a similar problem?

See msattler's reply in his last post.

Claggy

Profile Jaye Ellen
Send message
Joined: 29 Nov 08
Posts: 24
Credit: 5,319,292
RAC: 3,101
United States
Message 1232834 - Posted: 18 May 2012, 16:53:39 UTC - in response to Message 1232825.

Hi all,
I also have been unable to send or rcv stuff from BOINC (probly from your power problems) --- I'm hoping the comm restarts soon !!!

Jaye Ellen

Profile perryjay
Volunteer tester
Avatar
Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 14,829,657
RAC: 11,909
United States
Message 1232846 - Posted: 18 May 2012, 17:11:55 UTC

I just managed to report 181 tasks and actually got 1 (one) downloaded in return. That's okay though, I'm just happy things are moving.
____________


PROUD MEMBER OF Team Starfire World BOINC

Profile Ex
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 12 Mar 12
Posts: 2895
Credit: 1,680,153
RAC: 1,204
United States
Message 1232850 - Posted: 18 May 2012, 17:15:47 UTC
Last modified: 18 May 2012, 17:17:39 UTC

Thanks to the scientists for a great job! +10,000!

and thanks for the update, I think some of us thought the world was ending. :-)

Glad to see things coming online, and people be patient; it just came back online after days of being down, I bet it'll be 24 hours before everyone's machines report and start crunching (getting work) again.
____________
-Dave #2

3.2.0-33

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38149
Credit: 555,266,322
RAC: 620,560
United States
Message 1232854 - Posted: 18 May 2012, 17:23:09 UTC - in response to Message 1232821.

Matt....
Could you possibly check on one thing for the kitties?
I have one rig trying to report about 390 completed tasks.
I have others that are going to have much higher numbers to report than that.

Getting HTTP server errors.
I know this might be due to heavy traffic, or the servers not quite up to speed yet.

But at one time, there were some server settings that were causing problems with reporting large numbers of completed tasks, and they were adjusted at that time.

Could you have a look and make sure that the server end is OK please?

Meow.

Bump before I have to leave for work...
Still getting HTTP internal server errors trying to report.
Thanks.
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

Profile dg6580004
Avatar
Send message
Joined: 14 Mar 12
Posts: 31
Credit: 1,076,077
RAC: 8,378
United States
Message 1232858 - Posted: 18 May 2012, 17:28:18 UTC - in response to Message 1232850.

Nice job on the restore, "everything takes longer than it takes".........keep up the great support.

Until I saw the power outage news posted on the Net, I thought maybe the signal had been found...........Oh well, back to work

1 · 2 · 3 · 4 . . . 9 · Next

Message boards : News : Major Power Outage at SSL

Copyright © 2014 University of California