Major Power Outage at SSL

Message boards : News : Major Power Outage at SSL
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · 4 . . . 9 · Next

AuthorMessage
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 1232771 - Posted: 18 May 2012, 15:57:18 UTC

There was a major power outage on Tuesday evening that affected several buildings here on campus including the entire Space Sciences Laboratory. Power has been restored this morning, and we are slowly getting the project back on line.
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 1232771 · Report as offensive
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 1232776 - Posted: 18 May 2012, 15:59:18 UTC

More detail (from Jon - another systems administrator here at the lab):

Power failed when there was a short in the power distribution lines that feed the hill buildings [including Space Sciences Laboratory, Math Science Research Institute, Lawrence Hall of Science]. The short occurred because of deterioration of the insulation on these power lines, which are buried in a conduit in the ground. The power distribution lines are thousands of feet long, originating in a substation at LBL below LHS. Most of this conduit only contain one power line, but we were lucky in that the section that failed had two power lines in it, so the electricians could just utilize the second unused line to restore electrical service. It was an arduous process finding just where the short was, since they had to disconnect one segment at a time and test it. This is why it took so long to find and repair.

- Matt
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 1232776 · Report as offensive
Profile Ronald R CODNEY
Avatar

Send message
Joined: 19 Nov 11
Posts: 87
Credit: 420,920
RAC: 0
United States
Message 1232792 - Posted: 18 May 2012, 16:10:04 UTC

Heh Matt:
Is there gonna b an ISP change for uploads/downloads, or a reset requirement by the masses?

Thanks for filling us in also.
ID: 1232792 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51477
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1232809 - Posted: 18 May 2012, 16:23:26 UTC
Last modified: 18 May 2012, 16:23:44 UTC

Thanks for the news, Matt.
And thanks to you and Eric and Jeff for getting out of bed so early today to get the servers back online.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1232809 · Report as offensive
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1444
Credit: 957,058
RAC: 0
United States
Message 1232810 - Posted: 18 May 2012, 16:24:08 UTC - in response to Message 1232797.  

To get it all up and back on line from scratch in two hours was a major feat of teamwork, I take my hat off to you and the lads, well done!


Thanks. A mixture of "all hands on deck" and incredible luck that nothing really got corrupted/fried when the power suddenly disappeared. There are some RAID resyncs happening at the moment, but looking good thus far...

- Matt
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude
ID: 1232810 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22445
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1232815 - Posted: 18 May 2012, 16:34:33 UTC

Glad to hear nothing got fried.
Frying leading to corruption is one of the biggest hazards of a big mains cable shorting, especially one that is of any decent length.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1232815 · Report as offensive
Profile Keith T.
Volunteer tester
Avatar

Send message
Joined: 23 Aug 99
Posts: 962
Credit: 537,293
RAC: 9
United Kingdom
Message 1232817 - Posted: 18 May 2012, 16:36:27 UTC

Well done to all.

Uploads, reporting and Downloads are all working here. Even SETI Beta is working !

I noticed that SSL is over 50 years old now, are the cables a similar age ?
ID: 1232817 · Report as offensive
Profile perryjay
Volunteer tester
Avatar

Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 20,676,751
RAC: 0
United States
Message 1232818 - Posted: 18 May 2012, 16:36:41 UTC - in response to Message 1232810.  

Glad to have you guys back up and running. Thanks for all you do.


PROUD MEMBER OF Team Starfire World BOINC
ID: 1232818 · Report as offensive
Claggy
Volunteer tester

Send message
Joined: 5 Jul 99
Posts: 4654
Credit: 47,537,079
RAC: 4
United Kingdom
Message 1232819 - Posted: 18 May 2012, 16:36:44 UTC - in response to Message 1232810.  

To get it all up and back on line from scratch in two hours was a major feat of teamwork, I take my hat off to you and the lads, well done!


Thanks. A mixture of "all hands on deck" and incredible luck that nothing really got corrupted/fried when the power suddenly disappeared. There are some RAID resyncs happening at the moment, but looking good thus far...

- Matt

Thanks for the update, and well done for getting everything up so quick too,

Claggy
ID: 1232819 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51477
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1232821 - Posted: 18 May 2012, 16:37:57 UTC
Last modified: 18 May 2012, 16:38:40 UTC

Matt....
Could you possibly check on one thing for the kitties?
I have one rig trying to report about 390 completed tasks.
I have others that are going to have much higher numbers to report than that.

Getting HTTP server errors.
I know this might be due to heavy traffic, or the servers not quite up to speed yet.

But at one time, there were some server settings that were causing problems with reporting large numbers of completed tasks, and they were adjusted at that time.

Could you have a look and make sure that the server end is OK please?

Meow.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1232821 · Report as offensive
Lynn Curtis

Send message
Joined: 20 Feb 12
Posts: 1
Credit: 5,353
RAC: 0
United States
Message 1232825 - Posted: 18 May 2012, 16:45:31 UTC - in response to Message 1232771.  

Haven't been able to Upload to SETI via BOINC. Are there still issues with connecting to the server?
ID: 1232825 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51477
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1232827 - Posted: 18 May 2012, 16:49:58 UTC - in response to Message 1232825.  

Haven't been able to Upload to SETI via BOINC. Are there still issues with connecting to the server?

Just came back up after a two day outage and the servers are getting hammered.
Let Boinc keep trying and it should get through after a while, but it might take a bit to settle down.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1232827 · Report as offensive
Steven Proveaux

Send message
Joined: 10 Jun 99
Posts: 1
Credit: 1,335,835
RAC: 0
United States
Message 1232829 - Posted: 18 May 2012, 16:50:22 UTC

I just uploaded the latest batch of work units since the power has been restored (Friday morning) but there has been no new downloads. has anyone had a similar problem?
ID: 1232829 · Report as offensive
Claggy
Volunteer tester

Send message
Joined: 5 Jul 99
Posts: 4654
Credit: 47,537,079
RAC: 4
United Kingdom
Message 1232833 - Posted: 18 May 2012, 16:52:42 UTC - in response to Message 1232829.  

I just uploaded the latest batch of work units since the power has been restored (Friday morning) but there has been no new downloads. has anyone had a similar problem?

See msattler's reply in his last post.

Claggy
ID: 1232833 · Report as offensive
Profile Jaye Ellen

Send message
Joined: 29 Nov 08
Posts: 26
Credit: 20,945,032
RAC: 45
United States
Message 1232834 - Posted: 18 May 2012, 16:53:39 UTC - in response to Message 1232825.  

Hi all,
I also have been unable to send or rcv stuff from BOINC (probly from your power problems) --- I'm hoping the comm restarts soon !!!

Jaye Ellen
ID: 1232834 · Report as offensive
Profile perryjay
Volunteer tester
Avatar

Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 20,676,751
RAC: 0
United States
Message 1232846 - Posted: 18 May 2012, 17:11:55 UTC

I just managed to report 181 tasks and actually got 1 (one) downloaded in return. That's okay though, I'm just happy things are moving.


PROUD MEMBER OF Team Starfire World BOINC
ID: 1232846 · Report as offensive
Profile Ex: "Socialist"
Volunteer tester
Avatar

Send message
Joined: 12 Mar 12
Posts: 3433
Credit: 2,616,158
RAC: 2
United States
Message 1232850 - Posted: 18 May 2012, 17:15:47 UTC
Last modified: 18 May 2012, 17:17:39 UTC

Thanks to the scientists for a great job! +10,000!

and thanks for the update, I think some of us thought the world was ending. :-)

Glad to see things coming online, and people be patient; it just came back online after days of being down, I bet it'll be 24 hours before everyone's machines report and start crunching (getting work) again.
#resist
ID: 1232850 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51477
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1232854 - Posted: 18 May 2012, 17:23:09 UTC - in response to Message 1232821.  

Matt....
Could you possibly check on one thing for the kitties?
I have one rig trying to report about 390 completed tasks.
I have others that are going to have much higher numbers to report than that.

Getting HTTP server errors.
I know this might be due to heavy traffic, or the servers not quite up to speed yet.

But at one time, there were some server settings that were causing problems with reporting large numbers of completed tasks, and they were adjusted at that time.

Could you have a look and make sure that the server end is OK please?

Meow.

Bump before I have to leave for work...
Still getting HTTP internal server errors trying to report.
Thanks.
"Time is simply the mechanism that keeps everything from happening all at once."

ID: 1232854 · Report as offensive
Profile JaundicedEye
Avatar

Send message
Joined: 14 Mar 12
Posts: 5375
Credit: 30,870,693
RAC: 1
United States
Message 1232858 - Posted: 18 May 2012, 17:28:18 UTC - in response to Message 1232850.  

Nice job on the restore, "everything takes longer than it takes".........keep up the great support.

Until I saw the power outage news posted on the Net, I thought maybe the signal had been found...........Oh well, back to work
ID: 1232858 · Report as offensive
Jeffrey Petro

Send message
Joined: 24 Apr 12
Posts: 2
Credit: 41,248
RAC: 0
United States
Message 1232863 - Posted: 18 May 2012, 17:30:24 UTC

Am I reading this correctly that when the power went out the servers all
insta-shut?

I am shocked to learn that a facility like this has no backup...even to do a clean shut down.
ID: 1232863 · Report as offensive
1 · 2 · 3 · 4 . . . 9 · Next

Message boards : News : Major Power Outage at SSL


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.