Major Power Outage at SSL

Message boards : News : Major Power Outage at SSL

To post messages, you must log in.

1 · 2 · 3 · 4 . . . 9 · Next

AuthorMessage
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1441
Credit: 213,689
RAC: 0
United States
Message 1232771 - Posted: 18 May 2012, 15:57:18 UTC

There was a major power outage on Tuesday evening that affected several buildings here on campus including the entire Space Sciences Laboratory. Power has been restored this morning, and we are slowly getting the project back on line.


-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

ID: 1232771 · Report as offensive
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1441
Credit: 213,689
RAC: 0
United States
Message 1232776 - Posted: 18 May 2012, 15:59:18 UTC

More detail (from Jon - another systems administrator here at the lab):

Power failed when there was a short in the power distribution lines that feed the hill buildings [including Space Sciences Laboratory, Math Science Research Institute, Lawrence Hall of Science]. The short occurred because of deterioration of the insulation on these power lines, which are buried in a conduit in the ground. The power distribution lines are thousands of feet long, originating in a substation at LBL below LHS. Most of this conduit only contain one power line, but we were lucky in that the section that failed had two power lines in it, so the electricians could just utilize the second unused line to restore electrical service. It was an arduous process finding just where the short was, since they had to disconnect one segment at a time and test it. This is why it took so long to find and repair.

- Matt


-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

ID: 1232776 · Report as offensive
Profile Ronald R CODNEY
Avatar

Send message
Joined: 19 Nov 11
Posts: 87
Credit: 420,920
RAC: 0
United States
Message 1232792 - Posted: 18 May 2012, 16:10:04 UTC

Heh Matt:
Is there gonna b an ISP change for uploads/downloads, or a reset requirement by the masses?

Thanks for filling us in also.

ID: 1232792 · Report as offensive
Profile Chris SCrowdfunding Project Donor
Volunteer tester
Avatar

Send message
Joined: 19 Nov 00
Posts: 38176
Credit: 21,234,496
RAC: 27,834
United Kingdom
Message 1232797 - Posted: 18 May 2012, 16:12:56 UTC

To get it all up and back on line from scratch in two hours was a major feat of teamwork, I take my hat off to you and the lads, well done!

ID: 1232797 · Report as offensive
kittymanProject Donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 45861
Credit: 814,599,745
RAC: 121,869
United States
Message 1232809 - Posted: 18 May 2012, 16:23:26 UTC
Last modified: 18 May 2012, 16:23:44 UTC

Thanks for the news, Matt.
And thanks to you and Eric and Jeff for getting out of bed so early today to get the servers back online.


Kitties make wonderful traveling companions on your journey through life.

Have made a few friends in this life.
Most were cats.

ID: 1232809 · Report as offensive
Profile Matt Lebofsky
Volunteer moderator
Project administrator
Project developer
Project scientist
Avatar

Send message
Joined: 1 Mar 99
Posts: 1441
Credit: 213,689
RAC: 0
United States
Message 1232810 - Posted: 18 May 2012, 16:24:08 UTC - in response to Message 1232797.

To get it all up and back on line from scratch in two hours was a major feat of teamwork, I take my hat off to you and the lads, well done!


Thanks. A mixture of "all hands on deck" and incredible luck that nothing really got corrupted/fried when the power suddenly disappeared. There are some RAID resyncs happening at the moment, but looking good thus far...

- Matt
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

ID: 1232810 · Report as offensive
rob smithProject Donor
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 13300
Credit: 154,138,276
RAC: 112,432
United Kingdom
Message 1232815 - Posted: 18 May 2012, 16:34:33 UTC

Glad to hear nothing got fried.
Frying leading to corruption is one of the biggest hazards of a big mains cable shorting, especially one that is of any decent length.


Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

ID: 1232815 · Report as offensive
Profile Keith T.
Volunteer tester
Avatar

Send message
Joined: 23 Aug 99
Posts: 743
Credit: 244,276
RAC: 0
United Kingdom
Message 1232817 - Posted: 18 May 2012, 16:36:27 UTC

Well done to all.

Uploads, reporting and Downloads are all working here. Even SETI Beta is working !

I noticed that SSL is over 50 years old now, are the cables a similar age ?

ID: 1232817 · Report as offensive
Profile perryjay
Volunteer tester
Avatar

Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 20,676,751
RAC: 0
United States
Message 1232818 - Posted: 18 May 2012, 16:36:41 UTC - in response to Message 1232810.

Glad to have you guys back up and running. Thanks for all you do.




PROUD MEMBER OF Team Starfire World BOINC

ID: 1232818 · Report as offensive
ClaggyProject Donor
Volunteer tester

Send message
Joined: 5 Jul 99
Posts: 4622
Credit: 46,334,291
RAC: 2,939
United Kingdom
Message 1232819 - Posted: 18 May 2012, 16:36:44 UTC - in response to Message 1232810.

To get it all up and back on line from scratch in two hours was a major feat of teamwork, I take my hat off to you and the lads, well done!


Thanks. A mixture of "all hands on deck" and incredible luck that nothing really got corrupted/fried when the power suddenly disappeared. There are some RAID resyncs happening at the moment, but looking good thus far...

- Matt

Thanks for the update, and well done for getting everything up so quick too,

Claggy

ID: 1232819 · Report as offensive
kittymanProject Donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 45861
Credit: 814,599,745
RAC: 121,869
United States
Message 1232821 - Posted: 18 May 2012, 16:37:57 UTC
Last modified: 18 May 2012, 16:38:40 UTC

Matt....
Could you possibly check on one thing for the kitties?
I have one rig trying to report about 390 completed tasks.
I have others that are going to have much higher numbers to report than that.

Getting HTTP server errors.
I know this might be due to heavy traffic, or the servers not quite up to speed yet.

But at one time, there were some server settings that were causing problems with reporting large numbers of completed tasks, and they were adjusted at that time.

Could you have a look and make sure that the server end is OK please?

Meow.


Kitties make wonderful traveling companions on your journey through life.

Have made a few friends in this life.
Most were cats.

ID: 1232821 · Report as offensive
Lynn Curtis

Send message
Joined: 20 Feb 12
Posts: 1
Credit: 5,353
RAC: 0
United States
Message 1232825 - Posted: 18 May 2012, 16:45:31 UTC - in response to Message 1232771.

Haven't been able to Upload to SETI via BOINC. Are there still issues with connecting to the server?

ID: 1232825 · Report as offensive
kittymanProject Donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 45861
Credit: 814,599,745
RAC: 121,869
United States
Message 1232827 - Posted: 18 May 2012, 16:49:58 UTC - in response to Message 1232825.

Haven't been able to Upload to SETI via BOINC. Are there still issues with connecting to the server?

Just came back up after a two day outage and the servers are getting hammered.
Let Boinc keep trying and it should get through after a while, but it might take a bit to settle down.
Kitties make wonderful traveling companions on your journey through life.

Have made a few friends in this life.
Most were cats.

ID: 1232827 · Report as offensive
Steven Proveaux

Send message
Joined: 10 Jun 99
Posts: 1
Credit: 909,913
RAC: 306
United States
Message 1232829 - Posted: 18 May 2012, 16:50:22 UTC

I just uploaded the latest batch of work units since the power has been restored (Friday morning) but there has been no new downloads. has anyone had a similar problem?


ID: 1232829 · Report as offensive
ClaggyProject Donor
Volunteer tester

Send message
Joined: 5 Jul 99
Posts: 4622
Credit: 46,334,291
RAC: 2,939
United Kingdom
Message 1232833 - Posted: 18 May 2012, 16:52:42 UTC - in response to Message 1232829.

I just uploaded the latest batch of work units since the power has been restored (Friday morning) but there has been no new downloads. has anyone had a similar problem?

See msattler's reply in his last post.

Claggy

ID: 1232833 · Report as offensive
Profile Jaye Ellen

Send message
Joined: 29 Nov 08
Posts: 24
Credit: 12,131,403
RAC: 6,506
United States
Message 1232834 - Posted: 18 May 2012, 16:53:39 UTC - in response to Message 1232825.

Hi all,
I also have been unable to send or rcv stuff from BOINC (probly from your power problems) --- I'm hoping the comm restarts soon !!!

Jaye Ellen

ID: 1232834 · Report as offensive
Profile perryjay
Volunteer tester
Avatar

Send message
Joined: 20 Aug 02
Posts: 3377
Credit: 20,676,751
RAC: 0
United States
Message 1232846 - Posted: 18 May 2012, 17:11:55 UTC

I just managed to report 181 tasks and actually got 1 (one) downloaded in return. That's okay though, I'm just happy things are moving.




PROUD MEMBER OF Team Starfire World BOINC

ID: 1232846 · Report as offensive
Profile Ex
Volunteer tester
Avatar

Send message
Joined: 12 Mar 12
Posts: 2895
Credit: 2,077,828
RAC: 433
United States
Message 1232850 - Posted: 18 May 2012, 17:15:47 UTC
Last modified: 18 May 2012, 17:17:39 UTC

Thanks to the scientists for a great job! +10,000!

and thanks for the update, I think some of us thought the world was ending. :-)

Glad to see things coming online, and people be patient; it just came back online after days of being down, I bet it'll be 24 hours before everyone's machines report and start crunching (getting work) again.


-Dave #2

3.2.0-33

ID: 1232850 · Report as offensive
kittymanProject Donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 45861
Credit: 814,599,745
RAC: 121,869
United States
Message 1232854 - Posted: 18 May 2012, 17:23:09 UTC - in response to Message 1232821.

Matt....
Could you possibly check on one thing for the kitties?
I have one rig trying to report about 390 completed tasks.
I have others that are going to have much higher numbers to report than that.

Getting HTTP server errors.
I know this might be due to heavy traffic, or the servers not quite up to speed yet.

But at one time, there were some server settings that were causing problems with reporting large numbers of completed tasks, and they were adjusted at that time.

Could you have a look and make sure that the server end is OK please?

Meow.

Bump before I have to leave for work...
Still getting HTTP internal server errors trying to report.
Thanks.
Kitties make wonderful traveling companions on your journey through life.

Have made a few friends in this life.
Most were cats.

ID: 1232854 · Report as offensive
Profile JaundicedEyeProject Donor
Avatar

Send message
Joined: 14 Mar 12
Posts: 2404
Credit: 18,949,434
RAC: 9,510
United States
Message 1232858 - Posted: 18 May 2012, 17:28:18 UTC - in response to Message 1232850.

Nice job on the restore, "everything takes longer than it takes".........keep up the great support.

Until I saw the power outage news posted on the Net, I thought maybe the signal had been found...........Oh well, back to work

ID: 1232858 · Report as offensive
1 · 2 · 3 · 4 . . . 9 · Next

Message boards : News : Major Power Outage at SSL


 
©2016 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.