Message boards :
News :
Major Power Outage at SSL
Message board moderation
Author | Message |
---|---|
Matt Lebofsky Send message Joined: 1 Mar 99 Posts: 1444 Credit: 957,058 RAC: 0 |
There was a major power outage on Tuesday evening that affected several buildings here on campus including the entire Space Sciences Laboratory. Power has been restored this morning, and we are slowly getting the project back on line. -- BOINC/SETI@home network/web/science/development person -- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude |
Matt Lebofsky Send message Joined: 1 Mar 99 Posts: 1444 Credit: 957,058 RAC: 0 |
More detail (from Jon - another systems administrator here at the lab): Power failed when there was a short in the power distribution lines that feed the hill buildings [including Space Sciences Laboratory, Math Science Research Institute, Lawrence Hall of Science]. The short occurred because of deterioration of the insulation on these power lines, which are buried in a conduit in the ground. The power distribution lines are thousands of feet long, originating in a substation at LBL below LHS. Most of this conduit only contain one power line, but we were lucky in that the section that failed had two power lines in it, so the electricians could just utilize the second unused line to restore electrical service. It was an arduous process finding just where the short was, since they had to disconnect one segment at a time and test it. This is why it took so long to find and repair. - Matt -- BOINC/SETI@home network/web/science/development person -- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude |
Ronald R CODNEY Send message Joined: 19 Nov 11 Posts: 87 Credit: 420,920 RAC: 0 |
Heh Matt: Is there gonna b an ISP change for uploads/downloads, or a reset requirement by the masses? Thanks for filling us in also. |
kittyman Send message Joined: 9 Jul 00 Posts: 51478 Credit: 1,018,363,574 RAC: 1,004 |
Thanks for the news, Matt. And thanks to you and Eric and Jeff for getting out of bed so early today to get the servers back online. "Time is simply the mechanism that keeps everything from happening all at once." |
Matt Lebofsky Send message Joined: 1 Mar 99 Posts: 1444 Credit: 957,058 RAC: 0 |
To get it all up and back on line from scratch in two hours was a major feat of teamwork, I take my hat off to you and the lads, well done! Thanks. A mixture of "all hands on deck" and incredible luck that nothing really got corrupted/fried when the power suddenly disappeared. There are some RAID resyncs happening at the moment, but looking good thus far... - Matt -- BOINC/SETI@home network/web/science/development person -- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude |
rob smith Send message Joined: 7 Mar 03 Posts: 22541 Credit: 416,307,556 RAC: 380 |
Glad to hear nothing got fried. Frying leading to corruption is one of the biggest hazards of a big mains cable shorting, especially one that is of any decent length. Bob Smith Member of Seti PIPPS (Pluto is a Planet Protest Society) Somewhere in the (un)known Universe? |
Keith T. Send message Joined: 23 Aug 99 Posts: 962 Credit: 537,293 RAC: 9 |
Well done to all. Uploads, reporting and Downloads are all working here. Even SETI Beta is working ! I noticed that SSL is over 50 years old now, are the cables a similar age ? |
perryjay Send message Joined: 20 Aug 02 Posts: 3377 Credit: 20,676,751 RAC: 0 |
Glad to have you guys back up and running. Thanks for all you do. PROUD MEMBER OF Team Starfire World BOINC |
Claggy Send message Joined: 5 Jul 99 Posts: 4654 Credit: 47,537,079 RAC: 4 |
To get it all up and back on line from scratch in two hours was a major feat of teamwork, I take my hat off to you and the lads, well done! Thanks for the update, and well done for getting everything up so quick too, Claggy |
kittyman Send message Joined: 9 Jul 00 Posts: 51478 Credit: 1,018,363,574 RAC: 1,004 |
Matt.... Could you possibly check on one thing for the kitties? I have one rig trying to report about 390 completed tasks. I have others that are going to have much higher numbers to report than that. Getting HTTP server errors. I know this might be due to heavy traffic, or the servers not quite up to speed yet. But at one time, there were some server settings that were causing problems with reporting large numbers of completed tasks, and they were adjusted at that time. Could you have a look and make sure that the server end is OK please? Meow. "Time is simply the mechanism that keeps everything from happening all at once." |
Lynn Curtis Send message Joined: 20 Feb 12 Posts: 1 Credit: 5,353 RAC: 0 |
Haven't been able to Upload to SETI via BOINC. Are there still issues with connecting to the server? |
kittyman Send message Joined: 9 Jul 00 Posts: 51478 Credit: 1,018,363,574 RAC: 1,004 |
Haven't been able to Upload to SETI via BOINC. Are there still issues with connecting to the server? Just came back up after a two day outage and the servers are getting hammered. Let Boinc keep trying and it should get through after a while, but it might take a bit to settle down. "Time is simply the mechanism that keeps everything from happening all at once." |
Steven Proveaux Send message Joined: 10 Jun 99 Posts: 1 Credit: 1,335,835 RAC: 0 |
I just uploaded the latest batch of work units since the power has been restored (Friday morning) but there has been no new downloads. has anyone had a similar problem? |
Claggy Send message Joined: 5 Jul 99 Posts: 4654 Credit: 47,537,079 RAC: 4 |
I just uploaded the latest batch of work units since the power has been restored (Friday morning) but there has been no new downloads. has anyone had a similar problem? See msattler's reply in his last post. Claggy |
Jaye Ellen Send message Joined: 29 Nov 08 Posts: 26 Credit: 20,945,032 RAC: 45 |
Hi all, I also have been unable to send or rcv stuff from BOINC (probly from your power problems) --- I'm hoping the comm restarts soon !!! Jaye Ellen |
perryjay Send message Joined: 20 Aug 02 Posts: 3377 Credit: 20,676,751 RAC: 0 |
I just managed to report 181 tasks and actually got 1 (one) downloaded in return. That's okay though, I'm just happy things are moving. PROUD MEMBER OF Team Starfire World BOINC |
Ex: "Socialist" Send message Joined: 12 Mar 12 Posts: 3433 Credit: 2,616,158 RAC: 2 |
Thanks to the scientists for a great job! +10,000! and thanks for the update, I think some of us thought the world was ending. :-) Glad to see things coming online, and people be patient; it just came back online after days of being down, I bet it'll be 24 hours before everyone's machines report and start crunching (getting work) again. #resist |
kittyman Send message Joined: 9 Jul 00 Posts: 51478 Credit: 1,018,363,574 RAC: 1,004 |
Matt.... Bump before I have to leave for work... Still getting HTTP internal server errors trying to report. Thanks. "Time is simply the mechanism that keeps everything from happening all at once." |
JaundicedEye Send message Joined: 14 Mar 12 Posts: 5375 Credit: 30,870,693 RAC: 1 |
Nice job on the restore, "everything takes longer than it takes".........keep up the great support. Until I saw the power outage news posted on the Net, I thought maybe the signal had been found...........Oh well, back to work |
Jeffrey Petro Send message Joined: 24 Apr 12 Posts: 2 Credit: 41,248 RAC: 0 |
Am I reading this correctly that when the power went out the servers all insta-shut? I am shocked to learn that a facility like this has no backup...even to do a clean shut down. |
©2024 University of California
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.