We are now mostly recovered from a campus wide power outage.


log in

Advanced search

Message boards : News : We are now mostly recovered from a campus wide power outage.

1 · 2 · 3 · Next
Author Message
Jeff Cobb
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 1 Mar 99
Posts: 110
Credit: 40,367
RAC: 0
United States
Message 1422770 - Posted: 1 Oct 2013, 21:48:07 UTC

Last night there was a power outage that affected the entire Berkeley campus. The data center, where our servers are located, does have facility wide UPS so all servers stayed up. Unfortunately, the data center air conditioning did not stay up. Machines were getting hot, so the data center staff had to bring them down.
____________

Jeff Cobb
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 1 Mar 99
Posts: 110
Credit: 40,367
RAC: 0
United States
Message 1422771 - Posted: 1 Oct 2013, 21:48:57 UTC

All of our servers seem to be OK with exception of marvin, the AstroPulse database server. Marvin appears to have been rendered nonfunctional, although we have reason to believe that the disks are OK. In any case, we have a backup of the database. We will be bringing marvin back to the lab tomorrow for a postmortem.
____________

QSilver
Send message
Joined: 26 May 99
Posts: 231
Credit: 4,829,228
RAC: 2,621
United States
Message 1422777 - Posted: 1 Oct 2013, 21:57:23 UTC - in response to Message 1422771.

Thanks for the updates, Jeff.
____________

Profile Ageless
Avatar
Send message
Joined: 9 Jun 99
Posts: 12392
Credit: 2,665,317
RAC: 984
Netherlands
Message 1422793 - Posted: 1 Oct 2013, 22:25:04 UTC
Last modified: 1 Oct 2013, 22:32:55 UTC

Welcome back!

Ah, and you do know it'll only be temporarily? That Thursday California will rattle to a 9.7 Richter earthquake? (moves hands in a conjuring manner) ;-)
____________
Jord

Fighting for the correct use of the apostrophe, together with Weird Al Yankovic

N9JFE David SProject donor
Volunteer tester
Avatar
Send message
Joined: 4 Oct 99
Posts: 12447
Credit: 14,823,777
RAC: 4,752
United States
Message 1422840 - Posted: 2 Oct 2013, 0:23:12 UTC
Last modified: 2 Oct 2013, 0:28:12 UTC

Thanks for the info, Jeff. Is the effort to revive marvin the reason the outage was longer than it has usually been lately?

[edit]
Well, okay, I know you had to bring all the servers up, but since they were shut down properly (right?) that shouldn't have been too bad.
____________
David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.


Profile Donald L. JohnsonProject donor
Avatar
Send message
Joined: 5 Aug 02
Posts: 6324
Credit: 768,982
RAC: 965
United States
Message 1422859 - Posted: 2 Oct 2013, 1:04:44 UTC - in response to Message 1422793.
Last modified: 2 Oct 2013, 1:11:40 UTC

Welcome back!

Ah, and you do know it'll only be temporarily? That Thursday California will rattle to a 9.7 Richter earthquake? (moves hands in a conjuring manner) ;-)

My Wiccan friends on Mt. Tam say "Nay, nay, moosebreath".
But I have both my cars gassed up and ready to head up the hill if need be.
____________
Donald
Infernal Optimist / Submariner, retired

{BDC} Thomas DupontProject donor
Volunteer tester
Avatar
Send message
Joined: 9 Dec 11
Posts: 3985
Credit: 1,345,275
RAC: 784
France
Message 1422965 - Posted: 2 Oct 2013, 5:56:04 UTC - in response to Message 1422770.

Thanks for the heads-up Jeff ! :)
Glad to see everyone !
From France about this out(r)age >> http://www.meltycampus.fr/etats-unis-explosion-sur-le-campus-de-berkeley-video-a215177.html :(
____________

Profile Chris SProject donor
Volunteer tester
Avatar
Send message
Joined: 19 Nov 00
Posts: 32302
Credit: 14,269,523
RAC: 11,823
United Kingdom
Message 1422996 - Posted: 2 Oct 2013, 8:27:08 UTC

Many thanks Jeff, much appreciated.



rob smithProject donor
Volunteer tester
Send message
Joined: 7 Mar 03
Posts: 8732
Credit: 61,622,376
RAC: 55,178
United Kingdom
Message 1423099 - Posted: 2 Oct 2013, 15:44:05 UTC - in response to Message 1422771.

All of our servers seem to be OK with exception of marvin, the AstroPulse database server. Marvin appears to have been rendered nonfunctional, although we have reason to believe that the disks are OK. In any case, we have a backup of the database. We will be bringing marvin back to the lab tomorrow for a postmortem.



Thanks for the news, and your efforts.

Your description of performing a post-mortem brings all sorts of strange images to mind involving, yourself, Eric, Matt and surgical garb....
I hope it goes well, and doesn't turn out to be a post-mortem, but a successful surgical intervention leading to a complete recovery by the patient.
____________
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?

Profile Mr. KevvyProject donor
Volunteer tester
Avatar
Send message
Joined: 15 May 99
Posts: 731
Credit: 78,414,337
RAC: 49,400
Canada
Message 1423131 - Posted: 2 Oct 2013, 16:42:51 UTC - in response to Message 1422770.
Last modified: 2 Oct 2013, 16:46:56 UTC

Last night there was a power outage that affected the entire Berkeley campus.


Considering that both NASA and the NSF are down right now due to the Fed shutdown, this outage was more than a little disturbing at the time, and I'm glad it was only a coincidence!

Hope the NSF downtime doesn't impact this project.

All of our servers seem to be OK with exception of marvin, the AstroPulse database server.


And to add to the coincidence, there's the NSF-funded part of the project, too.
____________
“Never doubt that a small group of thoughtful, committed citizens can change the world; indeed, it's the only thing that ever has.”
--- Margaret Mead

Profile S@NL Etienne Dokkum
Volunteer tester
Avatar
Send message
Joined: 11 Jun 99
Posts: 174
Credit: 17,393,079
RAC: 9,028
Netherlands
Message 1423146 - Posted: 2 Oct 2013, 17:23:38 UTC

Thanks for keeping us up to date Jeff ! Best of luck bringing Marvin back from the after life...

Hope to hear good news soon !
____________

Profile KWSN THE Holy Hand Grenade!
Volunteer tester
Avatar
Send message
Joined: 20 Dec 05
Posts: 1988
Credit: 10,977,519
RAC: 14,769
United States
Message 1423176 - Posted: 2 Oct 2013, 17:58:57 UTC

Are the problems with Marvin preventing Beta from coming up? If not, could someone please start it???
____________
.

Sten-Arne
Volunteer tester
Send message
Joined: 1 Nov 08
Posts: 3675
Credit: 21,179,914
RAC: 8,515
Sweden
Message 1423180 - Posted: 2 Oct 2013, 18:11:13 UTC - in response to Message 1423176.
Last modified: 2 Oct 2013, 18:11:56 UTC

Are the problems with Marvin preventing Beta from coming up? If not, could someone please start it???


Beta has been up since the outage ended. In fact, Beta came back up before Main.
____________

Profile KWSN THE Holy Hand Grenade!
Volunteer tester
Avatar
Send message
Joined: 20 Dec 05
Posts: 1988
Credit: 10,977,519
RAC: 14,769
United States
Message 1423237 - Posted: 2 Oct 2013, 19:54:04 UTC - in response to Message 1423180.

Are the problems with Marvin preventing Beta from coming up? If not, could someone please start it???


Beta has been up since the outage ended. In fact, Beta came back up before Main.


Not if you look at Beta's "Server Status" page...
____________
.

Sten-Arne
Volunteer tester
Send message
Joined: 1 Nov 08
Posts: 3675
Credit: 21,179,914
RAC: 8,515
Sweden
Message 1423238 - Posted: 2 Oct 2013, 19:58:31 UTC - in response to Message 1423237.

Are the problems with Marvin preventing Beta from coming up? If not, could someone please start it???


Beta has been up since the outage ended. In fact, Beta came back up before Main.


Not if you look at Beta's "Server Status" page...


Reality always beats a server status page, that always has been, let's say not exactly in tune with what really happens.
____________

ClaggyProject donor
Volunteer tester
Send message
Joined: 5 Jul 99
Posts: 4207
Credit: 34,463,143
RAC: 20,534
United Kingdom
Message 1423246 - Posted: 2 Oct 2013, 20:16:04 UTC - in response to Message 1423237.
Last modified: 2 Oct 2013, 20:25:52 UTC

Are the problems with Marvin preventing Beta from coming up? If not, could someone please start it???


Beta has been up since the outage ended. In fact, Beta came back up before Main.


Not if you look at Beta's "Server Status" page...

Seti Beta's Server Status page has looked like that for years, the only entries that ever show as green are:

data-driven web pages
vote_monitor
splitter_throttle

Sometimes a splitter will show as green, but since Seti Beta is small scale not a lot of Wu's need to be split, so mostly stay orange or red,
the rest don't point to the actual process being run, even with beta_validate_v7 showing red, v7 Wu still validate there:

server status

The last conversation I had with Eric about the Server Status page. Beta is running on Bruno. The scripts that control the Deamons for the various services were conflicting with Seti Main. By disabling the reporting protion of the scripts they could run the Service Deamons, the Beta status page would not reflect what is or is not running.

I will poke an email in Eric's direction Monday.

Regards


Claggy

Profile CornhuskerProject donor
Send message
Joined: 20 Apr 09
Posts: 18
Credit: 17,696,231
RAC: 11,488
United States
Message 1423885 - Posted: 4 Oct 2013, 3:53:41 UTC

I still think you guys need to run an extension cord out here to Nebraska! :)
____________

Sten-Arne
Volunteer tester
Send message
Joined: 1 Nov 08
Posts: 3675
Credit: 21,179,914
RAC: 8,515
Sweden
Message 1424142 - Posted: 4 Oct 2013, 19:21:12 UTC
Last modified: 4 Oct 2013, 19:25:27 UTC

Marvin is still MIA, Astropulse science database continues to be disabled. Is there any updates when it comes to that?

Did Marvin end up as a total write-off, or?
____________

N9JFE David SProject donor
Volunteer tester
Avatar
Send message
Joined: 4 Oct 99
Posts: 12447
Credit: 14,823,777
RAC: 4,752
United States
Message 1424152 - Posted: 4 Oct 2013, 19:54:26 UTC
Last modified: 4 Oct 2013, 19:54:56 UTC

I would have expected that by now, they would have the AP science database up and running on some other machine, if marvin is beyond repair.
____________
David
Sitting on my butt while others boldly go,
Waiting for a message from a small furry creature from Alpha Centauri.


1 · 2 · 3 · Next

Message boards : News : We are now mostly recovered from a campus wide power outage.

Copyright © 2014 University of California