Panic Mode On (88) Server Problems?

Message boards : Number crunching : Panic Mode On (88) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 12 · 13 · 14 · 15 · 16 · 17 · 18 . . . 21 · Next

AuthorMessage
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1544013 - Posted: 18 Jul 2014, 21:18:30 UTC - in response to Message 1544008.  

Now THIS is mighty odd....
The Cricket graphs appear to have dropped dead about an hour and a half ago, but work is still flowing.
Can't really recall that scenario before.

I am going to guess that it is just an error in the cricket collection of the SNMP data.

I suspect so. The Crickets live on a server too, and it has gone down once in a long while.

Actually it looks like we got moved to the other router.
http://fragment1.berkeley.edu/newcricket/grapher.cgi?target=/router-interfaces/inr-210/gigabitethernet6_17&ranges=d%3Aw&view=Octets
That port is also named srb-ssi-seti-net and it looks like it picks up exactly where the other one ended.

Hmmmm....another Cricket graph bookmark to save.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1544013 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1544032 - Posted: 18 Jul 2014, 22:16:19 UTC - in response to Message 1544013.  

It may not be permanent. From IST:

Unscheduled Outage Earl Warren Hall Data Center
Posted by CSS IT ~az
On 7/18/2014 at 3:05 pm PST
Modified on 7/18/2014 at 3:07 pm PST
Modified by CSS IT ~az
Posted in Unscheduled Outage
Outage Type: UNSCHEDULED OUTAGE
Date Submitted: Friday, July 18, 2014
Outage Start/End Time: 1330 – TBD
Groups Impacted: 2000 Center Street
Equipment: srb2ds (Data Center Distribution Switch)

Description: The campus is currently experiencing issues with one of the two distribution switches at the core of the Data Center network.

This should not have a major impact to network connectivity, because the second redundant switch remains functional.

Members of the Network Operations and Services group are investigating the cause of the issue and working to restore connectivity as soon as possible.

Earl Warren Data Center
inr-210 SRB1
inr-211 SRB1

Since we simply switched from inr-211 to inr-210, we might go back again. Or maybe not until the other one fails....
ID: 1544032 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1544042 - Posted: 18 Jul 2014, 22:33:07 UTC - in response to Message 1544032.  
Last modified: 18 Jul 2014, 22:36:53 UTC


Since we simply switched from inr-211 to inr-210, we might go back again. Or maybe not until the other one fails....

Well, thank goodness, being a data center, they had a redundant switch.
I'll keep the other link at the ready if this one goes blank.
Thank you for the info, Richard. (And HAL9000 for sleuthing out the alternate graph!)
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1544042 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1544046 - Posted: 18 Jul 2014, 22:43:37 UTC

I noticed a large number of other ports seemed to switch as well. Not even ones on the same asic or even in the same blade. Looks like they might be looking at a backplane failure.

Since this equipment is handled by the campus IT staff. I imagine they have it under a service contract & will have the full redundancy restored soon.

If the servers were still in the lab and this happened we probably would have to have an emergency fundraiser to get replacement equipment.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1544046 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1544066 - Posted: 18 Jul 2014, 23:47:29 UTC

We seem to be back on 211.
ID: 1544066 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1544314 - Posted: 19 Jul 2014, 13:12:01 UTC - in response to Message 1544066.  

We seem to be back on 211.

And...switching back to the old graph linky.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1544314 · Report as offensive
Josef W. Segur
Volunteer developer
Volunteer tester

Send message
Joined: 30 Oct 99
Posts: 4504
Credit: 1,414,761
RAC: 0
United States
Message 1546133 - Posted: 23 Jul 2014, 2:05:53 UTC

For anyone who may have missed it, the front page description of the outages due to power work at the data center has been updated:

Update: Planned Power Outage - July 23rd-24th, 2014
Due to necessary electrical repairs at the data center, we need to shut down some servers for a brief period the morning of Wednesday, July 23rd, then again for a longer period throughout the work day of Thursday, July 24th. All the SETI data servers will be unreachable during these outages. The work should be complete by Thursday afternoon, and all services back on line shortly afterward.

22 Jul 2014, 21:01:05 UTC

                                                                   Joe
ID: 1546133 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1639
Credit: 12,921,799
RAC: 89
New Zealand
Message 1546504 - Posted: 23 Jul 2014, 21:37:19 UTC
Last modified: 23 Jul 2014, 21:41:19 UTC

Thanks for the reminder Joe. Does anybody anybody know for sure what time the servers are being turned off? I am thinking possibly just before 5 o'clock? This would make it in 2 hours and 23 minutes. As of the time of this post. Making it 00:00 UTC time 24/7/14
ID: 1546504 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11358
Credit: 29,581,041
RAC: 66
United States
Message 1546578 - Posted: 24 Jul 2014, 0:07:28 UTC

We are still up and APs are being split.
ID: 1546578 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1546613 - Posted: 24 Jul 2014, 1:32:02 UTC - in response to Message 1546504.  

Thanks for the reminder Joe. Does anybody anybody know for sure what time the servers are being turned off? I am thinking possibly just before 5 o'clock? This would make it in 2 hours and 23 minutes. As of the time of this post. Making it 00:00 UTC time 24/7/14

longer period throughout the work day of Thursday, July 24th

Here in the US the work day is generally considered 08:00 to 17:00. Given that they are UTC -7 this time of year it. Those times would put in the second outage from 2014-07-24 15:00 to 2014-07-25 00:00 UTC. + or - whenever they feel like starting to whenever they are done.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1546613 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1546782 - Posted: 24 Jul 2014, 6:36:53 UTC - in response to Message 1546613.  

Here in the US the work day is generally considered 08:00 to 17:00.

What happened to 9 to 5?
Grant
Darwin NT
ID: 1546782 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1546805 - Posted: 24 Jul 2014, 7:12:11 UTC - in response to Message 1546782.  

Here in the US the work day is generally considered 08:00 to 17:00.

What happened to 9 to 5?

It's an old relic from long ago whereby lunch break was included in your hours, as long as you didn't take too long (think like how iron workers building a sky scraper work to a point and then sit down and open their lunchbox and eat lunch for a few minutes and then go back to work).

At some point along the line, it was decided to make it 8 to 5 with an hour of non-paid lunch thrown into the middle somewhere, which can be handy if you have a quick errand you need to run midday (like going to the bank or something, which also closes at 5). So the work day is now 8 to 5 and since pretty much everyone goes to lunch between 11 and 1, often your quick errands end up taking a lot longer than you expected (fast-food queues are ridiculous during those times) or going to the bank takes forever because there is ONE teller still behind the counter and the other four have gone to lunch at the same time).


But that's enough of a detour. Yes, the song popularizes it as 9-5, but it is actually 8-5 because of an hour of off-the-clock lunch.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1546805 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1546813 - Posted: 24 Jul 2014, 7:36:59 UTC - in response to Message 1546805.  

Ah, probably why it's mostly 8:30 to 5:00 here. 30min for lunch.
Grant
Darwin NT
ID: 1546813 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1547095 - Posted: 25 Jul 2014, 1:36:39 UTC - in response to Message 1546813.  

Ah, probably why it's mostly 8:30 to 5:00 here. 30min for lunch.

I'm not old enough to ever know of the 9 to 5 work days. Other than in the song.
To make things even more weird. My work hours are actually 7:45 to 4:45.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1547095 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1547100 - Posted: 25 Jul 2014, 1:53:51 UTC - in response to Message 1547095.  

To make things even more weird. My work hours are actually 7:45 to 4:45.

I'm a Postie (US= mailman), so my start time is 05:30. Finish time is when I run out of mail- anywhere from 11:00 to 18:00.



:-)
Grant
Darwin NT
ID: 1547100 · Report as offensive
Profile James Sotherden
Avatar

Send message
Joined: 16 May 99
Posts: 10436
Credit: 110,373,059
RAC: 54
United States
Message 1547261 - Posted: 25 Jul 2014, 7:17:14 UTC - in response to Message 1546782.  

Here in the US the work day is generally considered 08:00 to 17:00.

What happened to 9 to 5?

I work from 2:30 PM untill 10:30 PM I get a paid lunch.
[/quote]

Old James
ID: 1547261 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1547270 - Posted: 25 Jul 2014, 7:40:18 UTC
Last modified: 25 Jul 2014, 7:40:31 UTC

I work from when the boss man calls me to start until he tells me I can quit.
Lately, that's been from 10 to 12 hours a day...
Busy times at the fire truck factory.

Oh, and on topic...
I hope the AP splitters can ramp up and keep up with demand to fill the RTS queue. Otherwise I am going to have to drop a bug in Eric's ear come morning.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1547270 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1547312 - Posted: 25 Jul 2014, 9:17:20 UTC - in response to Message 1547270.  

I hope the AP splitters can ramp up and keep up with demand to fill the RTS queue. Otherwise I am going to have to drop a bug in Eric's ear come morning.

The PFB splitters for MB have been struggling ever since they came on line.
Grant
Darwin NT
ID: 1547312 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1849
Credit: 268,616,081
RAC: 1,349
United States
Message 1547326 - Posted: 25 Jul 2014, 9:43:03 UTC - in response to Message 1547312.  
Last modified: 25 Jul 2014, 9:44:56 UTC

I hope the AP splitters can ramp up and keep up with demand to fill the RTS queue. Otherwise I am going to have to drop a bug in Eric's ear come morning.

The PFB splitters for MB have been struggling ever since they came on line.

Not especially related, but you do have to wonder if it isn't about time to pull down the V6 assimilators? Afaik, V6 work is finally finished?
ID: 1547326 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1547385 - Posted: 25 Jul 2014, 13:45:03 UTC - in response to Message 1547326.  

I hope the AP splitters can ramp up and keep up with demand to fill the RTS queue. Otherwise I am going to have to drop a bug in Eric's ear come morning.

The PFB splitters for MB have been struggling ever since they came on line.

Not especially related, but you do have to wonder if it isn't about time to pull down the V6 assimilators? Afaik, V6 work is finally finished?

Not related at all.
Sending message to Eric.
Meow.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1547385 · Report as offensive
Previous · 1 . . . 12 · 13 · 14 · 15 · 16 · 17 · 18 . . . 21 · Next

Message boards : Number crunching : Panic Mode On (88) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.