Panic Mode On (112) Server Problems?

Message boards : Number crunching : Panic Mode On (112) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 . . . 33 · Next

AuthorMessage
Dr Who Fan
Volunteer tester
Avatar

Send message
Joined: 8 Jan 01
Posts: 3251
Credit: 715,342
RAC: 4
United States
Message 1932885 - Posted: 30 Apr 2018, 15:22:57 UTC

Like everyone else posting here, I am also having problems reaching both the dev's and this web site without getting the "site took to long to respond" error.

Just did a Tracert and I do not recall seeing the "Request timed out" error message after reaching the Berkeley campus 128.x.x.x stuff and trying to reach the SETI servers before...

Tracing route to setiathome.berkeley.edu [208.68.240.110]

  8     *        *        *     Request timed out.
  9    52 ms    53 ms    53 ms  CENIC.ear1.LosAngeles1.Level3.net [4.35.156.66]
 10    54 ms    54 ms    53 ms  dc-lax-agg8--lax-agg6-100ge-#2.cenic.net [137.164.11.7]
 11    60 ms    60 ms    62 ms  dc-svl-agg8--lax-agg8--100g--#2.cenic.net [137.164.11.21]
 12    60 ms    61 ms    60 ms  dc-svl-agg4--svl-agg8-100ge-#1.cenic.net [137.164.11.28]
 13    60 ms    62 ms    61 ms  oak-agg4--svl-agg4-100ge-#2.cenic.net [137.164.46.167]
 14    63 ms    62 ms    62 ms  ucb--oak-agg4-10g.cenic.net [137.164.50.31]
 15    62 ms    61 ms    62 ms  t2-3.inr-201-sut.berkeley.edu [128.32.0.37]
 16    63 ms    64 ms    64 ms  et3-48.inr-311-ewdc.berkeley.edu [128.32.0.101]
 17     *        *        *     Request timed out.
 18     *        *        *     Request timed out.
 19     *        *        *     Request timed out.
 20     *        *        *     Request timed out.
 21     *        *       63 ms  muarae1.ssl.berkeley.edu [208.68.240.110]

Trace complete.

ID: 1932885 · Report as offensive
Profile JaundicedEye
Avatar

Send message
Joined: 14 Mar 12
Posts: 5375
Credit: 30,870,693
RAC: 1
United States
Message 1932886 - Posted: 30 Apr 2018, 15:25:55 UTC

I sense a disturbance in the Force...I do.

Uploads stalling, very slow response time on the site.

Anybody else?

"Sour Grapes make a bitter Whine." <(0)>
ID: 1932886 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51469
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1932887 - Posted: 30 Apr 2018, 15:26:19 UTC

Active
Unscheduled Outage Intermittent Network Issues
Posted by CSS IT ~ mp
On 4/30/2018 at 7:14 am PST
Posted in Unscheduled Outage

Outage Type: UNSCHEDULED OUTAGE
Date Submitted: Monday, April 30, 2018
Outage Start/End Time: 0700 –
Groups Impacted: Campus
Equipment:

Description: The Service Desk are receiving calls of intermittent network issues.

IST staff is working quickly to identify the source of the problem and to restore services as quickly as possible.

CMR:TBD
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1932887 · Report as offensive
Profile JaundicedEye
Avatar

Send message
Joined: 14 Mar 12
Posts: 5375
Credit: 30,870,693
RAC: 1
United States
Message 1932888 - Posted: 30 Apr 2018, 15:29:12 UTC

.....Meow.........thanks.

"Sour Grapes make a bitter Whine." <(0)>
ID: 1932888 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1932891 - Posted: 30 Apr 2018, 15:39:54 UTC

I started seeing issues with one or two machines not getting work because they were not asking because of server backoffs last night. Took a restart of BOINC to get them to ask for the missing work. Still having issues this morning. UL's very slow. Thanks kittyman for the IST update.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1932891 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1932892 - Posted: 30 Apr 2018, 16:11:12 UTC - in response to Message 1932887.  

Thanks for the update Meowman :)
ID: 1932892 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51469
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1932896 - Posted: 30 Apr 2018, 16:21:24 UTC

Active
Unscheduled Outage Intermittent Network Issues
Posted by CSS IT ~ mp
On 4/30/2018 at 7:14 am PST
Modified on 4/30/2018 at 8:19 am PST
Modified by CSS IT ~ mp
Posted in Unscheduled Outage

Outage Type: UNSCHEDULED OUTAGE (AMENDED)
Date Submitted: Monday, April 30, 2018
Outage Start/End Time: 0700 – TBD
Groups Impacted: Campus
Equipment: Campus Network

Description: UPDATE: Monday, April 30, 2018 9:17am – The Service Desk continues to receive reports of network issues affecting many services including CAS, VPN, and connectivity to other applications hosted on campus. The network team is working to correct the issue as quickly as possible. All workloads hosted in our environment are up and running and should respond normally as soon as the issue is resolved.

The Service Desk are receiving calls of intermittent network issues.

IST staff is working quickly to identify the source of the problem and to restore services as quickly as possible.

INC:INC0660327
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1932896 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1932900 - Posted: 30 Apr 2018, 17:05:17 UTC - in response to Message 1932871.  

Yeah, the times of the Cricket graphs. They were a really helpful tool.
Now we're driving blindfolded. :-)

Help us Obi-Wan Korpela, you're our only hope.

Somebody, I can't remember who corrected me on the routers the project goes through. I was told it was the Earl Warren Data Center ones. If that is the case, here is the cricket graphs for them.

Earl Warren Data Center Spine 1

Earl Warren Data Center Spine 2

Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1932900 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51469
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1932902 - Posted: 30 Apr 2018, 17:22:25 UTC

Active
Unscheduled Outage Intermittent Network Issues
Posted by CSS IT ~ mp
On 4/30/2018 at 7:14 am PST
Modified on 4/30/2018 at 9:19 am PST
Modified by CSS IT ~ mp
Posted in Unscheduled Outage

Outage Type: UNSCHEDULED OUTAGE (AMENDED)
Date Submitted: Monday, April 30, 2018
Outage Start/End Time: 0700 – TBD
Groups Impacted: Campus
Equipment: Campus Network

Description: UPDATE: Monday, April 30, 2018 10:17am – IST staff is aware of instability to the Palto Alto Firewall in the Earl Warrent Data Center and are troubleshooting to determine the cause and work toward a resolution.

Monday, April 30, 2018 9:17am – The Service Desk continues to receive reports of network issues affecting many services including CAS, VPN, and connectivity to other applications hosted on campus. The network team is working to correct the issue as quickly as possible. All workloads hosted in our environment are up and running and should respond normally as soon as the issue is resolved.

The Service Desk are receiving calls of intermittent network issues.

IST staff is working quickly to identify the source of the problem and to restore services as quickly as possible.

INC:INC0660327
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1932902 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 1932904 - Posted: 30 Apr 2018, 17:32:37 UTC

http://setiathome.berkeley.edu/forum_thread.php?id=82878

Posted by Eric in the news thread few min ago.
ID: 1932904 · Report as offensive
Profile Stargate (SA)
Volunteer tester
Avatar

Send message
Joined: 4 Mar 10
Posts: 1854
Credit: 2,258,721
RAC: 0
Australia
Message 1932911 - Posted: 30 Apr 2018, 19:02:24 UTC

Handy little website to let me know it was not on my end..

http://www.isitdownrightnow.com/setiathome.berkeley.edu.html
ID: 1932911 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1932915 - Posted: 30 Apr 2018, 19:18:21 UTC - in response to Message 1932902.  
Last modified: 30 Apr 2018, 19:21:01 UTC



Description: UPDATE: Monday, April 30, 2018 10:17am – IST staff is aware of instability to the Palto Alto Firewall in the Earl Warrent Data Center and are troubleshooting to determine the cause and work toward a resolution.

Guess my post of the routers should have used the packets graphs instead. Looks like packets In/Out have balanced on Spine 1. Spine 2 still having issues.


Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1932915 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1932954 - Posted: 1 May 2018, 6:34:01 UTC - in response to Message 1932836.  

Hmmm....seem to be having intermittent connection problems with the servers and forums again.
Anybody else noticing it this morning?

Meow?


. . It started here about 10:30 pm Monday AEST. But then about the same time my ISP went down so I blamed them. But as of 4pm Tuesday AEST the ISP is back up but I cannot upload anything at all. And very, very s..l..o..w accessing the forums.

Stephen

??
ID: 1932954 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13770
Credit: 208,696,464
RAC: 304
Australia
Message 1932956 - Posted: 1 May 2018, 7:55:16 UTC

Uploads are starting to go through, after a few tries, at about 2kB/s.
Grant
Darwin NT
ID: 1932956 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 1932957 - Posted: 1 May 2018, 7:59:22 UTC

Ah, we're back.

UPDATE: Monday, April 30, 2018 8:51pm – This evening at 7:06PM the data center firewalls reloaded on their own. The vendor is currently working to restore service.

They're done restoring, I suppose. On to the next downtime.
ID: 1932957 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13770
Credit: 208,696,464
RAC: 304
Australia
Message 1932958 - Posted: 1 May 2018, 8:02:08 UTC - in response to Message 1932957.  
Last modified: 1 May 2018, 8:08:58 UTC

They're done restoring, I suppose.

I hope not.
Things are still very spotty (although that just could be the Seti upload server falling over under the strain. It hasn't been doing too well lately).


Edit-
Managed to get some work, download speeds are OK, but uploads are still requiring multiple re-tries to get them to go through, at worse than dialup speeds.
Grant
Darwin NT
ID: 1932958 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13770
Credit: 208,696,464
RAC: 304
Australia
Message 1932980 - Posted: 1 May 2018, 10:06:28 UTC

Uploads seem to be going through OK now, however downloads are now reluctant to actually download.
Hopefully things will be better in the morning.
Grant
Darwin NT
ID: 1932980 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14656
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1932984 - Posted: 1 May 2018, 10:20:39 UTC - in response to Message 1932980.  

'Results per hour' has spiked up to 285K, and a lot of the ready-to-send queue - has been sent. I'd imagine we're now into download congestion territory, which is a better problem to have.
ID: 1932984 · Report as offensive
Profile Brent Norman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester

Send message
Joined: 1 Dec 99
Posts: 2786
Credit: 685,657,289
RAC: 835
Canada
Message 1932988 - Posted: 1 May 2018, 11:07:12 UTC - in response to Message 1932984.  

We will see now how the system can handle the big influx of tasks all coming in at once ....
ID: 1932988 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51469
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1932998 - Posted: 1 May 2018, 13:17:50 UTC

Description: UPDATE: Tuesday, May 1, 2018 1:59am – The firewall has been stable since 12:42am, services appear to be restored. The vendor will continue monitoring.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1932998 · Report as offensive
Previous · 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 . . . 33 · Next

Message boards : Number crunching : Panic Mode On (112) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.