Panic Mode On (82) Server Problems?

Message boards : Number crunching : Panic Mode On (82) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 17 · 18 · 19 · 20 · 21 · 22 · 23 . . . 24 · Next

AuthorMessage
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1354506 - Posted: 7 Apr 2013, 22:50:31 UTC - in response to Message 1354504.  
Last modified: 7 Apr 2013, 22:52:33 UTC

Well, one new dataset was loaded and is now splitting AP.
The Crickets responded by jumping up to around 140Mb/s.

Not hard to see why the servers got so tangled up so fast when AP hit the pipe with a 100Mb cap.

I want to see a Shortie Storm for comparison.

Claggy

LOL...
I was thinking exactly the same thing.

The recovery time from this coming Tuesday's outage should be interesting to see as well.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1354506 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1355190 - Posted: 10 Apr 2013, 5:35:06 UTC - in response to Message 1354506.  


Just had a look at the graphs & numbers.
No AP work going out at the momnent (other than resends etc), and there's a whole bunch of shorties coming through at the moment.
Network traffic is sitting at around 130Mb/s, just for MB alone...
Grant
Darwin NT
ID: 1355190 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22204
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1355193 - Posted: 10 Apr 2013, 5:39:42 UTC

.....and we wondered why there was so much network congestion with a 100Mb pipe....
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1355193 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1355196 - Posted: 10 Apr 2013, 5:49:47 UTC

LOL...
Ya ever see a 300Mb spike of uploads before???
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1355196 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1355213 - Posted: 10 Apr 2013, 6:16:47 UTC - in response to Message 1355196.  

LOL...
Ya ever see a 300Mb spike of uploads before???

Yeah, i saw those spikes & i'm not sure exactly what happened there.
I had a look at my logs, and as per usual uploads were still going through as the WUs completed all through the outage.
Yet there was a small (100Mb/s) surge about 1/3rd of the way through the outage, then that massive surge several hours after the outage, when the initial surge of downloads had completed- ie all uploads had already gone through & people were able to get new work.
Grant
Darwin NT
ID: 1355213 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1355240 - Posted: 10 Apr 2013, 7:56:28 UTC
Last modified: 10 Apr 2013, 8:06:13 UTC

Whoa boy.....

Here's your shorty storm, Claggy.

Crickets are a jumpin, and not an AP being split.

280Mb/s, and 111k results/hour coming back in.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1355240 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1355306 - Posted: 10 Apr 2013, 13:57:30 UTC - in response to Message 1355240.  

Whoa boy.....

Here's your shorty storm, Claggy.

Crickets are a jumpin, and not an AP being split.

280Mb/s, and 111k results/hour coming back in.

It only lasted a short while. If that is all it looks like now I'm not sure we can call it a "shortie storm". Maybe a "shortie cloud burst" or something.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1355306 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1355354 - Posted: 10 Apr 2013, 16:49:39 UTC
Last modified: 10 Apr 2013, 16:53:00 UTC

Hmmm....
Wonder what in the world is causing the huge upload or inbound spikes.
Just had another one last hour.

Could that be data being transferred from the lab to the servers at the colo?
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1355354 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 1355370 - Posted: 10 Apr 2013, 17:06:12 UTC - in response to Message 1355354.  

Hmmm....
Wonder what in the world is causing the huge upload or inbound spikes.
Just had another one last hour.

Could that be data being transferred from the lab to the servers at the colo?

Well, the raw data for splitting has got to get there somehow - and I don't see Matt taking it down in his lunchbox every other day.
ID: 1355370 · Report as offensive
Gone

Send message
Joined: 31 May 99
Posts: 150
Credit: 125,779,206
RAC: 0
United Kingdom
Message 1355837 - Posted: 11 Apr 2013, 19:51:42 UTC
Last modified: 11 Apr 2013, 19:52:20 UTC

360 Mb/s !

At least the spikes won't go over 9000.
ID: 1355837 · Report as offensive
ExchangeMan
Volunteer tester

Send message
Joined: 9 Jan 00
Posts: 115
Credit: 157,719,104
RAC: 0
United States
Message 1355846 - Posted: 11 Apr 2013, 20:35:16 UTC

I was wondering about those huge inbound spikes also. No way could that be returning work unit results.

ID: 1355846 · Report as offensive
Profile HAL9000
Volunteer tester
Avatar

Send message
Joined: 11 Sep 99
Posts: 6534
Credit: 196,805,888
RAC: 57
United States
Message 1355849 - Posted: 11 Apr 2013, 20:45:05 UTC - in response to Message 1355837.  

360 Mb/s !

At least the spikes won't go over 9000.

Wow the inbound has hit Max: 446.93 Mbits/sec. Good to see that they can make more use of the available inbound bandwidth.
SETI@home classic workunits: 93,865 CPU time: 863,447 hours
Join the [url=http://tinyurl.com/8y46zvu]BP6/VP6 User Group[
ID: 1355849 · Report as offensive
Gone

Send message
Joined: 31 May 99
Posts: 150
Credit: 125,779,206
RAC: 0
United Kingdom
Message 1355850 - Posted: 11 Apr 2013, 20:54:00 UTC

Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
ID: 1355850 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1355896 - Posted: 11 Apr 2013, 23:16:53 UTC

Inbound is probably something like bringing archived tapes in from off-site storage.

Also.. something is abnormal.. it is taking 17 seconds to download APs instead of 9-11. Maybe the extra packet load for the mystery inbound traffic is making transfers take slightly longer.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1355896 · Report as offensive
Starman
Avatar

Send message
Joined: 15 May 99
Posts: 204
Credit: 81,351,915
RAC: 25
Canada
Message 1355972 - Posted: 12 Apr 2013, 4:08:47 UTC

Just checked and I'm downloading Astopulse v6 6.04 in 4 seconds, that's from the time is shows start to time says stop: So that's pretty impressive.

11/04/2013 9:00:26 PM | SETI@home | Started download of ap_24fe13ab_B0_P0_00182_20130411_25067.wu
11/04/2013 9:00:30 PM | SETI@home | Finished download of ap_24fe13ab_B0_P0_00182_20130411_25067.wu

ID: 1355972 · Report as offensive
Starman
Avatar

Send message
Joined: 15 May 99
Posts: 204
Credit: 81,351,915
RAC: 25
Canada
Message 1355974 - Posted: 12 Apr 2013, 4:16:25 UTC

Of course this helps.


ID: 1355974 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1356001 - Posted: 12 Apr 2013, 6:31:51 UTC
Last modified: 12 Apr 2013, 6:33:40 UTC

Yeah, something is definitely odd..

2013-04-12 01:09:41 SETI@home Started download of ap_23fe13ab_B4_P0_00180_20130411_13542.wu
2013-04-12 01:10:23 SETI@home Finished download of ap_23fe13ab_B4_P0_00180_20130411_13542.wu
2013-04-12 01:19:54 SETI@home Started download of ap_14fe13ab_B5_P1_00050_20130411_23089.wu
2013-04-12 01:21:00 SETI@home Finished download of ap_14fe13ab_B5_P1_00050_20130411_23089.wu


They're slowing down a bit. I downloaded the new nvidia drivers at 4.2MB/sec, so... I know it's not my connection, as noted here:


Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1356001 · Report as offensive
Thomas
Volunteer tester

Send message
Joined: 9 Dec 11
Posts: 1499
Credit: 1,345,576
RAC: 0
France
Message 1356016 - Posted: 12 Apr 2013, 9:27:17 UTC - in response to Message 1356015.  

Yep ! Mistake... We lost all the University of Berkeley, Chris... Very strange... (during two hours 07:15-09:15 UTC).
ID: 1356016 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 1356017 - Posted: 12 Apr 2013, 9:29:09 UTC - in response to Message 1356015.  
Last modified: 12 Apr 2013, 9:39:36 UTC

We lost you for 3 hours - what happened?


Yeah, everything on berkeley.edu went down.
All my uploads have gone through, but i'm not getting any response from the Scheduler at the moment, other than
12/04/2013 18:54:04 SETI@home Scheduler request failed: HTTP internal server error


EDIT- a couple more attempts & work was delivered in full.

Antoher EDIT- and a new download record set for the Cricket graphs, 295Mb/s.
Grant
Darwin NT
ID: 1356017 · Report as offensive
Sakletare
Avatar

Send message
Joined: 18 May 99
Posts: 132
Credit: 23,423,829
RAC: 0
Sweden
Message 1356021 - Posted: 12 Apr 2013, 9:43:53 UTC - in response to Message 1356017.  

Yeah, everything on berkeley.edu went down.

Yes the whole university dropped off the internet. I checked several is-it-up sites and they all reported it as down.
ID: 1356021 · Report as offensive
Previous · 1 . . . 17 · 18 · 19 · 20 · 21 · 22 · 23 . . . 24 · Next

Message boards : Number crunching : Panic Mode On (82) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.