Panic Mode On (80) Server Problems?


log in

Advanced search

Message boards : Number crunching : Panic Mode On (80) Server Problems?

Previous · 1 . . . 12 · 13 · 14 · 15 · 16 · 17 · 18 . . . 25 · Next
Author Message
Profile jrusling
Avatar
Send message
Joined: 8 Sep 02
Posts: 37
Credit: 4,764,889
RAC: 0
United States
Message 1327547 - Posted: 14 Jan 2013, 2:36:13 UTC - in response to Message 1327542.

It looks like it will take someone on the staff resetting something. The Cricket graphs have gone south.

Likely won't happen. In the morning, everything goes offline for two days. I guess it's a way of "ramping down" before the outage.


Also, I don't know if you knew about this, but some time back (year+), Boincstats changed some things around and you have someone else's stats in your signature. You should head back over to boincstats and get the updated code for that.

Thanks, I have images turned off and had not noticed.

Cosmic_Ocean
Avatar
Send message
Joined: 23 Dec 00
Posts: 2245
Credit: 8,555,906
RAC: 4,290
United States
Message 1327578 - Posted: 14 Jan 2013, 4:06:59 UTC

And from the way the crickets appear... it looks like an early shutdown.. but SSP froze some time ago.. so things broke just in time to be shut down for two days. I'm just going to go ahead and set both of my machines to "network activity suspended."
____________

Linux laptop uptime: 1484d 22h 42m
Ended due to UPS failure, found 14 hours after the fact

BarryAZ
Send message
Joined: 1 Apr 01
Posts: 2580
Credit: 12,013,414
RAC: 4,489
United States
Message 1327586 - Posted: 14 Jan 2013, 4:31:53 UTC - in response to Message 1327578.

With the air conditioning change over, plus other maintenance, plus dealing with what is likely to be a rather large backlog, plus trying to troubleshoot whatever problem is troubling at the moment, seems like we can expect that the collective dust won't settle until the end of the coming week.

Profile S@NL Etienne Dokkum
Volunteer tester
Avatar
Send message
Joined: 11 Jun 99
Posts: 159
Credit: 15,601,983
RAC: 7,317
Netherlands
Message 1327605 - Posted: 14 Jan 2013, 5:55:24 UTC

well, I guess this is it... Servers are dead already, by the time it's morning at Berkeley they'll pull the plug for AC maintenance.

But at least I got enough WU's for Seti Beta to crunch, so no complaints here.

See you all on the other side ! Bye
____________

zoom314Project donor
Avatar
Send message
Joined: 30 Nov 03
Posts: 46011
Credit: 36,544,964
RAC: 5,457
Message 1327609 - Posted: 14 Jan 2013, 6:02:30 UTC - in response to Message 1327605.

well, I guess this is it... Servers are dead already, by the time it's morning at Berkeley they'll pull the plug for AC maintenance.

But at least I got enough WU's for Seti Beta to crunch, so no complaints here.

See you all on the other side ! Bye

I've Einstein all ready to take over, so I'll see Y'all on the flip side of the LP...
____________
My Facebook, War Commander, 2015

Profile Wiggo
Avatar
Send message
Joined: 24 Jan 00
Posts: 6712
Credit: 92,349,252
RAC: 73,678
Australia
Message 1327620 - Posted: 14 Jan 2013, 7:03:13 UTC - in response to Message 1327609.

Backup projects took over all my GPU's yesterday, my 2500K this morning and my Q6600 by tonight but my old E6300 maybe lucky enough to make it through.

Cheers.
____________

Profile ivan
Volunteer tester
Avatar
Send message
Joined: 5 Mar 01
Posts: 597
Credit: 134,224,136
RAC: 116,415
United Kingdom
Message 1327713 - Posted: 15 Jan 2013, 23:00:55 UTC

Looking at the build-up of Results Ready to Send and the fact that scheduling and download servers seem to be imminently on-line, I feel a log-jam coming on...
____________

zoom314Project donor
Avatar
Send message
Joined: 30 Nov 03
Posts: 46011
Credit: 36,544,964
RAC: 5,457
Message 1327717 - Posted: 15 Jan 2013, 23:04:53 UTC

All I want to do is report 92 wu's, so that I can remove a damaged video card...
____________
My Facebook, War Commander, 2015

Profile MikeProject donor
Volunteer tester
Avatar
Send message
Joined: 17 Feb 01
Posts: 23688
Credit: 32,400,485
RAC: 24,134
Germany
Message 1327724 - Posted: 15 Jan 2013, 23:25:14 UTC

I like this one.

16.01.2013 00:23:11 SETI@home Beta Test [sched_op] Found 0 scheduler URLs in master file
16.01.2013 00:23:11 SETI@home Beta Test [sched_op] Deferring communication for 1 days 0 hr 0 min 0 sec
16.01.2013 00:23:11 SETI@home Beta Test [sched_op] Reason: 7 consecutive failures fetching scheduler list

____________

mikeej42
Send message
Joined: 26 Oct 00
Posts: 109
Credit: 788,999,260
RAC: 54,101
United States
Message 1327725 - Posted: 15 Jan 2013, 23:28:58 UTC

[As of 15 Jan 2013, 23:20:24 UTC] Ils sont parti

The scduler and download servers are green again....
____________

Profile ivan
Volunteer tester
Avatar
Send message
Joined: 5 Mar 01
Posts: 597
Credit: 134,224,136
RAC: 116,415
United Kingdom
Message 1327729 - Posted: 15 Jan 2013, 23:44:34 UTC

24 ghosts already on the only machine that's managed to make contact (but no ACK, of course...).
____________

Cosmic_Ocean
Avatar
Send message
Joined: 23 Dec 00
Posts: 2245
Credit: 8,555,906
RAC: 4,290
United States
Message 1327754 - Posted: 16 Jan 2013, 1:39:36 UTC

Ow. My downloads hurt so much. 15 APs and counting. Longest-running one is at 43 minutes and is at 1.19%. Going to be a long night.
____________

Linux laptop uptime: 1484d 22h 42m
Ended due to UPS failure, found 14 hours after the fact

zoom314Project donor
Avatar
Send message
Joined: 30 Nov 03
Posts: 46011
Credit: 36,544,964
RAC: 5,457
Message 1327763 - Posted: 16 Jan 2013, 1:59:24 UTC

The 92 wu's reported here, switched to cpu only.
____________
My Facebook, War Commander, 2015

Lee Gresham
Avatar
Send message
Joined: 12 Aug 03
Posts: 131
Credit: 99,547,502
RAC: 66,268
United States
Message 1327803 - Posted: 16 Jan 2013, 4:59:41 UTC

I just wish they'd left astropulse down until 1 mil+ rest of us got some work!
____________
Delta-V

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5774
Credit: 57,575,235
RAC: 48,417
Australia
Message 1327848 - Posted: 16 Jan 2013, 7:02:01 UTC - in response to Message 1327803.
Last modified: 16 Jan 2013, 7:35:31 UTC

This has got to be the worst traffic jam in a long time. Even after the previous outages (expected & not) downloads would generally complete, even if it took a while. After this outage it's taking lots of retries & Network activity suspended then Network activity available again just to get things to download.
They start off ok at 5-10kB/s, then just slow down after 5-10 seconds & stall. Add to that lots of Scheduler timeouts & HTTP service not available & i think it's going to take a fair while for things to settle down.



EDIT- the downloads are no longer stalling, now it's taking several minutes for data to start downloading after the download technically started. Then after 5-15 min it will then time out after sporradically downloading.
____________
Grant
Darwin NT.

clive G1FYE
Volunteer moderator
Send message
Joined: 4 Nov 04
Posts: 1300
Credit: 23,054,144
RAC: 0
United Kingdom
Message 1327877 - Posted: 16 Jan 2013, 10:01:19 UTC

I have my max of 100 tasks,
Stuck in download limbo, Grrrrrr.
and 54 of them are partly downloaded and impossible to move,
I wonder if all these partial downloads that we all have help jam the system up in any way ?

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5774
Credit: 57,575,235
RAC: 48,417
Australia
Message 1327881 - Posted: 16 Jan 2013, 10:19:50 UTC - in response to Message 1327877.

and 54 of them are partly downloaded and impossible to move,

Network activity suspended, Network activity available. Give it 5-10 min. Then repeat. Over & over again. And again.
____________
Grant
Darwin NT.

Profile Bernie Vine
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 26 May 99
Posts: 6884
Credit: 25,351,825
RAC: 33,514
United Kingdom
Message 1327882 - Posted: 16 Jan 2013, 10:27:12 UTC
Last modified: 16 Jan 2013, 11:10:05 UTC

Across my 6 crunchers (1 main and 5 Beta) I have 300+ stuck d/l's 130 of them partly downloaded!!

PS Now 430+ d/l and 170+ partials!!
____________


Today is life, the only life we're sure of. Make the most of today.

ExchangeMan
Volunteer tester
Send message
Joined: 9 Jan 00
Posts: 108
Credit: 129,775,472
RAC: 186,327
United States
Message 1327913 - Posted: 16 Jan 2013, 13:12:57 UTC

I'm wondering if the network is using some type of traffic shaping. I've noticed that usually when downloads start, there is a burst of download speed then after 10-15 seconds it comes down to a crawl and may stop entirely. I would really like to know why download traffic is so ragged in response. I understand the network is very congested with excessive demand, but downloads coming to a complete stop and then timing out are very frustrating - even a download of 1k/second would let you know that the network is still alive.


____________

Profile CLYDEProject donor
Volunteer tester
Avatar
Send message
Joined: 9 Aug 99
Posts: 1769
Credit: 21,951,888
RAC: 36,764
United States
Message 1327923 - Posted: 16 Jan 2013, 13:51:04 UTC
Last modified: 16 Jan 2013, 13:52:59 UTC

Hey guys/gals - CALM DOWN!!!

This Project is about the Search for Extraterrestrial Intelligence (SETI). NOT Credit Totals nor 'Bragging Rights'.

How about letting the servers & network sort themselves out.

Repeatedly hitting the Update Button probably makes the situation worse! (I should probably follow my own advice)
____________

Previous · 1 . . . 12 · 13 · 14 · 15 · 16 · 17 · 18 . . . 25 · Next

Message boards : Number crunching : Panic Mode On (80) Server Problems?

Copyright © 2014 University of California