Panic Mode On (79) Server Problems?


log in

Advanced search

Message boards : Number crunching : Panic Mode On (79) Server Problems?

Previous · 1 . . . 18 · 19 · 20 · 21 · 22 · 23 · Next
Author Message
TBar
Volunteer tester
Send message
Joined: 22 May 99
Posts: 1383
Credit: 51,018,000
RAC: 105,060
United States
Message 1321052 - Posted: 28 Dec 2012, 23:32:12 UTC

Ye-Haw a Shorty Storm with a Borked Upload server. How many do you think it will hang?

12/28/2012 6:13:56 PM | SETI@home | Started upload of 03se12aa.10570.17249.140733193388039.10.238_2_0 12/28/2012 6:14:19 PM | | Project communication failed: attempting access to reference site 12/28/2012 6:14:19 PM | SETI@home | Temporarily failed upload of 03se12aa.10570.17249.140733193388039.10.238_2_0: connect() failed 12/28/2012 6:14:19 PM | SETI@home | Backing off 3 min 50 sec on upload of 03se12aa.10570.17249.140733193388039.10.238_2_0 12/28/2012 6:14:21 PM | | Internet access OK - project servers may be temporarily down. 12/28/2012 6:17:59 PM | SETI@home | Computation for task 07oc12af.16388.6202.140733193388038.10.1_1 finished 12/28/2012 6:17:59 PM | SETI@home | Starting task 07oc12ag.10905.12746.9.10.247_1 using setiathome_enhanced version 609 (cuda23) in slot 3 12/28/2012 6:18:01 PM | SETI@home | Started upload of 07oc12af.16388.6202.140733193388038.10.1_1_0 12/28/2012 6:18:10 PM | SETI@home | Started upload of 03se12aa.10570.17249.140733193388039.10.238_2_0 12/28/2012 6:18:23 PM | | Project communication failed: attempting access to reference site 12/28/2012 6:18:23 PM | SETI@home | Temporarily failed upload of 07oc12af.16388.6202.140733193388038.10.1_1_0: connect() failed 12/28/2012 6:18:23 PM | SETI@home | Backing off 3 min 34 sec on upload of 07oc12af.16388.6202.140733193388038.10.1_1_0 12/28/2012 6:18:25 PM | | Internet access OK - project servers may be temporarily down. 12/28/2012 6:21:45 PM | | Project communication failed: attempting access to reference site 12/28/2012 6:21:45 PM | SETI@home | Temporarily failed upload of 03se12aa.10570.17249.140733193388039.10.238_2_0: connect() failed 12/28/2012 6:21:45 PM | SETI@home | Backing off 4 min 49 sec on upload of 03se12aa.10570.17249.140733193388039.10.238_2_0 12/28/2012 6:21:47 PM | | Internet access OK - project servers may be temporarily down. 12/28/2012 6:22:09 PM | SETI@home | Computation for task 07oc12ag.10905.12746.9.10.247_1 finished 12/28/2012 6:22:09 PM | SETI@home | Starting task 07oc12ah.10878.22562.6.10.76_1 using setiathome_enhanced version 609 (cuda23) in slot 3 12/28/2012 6:22:11 PM | SETI@home | Started upload of 07oc12ag.10905.12746.9.10.247_1_0 12/28/2012 6:22:49 PM | | Project communication failed: attempting access to reference site 12/28/2012 6:22:49 PM | SETI@home | Temporarily failed upload of 07oc12ag.10905.12746.9.10.247_1_0: connect() failed 12/28/2012 6:22:49 PM | SETI@home | Backing off 3 min 11 sec on upload of 07oc12ag.10905.12746.9.10.247_1_0 12/28/2012 6:22:50 PM | | Internet access OK - project servers may be temporarily down. 12/28/2012 6:26:11 PM | SETI@home | Computation for task 07oc12ah.10878.22562.6.10.76_1 finished 12/28/2012 6:26:11 PM | SETI@home | Starting task 01au12ab.23909.24895.10.10.9_2 using setiathome_enhanced version 609 (cuda23) in slot 3 12/28/2012 6:26:13 PM | SETI@home | Started upload of 07oc12ah.10878.22562.6.10.76_1_0 12/28/2012 6:26:26 PM | SETI@home | Computation for task 01au12ab.23909.24895.10.10.9_2 finished 12/28/2012 6:26:26 PM | SETI@home | Starting task 07oc12af.5577.9065.140733193388039.10.69_0 using setiathome_enhanced version 609 (cuda23) in slot 3 12/28/2012 6:26:28 PM | SETI@home | Started upload of 01au12ab.23909.24895.10.10.9_2_0 12/28/2012 6:26:35 PM | | Project communication failed: attempting access to reference site 12/28/2012 6:26:35 PM | SETI@home | Temporarily failed upload of 07oc12ah.10878.22562.6.10.76_1_0: connect() failed 12/28/2012 6:26:35 PM | SETI@home | Backing off 3 min 46 sec on upload of 07oc12ah.10878.22562.6.10.76_1_0 12/28/2012 6:26:36 PM | | Internet access OK - project servers may be temporarily down. 12/28/2012 6:26:50 PM | | Project communication failed: attempting access to reference site 12/28/2012 6:26:50 PM | SETI@home | Temporarily failed upload of 01au12ab.23909.24895.10.10.9_2_0: connect() failed 12/28/2012 6:26:50 PM | SETI@home | Backing off 3 min 19 sec on upload of 01au12ab.23909.24895.10.10.9_2_0 12/28/2012 6:26:52 PM | | Internet access OK - project servers may be temporarily down.


:-(

WinterKnight
Volunteer tester
Send message
Joined: 18 May 99
Posts: 8686
Credit: 25,031,773
RAC: 30,086
United Kingdom
Message 1321054 - Posted: 28 Dec 2012, 23:38:51 UTC - in response to Message 1321047.

It took a few requests but eventually got a few GPU tasks and they all came in at >50kbs.

10-20kB/s here at the moment.

Now with uploads 1kB/s is doing well (when it does eventually go through).

Thats cause you live in the middle of nowhere, or at least can see nowhere from there.

My cousin keeps me informed of the actions of your ISP's and the telco's over there he is not very impressed having lived in the UK, Boston and southern California.

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5868
Credit: 60,598,316
RAC: 47,541
Australia
Message 1321065 - Posted: 29 Dec 2012, 0:49:01 UTC - in response to Message 1321054.

My cousin keeps me informed of the actions of your ISP's and the telco's over there he is not very impressed having lived in the UK, Boston and southern California.

We're not overly thrilled with them either, but they do have the population density & total numbers argument on their side.
A landmass the area of mainland USA, with a total population that's not even 3 times that of London (22.6 million v 8.2 million).
____________
Grant
Darwin NT.

dsh
Send message
Joined: 6 Jan 08
Posts: 116
Credit: 9,914,086
RAC: 11,159
United States
Message 1321082 - Posted: 29 Dec 2012, 1:37:44 UTC - in response to Message 1321080.

I have not seen the Seti servers so totally screwed in a long while.
I mean, we have a trifecta going here.
Uploads, downloads, AND scheduler requests all nearly impossible.

Makes me wonder if we have a dang DOS attack on the servers going again.


Perhaps just busy. I had an AP upload and two AP downloads finished within the last 10 minutes.

Profile Zapped "Sixth Sense" Sparky
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 30 Aug 08
Posts: 8385
Credit: 1,296,619
RAC: 908
United Kingdom
Message 1321087 - Posted: 29 Dec 2012, 1:56:23 UTC - in response to Message 1321080.

I have not seen the Seti servers so totally screwed in a long while.
I mean, we have a trifecta going here.
Uploads, downloads, AND scheduler requests all nearly impossible.

Makes me wonder if we have a dang DOS attack on the servers going again.

I dunno, my cache was wiped out due to my lappy having a fit last night resulting in about 15hrs of no crunching. It took a couple of hours before Boinc received the message that the tasks had been reported and since about 18:55:08 UTC to 22:34:42 UTC my cache has been rebuilt.

A lot of timeouts, server returned nothing and so on though :( I'm glad I'm not a mega cruncher, I'd probably be doing a lot of swearing right about now :)
____________
In an alternate universe, it was a ZX81 that asked for clothes, boots and motorcycle.

Client error 418: I'm a teapot

Tropical Goldfish Fish 15: Squeaky bras 'R us

Illusions of normality sufferer

Profile Zapped "Sixth Sense" Sparky
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 30 Aug 08
Posts: 8385
Credit: 1,296,619
RAC: 908
United Kingdom
Message 1321096 - Posted: 29 Dec 2012, 2:39:21 UTC - in response to Message 1321088.

I have not seen the Seti servers so totally screwed in a long while.
I mean, we have a trifecta going here.
Uploads, downloads, AND scheduler requests all nearly impossible.

Makes me wonder if we have a dang DOS attack on the servers going again.

I dunno, my cache was wiped out due to my lappy having a fit last night resulting in about 15hrs of no crunching. It took a couple of hours before Boinc received the message that the tasks had been reported and since about 18:55:08 UTC to 22:34:42 UTC my cache has been rebuilt.


And your cache consists of 16 WUs.............

All astro's, the quicker I get 'em the quicker the pipe clears up :)
____________
In an alternate universe, it was a ZX81 that asked for clothes, boots and motorcycle.

Client error 418: I'm a teapot

Tropical Goldfish Fish 15: Squeaky bras 'R us

Illusions of normality sufferer

zoom314Project donor
Avatar
Send message
Joined: 30 Nov 03
Posts: 46519
Credit: 36,861,886
RAC: 4,974
United States
Message 1321103 - Posted: 29 Dec 2012, 3:09:44 UTC

Upload timeout has been here...
____________
My Facebook, War Commander, 2015

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5868
Credit: 60,598,316
RAC: 47,541
Australia
Message 1321117 - Posted: 29 Dec 2012, 4:19:22 UTC - in response to Message 1321103.
Last modified: 29 Dec 2012, 4:36:45 UTC

Scheduler request errors are now occuring within a couple of seconds.
If that doesn't happen then a timeout is more likely than a response. Add to that the shorties & the continuing upload issues.


EDIT- and these shorties are *really* short.
Running 3 at a time on my GTX 560Ti they're being done in just over 4 minutes.
____________
Grant
Darwin NT.

zoom314Project donor
Avatar
Send message
Joined: 30 Nov 03
Posts: 46519
Credit: 36,861,886
RAC: 4,974
United States
Message 1321122 - Posted: 29 Dec 2012, 4:31:53 UTC - in response to Message 1321119.


Scheduler request errors are now occuring within a couple of seconds.
If that doesn't happen then a timeout is more likely than a response. Add to that the shorties & the continuing upload issues.

I have been seeing my caches, such as were stingily allowed recently, diminish greatly.
As is the temperature in the crunching den of the kitties.

When I walked in after work this afternoon and saw the room temperature, I knew something was wrong/. It had already gone to crap hours before, while I was at work.
Multi thousand watts of Seti crunching power not being dissipated by hard working GPUs into the cold winter air.

Meowsigh.

I had to resort to Einstein and even though My card should be able to handle 6 wu's at a time, problem is, the cpu is stuck at 4, sigh, maybe next month I can fix that... Nothing like 2 months of saving...
____________
My Facebook, War Commander, 2015

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5868
Credit: 60,598,316
RAC: 47,541
Australia
Message 1321131 - Posted: 29 Dec 2012, 5:03:24 UTC - in response to Message 1321122.
Last modified: 29 Dec 2012, 5:04:45 UTC

Just ran out of work on one GPU.
Even if i can upload the results, i get nothing but Scheduler errors or timeouts. By then, i can't get new work because of all the uploads that have backed up again. When i finally get them all uploaded, i get a Scheduler error or timeout again.
Etc, etc, etc,
____________
Grant
Darwin NT.

Profile Wiggo
Avatar
Send message
Joined: 24 Jan 00
Posts: 7358
Credit: 96,903,496
RAC: 66,636
Australia
Message 1321148 - Posted: 29 Dec 2012, 5:58:04 UTC - in response to Message 1321131.

Just ran out of work on one GPU.
Even if i can upload the results, i get nothing but Scheduler errors or timeouts. By then, i can't get new work because of all the uploads that have backed up again. When i finally get them all uploaded, i get a Scheduler error or timeout again.
Etc, etc, etc,

Having 2 or 3 good backup projects at times like these helps (I must see about getting 2 of them added to our Team's participating projects list). ;)

Cheers.
____________

Profile Wiggo
Avatar
Send message
Joined: 24 Jan 00
Posts: 7358
Credit: 96,903,496
RAC: 66,636
Australia
Message 1321150 - Posted: 29 Dec 2012, 6:13:03 UTC - in response to Message 1321148.

Seems as if something has received a fix as my uploads are suddenly start to clear. :)

Cheers.
____________

TBar
Volunteer tester
Send message
Joined: 22 May 99
Posts: 1383
Credit: 51,018,000
RAC: 105,060
United States
Message 1321151 - Posted: 29 Dec 2012, 6:16:04 UTC

It may be about over. My Uploads aren't hanging anymore, most of the 4 minute MB Shorties have been replaced by newly downloaded 20 min Longies, and the Update is working more than not. Maybe...

Keith White
Avatar
Send message
Joined: 29 May 99
Posts: 370
Credit: 2,896,743
RAC: 2,450
United States
Message 1321153 - Posted: 29 Dec 2012, 6:21:51 UTC - in response to Message 1321133.

Uploads are still working but it takes a while and can be very temperamental. However I can't get a scheduler to not timeout and without that I can't see if downloads are equally temperamental.

Something is stopping up the pipes. Cricket now shows a nasty spike of incoming packets with a similar spike on the outgoing packets. Incoming packets now around 1/3 higher than is was yesterday and outgoing packets at nearly 10 kpkt/s.

Looks as if the chart started to go off "normal" about 24 hours ago. I imagine we are seeing some kind of cascading failure condition, clogging the pipes with retries that then trigger more hosts to fail connecting which then triggers more retires that clogs the pipes even more, etc.

This is what I'm seeing in my log

12/29/2012 1:09:16 AM | SETI@home | Reporting 6 completed tasks, requesting new tasks for CPU and ATI 12/29/2012 1:09:18 AM | | Project communication failed: attempting access to reference site 12/29/2012 1:09:18 AM | SETI@home | Scheduler request failed: Server returned nothing (no headers, no data) 12/29/2012 1:09:20 AM | | Internet access OK - project servers may be temporarily down.


____________
"Life is just nature's way of keeping meat fresh." - The Doctor

Previous · 1 . . . 18 · 19 · 20 · 21 · 22 · 23 · Next

Message boards : Number crunching : Panic Mode On (79) Server Problems?

Copyright © 2014 University of California