Panic Mode On (79) Server Problems?


log in

Advanced search

Message boards : Number crunching : Panic Mode On (79) Server Problems?

Previous · 1 . . . 18 · 19 · 20 · 21 · 22 · 23 · Next
Author Message
TBar
Volunteer tester
Send message
Joined: 22 May 99
Posts: 1177
Credit: 41,442,504
RAC: 111,436
United States
Message 1321052 - Posted: 28 Dec 2012, 23:32:12 UTC

Ye-Haw a Shorty Storm with a Borked Upload server. How many do you think it will hang?

12/28/2012 6:13:56 PM | SETI@home | Started upload of 03se12aa.10570.17249.140733193388039.10.238_2_0
12/28/2012 6:14:19 PM | | Project communication failed: attempting access to reference site
12/28/2012 6:14:19 PM | SETI@home | Temporarily failed upload of 03se12aa.10570.17249.140733193388039.10.238_2_0: connect() failed
12/28/2012 6:14:19 PM | SETI@home | Backing off 3 min 50 sec on upload of 03se12aa.10570.17249.140733193388039.10.238_2_0
12/28/2012 6:14:21 PM | | Internet access OK - project servers may be temporarily down.
12/28/2012 6:17:59 PM | SETI@home | Computation for task 07oc12af.16388.6202.140733193388038.10.1_1 finished
12/28/2012 6:17:59 PM | SETI@home | Starting task 07oc12ag.10905.12746.9.10.247_1 using setiathome_enhanced version 609 (cuda23) in slot 3
12/28/2012 6:18:01 PM | SETI@home | Started upload of 07oc12af.16388.6202.140733193388038.10.1_1_0
12/28/2012 6:18:10 PM | SETI@home | Started upload of 03se12aa.10570.17249.140733193388039.10.238_2_0
12/28/2012 6:18:23 PM | | Project communication failed: attempting access to reference site
12/28/2012 6:18:23 PM | SETI@home | Temporarily failed upload of 07oc12af.16388.6202.140733193388038.10.1_1_0: connect() failed
12/28/2012 6:18:23 PM | SETI@home | Backing off 3 min 34 sec on upload of 07oc12af.16388.6202.140733193388038.10.1_1_0
12/28/2012 6:18:25 PM | | Internet access OK - project servers may be temporarily down.
12/28/2012 6:21:45 PM | | Project communication failed: attempting access to reference site
12/28/2012 6:21:45 PM | SETI@home | Temporarily failed upload of 03se12aa.10570.17249.140733193388039.10.238_2_0: connect() failed
12/28/2012 6:21:45 PM | SETI@home | Backing off 4 min 49 sec on upload of 03se12aa.10570.17249.140733193388039.10.238_2_0
12/28/2012 6:21:47 PM | | Internet access OK - project servers may be temporarily down.
12/28/2012 6:22:09 PM | SETI@home | Computation for task 07oc12ag.10905.12746.9.10.247_1 finished
12/28/2012 6:22:09 PM | SETI@home | Starting task 07oc12ah.10878.22562.6.10.76_1 using setiathome_enhanced version 609 (cuda23) in slot 3
12/28/2012 6:22:11 PM | SETI@home | Started upload of 07oc12ag.10905.12746.9.10.247_1_0
12/28/2012 6:22:49 PM | | Project communication failed: attempting access to reference site
12/28/2012 6:22:49 PM | SETI@home | Temporarily failed upload of 07oc12ag.10905.12746.9.10.247_1_0: connect() failed
12/28/2012 6:22:49 PM | SETI@home | Backing off 3 min 11 sec on upload of 07oc12ag.10905.12746.9.10.247_1_0
12/28/2012 6:22:50 PM | | Internet access OK - project servers may be temporarily down.
12/28/2012 6:26:11 PM | SETI@home | Computation for task 07oc12ah.10878.22562.6.10.76_1 finished
12/28/2012 6:26:11 PM | SETI@home | Starting task 01au12ab.23909.24895.10.10.9_2 using setiathome_enhanced version 609 (cuda23) in slot 3
12/28/2012 6:26:13 PM | SETI@home | Started upload of 07oc12ah.10878.22562.6.10.76_1_0
12/28/2012 6:26:26 PM | SETI@home | Computation for task 01au12ab.23909.24895.10.10.9_2 finished
12/28/2012 6:26:26 PM | SETI@home | Starting task 07oc12af.5577.9065.140733193388039.10.69_0 using setiathome_enhanced version 609 (cuda23) in slot 3
12/28/2012 6:26:28 PM | SETI@home | Started upload of 01au12ab.23909.24895.10.10.9_2_0
12/28/2012 6:26:35 PM | | Project communication failed: attempting access to reference site
12/28/2012 6:26:35 PM | SETI@home | Temporarily failed upload of 07oc12ah.10878.22562.6.10.76_1_0: connect() failed
12/28/2012 6:26:35 PM | SETI@home | Backing off 3 min 46 sec on upload of 07oc12ah.10878.22562.6.10.76_1_0
12/28/2012 6:26:36 PM | | Internet access OK - project servers may be temporarily down.
12/28/2012 6:26:50 PM | | Project communication failed: attempting access to reference site
12/28/2012 6:26:50 PM | SETI@home | Temporarily failed upload of 01au12ab.23909.24895.10.10.9_2_0: connect() failed
12/28/2012 6:26:50 PM | SETI@home | Backing off 3 min 19 sec on upload of 01au12ab.23909.24895.10.10.9_2_0
12/28/2012 6:26:52 PM | | Internet access OK - project servers may be temporarily down.


:-(

WinterKnight
Volunteer tester
Send message
Joined: 18 May 99
Posts: 8503
Credit: 23,090,634
RAC: 15,745
United Kingdom
Message 1321054 - Posted: 28 Dec 2012, 23:38:51 UTC - in response to Message 1321047.

It took a few requests but eventually got a few GPU tasks and they all came in at >50kbs.

10-20kB/s here at the moment.

Now with uploads 1kB/s is doing well (when it does eventually go through).

Thats cause you live in the middle of nowhere, or at least can see nowhere from there.

My cousin keeps me informed of the actions of your ISP's and the telco's over there he is not very impressed having lived in the UK, Boston and southern California.

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5694
Credit: 56,330,976
RAC: 48,793
Australia
Message 1321065 - Posted: 29 Dec 2012, 0:49:01 UTC - in response to Message 1321054.

My cousin keeps me informed of the actions of your ISP's and the telco's over there he is not very impressed having lived in the UK, Boston and southern California.

We're not overly thrilled with them either, but they do have the population density & total numbers argument on their side.
A landmass the area of mainland USA, with a total population that's not even 3 times that of London (22.6 million v 8.2 million).
____________
Grant
Darwin NT.

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38320
Credit: 559,318,740
RAC: 639,576
United States
Message 1321080 - Posted: 29 Dec 2012, 1:28:04 UTC

I have not seen the Seti servers so totally screwed in a long while.
I mean, we have a trifecta going here.
Uploads, downloads, AND scheduler requests all nearly impossible.

Makes me wonder if we have a dang DOS attack on the servers going again.
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

dsh
Send message
Joined: 6 Jan 08
Posts: 100
Credit: 8,905,955
RAC: 13,403
United States
Message 1321082 - Posted: 29 Dec 2012, 1:37:44 UTC - in response to Message 1321080.

I have not seen the Seti servers so totally screwed in a long while.
I mean, we have a trifecta going here.
Uploads, downloads, AND scheduler requests all nearly impossible.

Makes me wonder if we have a dang DOS attack on the servers going again.


Perhaps just busy. I had an AP upload and two AP downloads finished within the last 10 minutes.

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38320
Credit: 559,318,740
RAC: 639,576
United States
Message 1321083 - Posted: 29 Dec 2012, 1:41:46 UTC - in response to Message 1321082.
Last modified: 29 Dec 2012, 1:43:15 UTC

I have not seen the Seti servers so totally screwed in a long while.
I mean, we have a trifecta going here.
Uploads, downloads, AND scheduler requests all nearly impossible.

Makes me wonder if we have a dang DOS attack on the servers going again.


Perhaps just busy. I had an AP upload and two AP downloads finished within the last 10 minutes.

Not in my experience.
Usually at least downloads remain pretty efficient even during peak loading after outages.
I feel something else is awry here.
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

Profile Zapped Sparky
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 30 Aug 08
Posts: 6646
Credit: 1,200,844
RAC: 77
United Kingdom
Message 1321087 - Posted: 29 Dec 2012, 1:56:23 UTC - in response to Message 1321080.

I have not seen the Seti servers so totally screwed in a long while.
I mean, we have a trifecta going here.
Uploads, downloads, AND scheduler requests all nearly impossible.

Makes me wonder if we have a dang DOS attack on the servers going again.

I dunno, my cache was wiped out due to my lappy having a fit last night resulting in about 15hrs of no crunching. It took a couple of hours before Boinc received the message that the tasks had been reported and since about 18:55:08 UTC to 22:34:42 UTC my cache has been rebuilt.

A lot of timeouts, server returned nothing and so on though :( I'm glad I'm not a mega cruncher, I'd probably be doing a lot of swearing right about now :)
____________
In an alternate universe, it was a ZX81 that asked for clothes, boots and motorcycle.

Client error 418: I'm a teapot

Tropical Goldfish Fish 13: You're not crazy if you crunch for Seti :)

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38320
Credit: 559,318,740
RAC: 639,576
United States
Message 1321088 - Posted: 29 Dec 2012, 1:58:54 UTC - in response to Message 1321087.
Last modified: 29 Dec 2012, 1:59:17 UTC

I have not seen the Seti servers so totally screwed in a long while.
I mean, we have a trifecta going here.
Uploads, downloads, AND scheduler requests all nearly impossible.

Makes me wonder if we have a dang DOS attack on the servers going again.

I dunno, my cache was wiped out due to my lappy having a fit last night resulting in about 15hrs of no crunching. It took a couple of hours before Boinc received the message that the tasks had been reported and since about 18:55:08 UTC to 22:34:42 UTC my cache has been rebuilt.


And your cache consists of 16 WUs.............
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

Profile Zapped Sparky
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 30 Aug 08
Posts: 6646
Credit: 1,200,844
RAC: 77
United Kingdom
Message 1321096 - Posted: 29 Dec 2012, 2:39:21 UTC - in response to Message 1321088.

I have not seen the Seti servers so totally screwed in a long while.
I mean, we have a trifecta going here.
Uploads, downloads, AND scheduler requests all nearly impossible.

Makes me wonder if we have a dang DOS attack on the servers going again.

I dunno, my cache was wiped out due to my lappy having a fit last night resulting in about 15hrs of no crunching. It took a couple of hours before Boinc received the message that the tasks had been reported and since about 18:55:08 UTC to 22:34:42 UTC my cache has been rebuilt.


And your cache consists of 16 WUs.............

All astro's, the quicker I get 'em the quicker the pipe clears up :)
____________
In an alternate universe, it was a ZX81 that asked for clothes, boots and motorcycle.

Client error 418: I'm a teapot

Tropical Goldfish Fish 13: You're not crazy if you crunch for Seti :)

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38320
Credit: 559,318,740
RAC: 639,576
United States
Message 1321098 - Posted: 29 Dec 2012, 2:41:35 UTC - in response to Message 1321096.

I have not seen the Seti servers so totally screwed in a long while.
I mean, we have a trifecta going here.
Uploads, downloads, AND scheduler requests all nearly impossible.

Makes me wonder if we have a dang DOS attack on the servers going again.

I dunno, my cache was wiped out due to my lappy having a fit last night resulting in about 15hrs of no crunching. It took a couple of hours before Boinc received the message that the tasks had been reported and since about 18:55:08 UTC to 22:34:42 UTC my cache has been rebuilt.


And your cache consists of 16 WUs.............

All astro's, the quicker I get 'em the quicker the pipe clears up :)

LOL...I think it'll take more than your li'l lappy to clean up that mess.
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

zoom314
Avatar
Send message
Joined: 30 Nov 03
Posts: 45786
Credit: 36,410,707
RAC: 7,403
Message 1321103 - Posted: 29 Dec 2012, 3:09:44 UTC

Upload timeout has been here...
____________

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5694
Credit: 56,330,976
RAC: 48,793
Australia
Message 1321117 - Posted: 29 Dec 2012, 4:19:22 UTC - in response to Message 1321103.
Last modified: 29 Dec 2012, 4:36:45 UTC

Scheduler request errors are now occuring within a couple of seconds.
If that doesn't happen then a timeout is more likely than a response. Add to that the shorties & the continuing upload issues.


EDIT- and these shorties are *really* short.
Running 3 at a time on my GTX 560Ti they're being done in just over 4 minutes.
____________
Grant
Darwin NT.

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38320
Credit: 559,318,740
RAC: 639,576
United States
Message 1321119 - Posted: 29 Dec 2012, 4:25:41 UTC - in response to Message 1321117.
Last modified: 29 Dec 2012, 4:26:43 UTC


Scheduler request errors are now occuring within a couple of seconds.
If that doesn't happen then a timeout is more likely than a response. Add to that the shorties & the continuing upload issues.

I have been seeing my caches, such as were stingily allowed recently, diminish greatly.
As is the temperature in the crunching den of the kitties.

When I walked in after work this afternoon and saw the room temperature, I knew something was wrong/. It had already gone to crap hours before, while I was at work.
Multi thousand watts of Seti crunching power not being dissipated by hard working GPUs into the cold winter air.

Meowsigh.
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

zoom314
Avatar
Send message
Joined: 30 Nov 03
Posts: 45786
Credit: 36,410,707
RAC: 7,403
Message 1321122 - Posted: 29 Dec 2012, 4:31:53 UTC - in response to Message 1321119.


Scheduler request errors are now occuring within a couple of seconds.
If that doesn't happen then a timeout is more likely than a response. Add to that the shorties & the continuing upload issues.

I have been seeing my caches, such as were stingily allowed recently, diminish greatly.
As is the temperature in the crunching den of the kitties.

When I walked in after work this afternoon and saw the room temperature, I knew something was wrong/. It had already gone to crap hours before, while I was at work.
Multi thousand watts of Seti crunching power not being dissipated by hard working GPUs into the cold winter air.

Meowsigh.

I had to resort to Einstein and even though My card should be able to handle 6 wu's at a time, problem is, the cpu is stuck at 4, sigh, maybe next month I can fix that... Nothing like 2 months of saving...
____________

Grant (SSSF)
Send message
Joined: 19 Aug 99
Posts: 5694
Credit: 56,330,976
RAC: 48,793
Australia
Message 1321131 - Posted: 29 Dec 2012, 5:03:24 UTC - in response to Message 1321122.
Last modified: 29 Dec 2012, 5:04:45 UTC

Just ran out of work on one GPU.
Even if i can upload the results, i get nothing but Scheduler errors or timeouts. By then, i can't get new work because of all the uploads that have backed up again. When i finally get them all uploaded, i get a Scheduler error or timeout again.
Etc, etc, etc,
____________
Grant
Darwin NT.

msattler
Volunteer tester
Avatar
Send message
Joined: 9 Jul 00
Posts: 38320
Credit: 559,318,740
RAC: 639,576
United States
Message 1321133 - Posted: 29 Dec 2012, 5:06:26 UTC - in response to Message 1321131.


Just ran out of work on one GPU.
Even if i can upload the results, i get nothing but Scheduler errors or timeouts. By then, i can't get new work because of all the uploads that have backed up again.

I am ditching soon for the night.......
Hope the kitties and myself don't freeze overnight due to the LACK OF FREAKING CRUNCHING>>>>>>>>>ARRGGGGGGHHHHHHHHHHHHHHHHHHHHHHHHHH.
____________
*********************************************
Embrace your inner kitty...ya know ya wanna!

I have met a few friends in my life.
Most were cats.

Profile Wiggo
Avatar
Send message
Joined: 24 Jan 00
Posts: 6494
Credit: 90,451,794
RAC: 74,369
Australia
Message 1321148 - Posted: 29 Dec 2012, 5:58:04 UTC - in response to Message 1321131.

Just ran out of work on one GPU.
Even if i can upload the results, i get nothing but Scheduler errors or timeouts. By then, i can't get new work because of all the uploads that have backed up again. When i finally get them all uploaded, i get a Scheduler error or timeout again.
Etc, etc, etc,

Having 2 or 3 good backup projects at times like these helps (I must see about getting 2 of them added to our Team's participating projects list). ;)

Cheers.
____________

Profile Wiggo
Avatar
Send message
Joined: 24 Jan 00
Posts: 6494
Credit: 90,451,794
RAC: 74,369
Australia
Message 1321150 - Posted: 29 Dec 2012, 6:13:03 UTC - in response to Message 1321148.

Seems as if something has received a fix as my uploads are suddenly start to clear. :)

Cheers.
____________

TBar
Volunteer tester
Send message
Joined: 22 May 99
Posts: 1177
Credit: 41,442,504
RAC: 111,436
United States
Message 1321151 - Posted: 29 Dec 2012, 6:16:04 UTC

It may be about over. My Uploads aren't hanging anymore, most of the 4 minute MB Shorties have been replaced by newly downloaded 20 min Longies, and the Update is working more than not. Maybe...

Keith White
Avatar
Send message
Joined: 29 May 99
Posts: 370
Credit: 2,699,687
RAC: 2,244
United States
Message 1321153 - Posted: 29 Dec 2012, 6:21:51 UTC - in response to Message 1321133.

Uploads are still working but it takes a while and can be very temperamental. However I can't get a scheduler to not timeout and without that I can't see if downloads are equally temperamental.

Something is stopping up the pipes. Cricket now shows a nasty spike of incoming packets with a similar spike on the outgoing packets. Incoming packets now around 1/3 higher than is was yesterday and outgoing packets at nearly 10 kpkt/s.

Looks as if the chart started to go off "normal" about 24 hours ago. I imagine we are seeing some kind of cascading failure condition, clogging the pipes with retries that then trigger more hosts to fail connecting which then triggers more retires that clogs the pipes even more, etc.

This is what I'm seeing in my log

12/29/2012 1:09:16 AM | SETI@home | Reporting 6 completed tasks, requesting new tasks for CPU and ATI
12/29/2012 1:09:18 AM | | Project communication failed: attempting access to reference site
12/29/2012 1:09:18 AM | SETI@home | Scheduler request failed: Server returned nothing (no headers, no data)
12/29/2012 1:09:20 AM | | Internet access OK - project servers may be temporarily down.


____________
"Life is just nature's way of keeping meat fresh." - The Doctor

Previous · 1 . . . 18 · 19 · 20 · 21 · 22 · 23 · Next

Message boards : Number crunching : Panic Mode On (79) Server Problems?

Copyright © 2014 University of California