Panic Mode On (79) Server Problems?

Message boards : Number crunching : Panic Mode On (79) Server Problems?

To post messages, you must log in.

Previous · 1 . . . 18 · 19 · 20 · 21 · 22 · 23 · Next

AuthorMessage
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 3057
Credit: 122,164,842
RAC: 88,915
United States
Message 1321052 - Posted: 28 Dec 2012, 23:32:12 UTC

Ye-Haw a Shorty Storm with a Borked Upload server. How many do you think it will hang?

12/28/2012 6:13:56 PM | SETI@home | Started upload of 03se12aa.10570.17249.140733193388039.10.238_2_0
12/28/2012 6:14:19 PM |  | Project communication failed: attempting access to reference site
12/28/2012 6:14:19 PM | SETI@home | Temporarily failed upload of 03se12aa.10570.17249.140733193388039.10.238_2_0: connect() failed
12/28/2012 6:14:19 PM | SETI@home | Backing off 3 min 50 sec on upload of 03se12aa.10570.17249.140733193388039.10.238_2_0
12/28/2012 6:14:21 PM |  | Internet access OK - project servers may be temporarily down.
12/28/2012 6:17:59 PM | SETI@home | Computation for task 07oc12af.16388.6202.140733193388038.10.1_1 finished
12/28/2012 6:17:59 PM | SETI@home | Starting task 07oc12ag.10905.12746.9.10.247_1 using setiathome_enhanced version 609 (cuda23) in slot 3
12/28/2012 6:18:01 PM | SETI@home | Started upload of 07oc12af.16388.6202.140733193388038.10.1_1_0
12/28/2012 6:18:10 PM | SETI@home | Started upload of 03se12aa.10570.17249.140733193388039.10.238_2_0
12/28/2012 6:18:23 PM |  | Project communication failed: attempting access to reference site
12/28/2012 6:18:23 PM | SETI@home | Temporarily failed upload of 07oc12af.16388.6202.140733193388038.10.1_1_0: connect() failed
12/28/2012 6:18:23 PM | SETI@home | Backing off 3 min 34 sec on upload of 07oc12af.16388.6202.140733193388038.10.1_1_0
12/28/2012 6:18:25 PM |  | Internet access OK - project servers may be temporarily down.
12/28/2012 6:21:45 PM |  | Project communication failed: attempting access to reference site
12/28/2012 6:21:45 PM | SETI@home | Temporarily failed upload of 03se12aa.10570.17249.140733193388039.10.238_2_0: connect() failed
12/28/2012 6:21:45 PM | SETI@home | Backing off 4 min 49 sec on upload of 03se12aa.10570.17249.140733193388039.10.238_2_0
12/28/2012 6:21:47 PM |  | Internet access OK - project servers may be temporarily down.
12/28/2012 6:22:09 PM | SETI@home | Computation for task 07oc12ag.10905.12746.9.10.247_1 finished
12/28/2012 6:22:09 PM | SETI@home | Starting task 07oc12ah.10878.22562.6.10.76_1 using setiathome_enhanced version 609 (cuda23) in slot 3
12/28/2012 6:22:11 PM | SETI@home | Started upload of 07oc12ag.10905.12746.9.10.247_1_0
12/28/2012 6:22:49 PM |  | Project communication failed: attempting access to reference site
12/28/2012 6:22:49 PM | SETI@home | Temporarily failed upload of 07oc12ag.10905.12746.9.10.247_1_0: connect() failed
12/28/2012 6:22:49 PM | SETI@home | Backing off 3 min 11 sec on upload of 07oc12ag.10905.12746.9.10.247_1_0
12/28/2012 6:22:50 PM |  | Internet access OK - project servers may be temporarily down.
12/28/2012 6:26:11 PM | SETI@home | Computation for task 07oc12ah.10878.22562.6.10.76_1 finished
12/28/2012 6:26:11 PM | SETI@home | Starting task 01au12ab.23909.24895.10.10.9_2 using setiathome_enhanced version 609 (cuda23) in slot 3
12/28/2012 6:26:13 PM | SETI@home | Started upload of 07oc12ah.10878.22562.6.10.76_1_0
12/28/2012 6:26:26 PM | SETI@home | Computation for task 01au12ab.23909.24895.10.10.9_2 finished
12/28/2012 6:26:26 PM | SETI@home | Starting task 07oc12af.5577.9065.140733193388039.10.69_0 using setiathome_enhanced version 609 (cuda23) in slot 3
12/28/2012 6:26:28 PM | SETI@home | Started upload of 01au12ab.23909.24895.10.10.9_2_0
12/28/2012 6:26:35 PM |  | Project communication failed: attempting access to reference site
12/28/2012 6:26:35 PM | SETI@home | Temporarily failed upload of 07oc12ah.10878.22562.6.10.76_1_0: connect() failed
12/28/2012 6:26:35 PM | SETI@home | Backing off 3 min 46 sec on upload of 07oc12ah.10878.22562.6.10.76_1_0
12/28/2012 6:26:36 PM |  | Internet access OK - project servers may be temporarily down.
12/28/2012 6:26:50 PM |  | Project communication failed: attempting access to reference site
12/28/2012 6:26:50 PM | SETI@home | Temporarily failed upload of 01au12ab.23909.24895.10.10.9_2_0: connect() failed
12/28/2012 6:26:50 PM | SETI@home | Backing off 3 min 19 sec on upload of 01au12ab.23909.24895.10.10.9_2_0
12/28/2012 6:26:52 PM |  | Internet access OK - project servers may be temporarily down.


:-(

ID: 1321052 · Report as offensive
WinterKnight
Volunteer tester

Send message
Joined: 18 May 99
Posts: 10168
Credit: 30,528,877
RAC: 3,765
United Kingdom
Message 1321054 - Posted: 28 Dec 2012, 23:38:51 UTC - in response to Message 1321047.

It took a few requests but eventually got a few GPU tasks and they all came in at >50kbs.

10-20kB/s here at the moment.

Now with uploads 1kB/s is doing well (when it does eventually go through).

Thats cause you live in the middle of nowhere, or at least can see nowhere from there.

My cousin keeps me informed of the actions of your ISP's and the telco's over there he is not very impressed having lived in the UK, Boston and southern California.

ID: 1321054 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 7475
Credit: 90,902,650
RAC: 45,365
Australia
Message 1321065 - Posted: 29 Dec 2012, 0:49:01 UTC - in response to Message 1321054.

My cousin keeps me informed of the actions of your ISP's and the telco's over there he is not very impressed having lived in the UK, Boston and southern California.

We're not overly thrilled with them either, but they do have the population density & total numbers argument on their side.
A landmass the area of mainland USA, with a total population that's not even 3 times that of London (22.6 million v 8.2 million).
Grant
Darwin NT

ID: 1321065 · Report as offensive
kittymanProject Donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 45875
Credit: 814,688,572
RAC: 121,421
United States
Message 1321080 - Posted: 29 Dec 2012, 1:28:04 UTC

I have not seen the Seti servers so totally screwed in a long while.
I mean, we have a trifecta going here.
Uploads, downloads, AND scheduler requests all nearly impossible.

Makes me wonder if we have a dang DOS attack on the servers going again.


Kitties make wonderful traveling companions on your journey through life.

Have made a few friends in this life.
Most were cats.

ID: 1321080 · Report as offensive
OTS
Volunteer tester

Send message
Joined: 6 Jan 08
Posts: 360
Credit: 20,533,537
RAC: 0
United States
Message 1321082 - Posted: 29 Dec 2012, 1:37:44 UTC - in response to Message 1321080.

I have not seen the Seti servers so totally screwed in a long while.
I mean, we have a trifecta going here.
Uploads, downloads, AND scheduler requests all nearly impossible.

Makes me wonder if we have a dang DOS attack on the servers going again.


Perhaps just busy. I had an AP upload and two AP downloads finished within the last 10 minutes.

ID: 1321082 · Report as offensive
kittymanProject Donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 45875
Credit: 814,688,572
RAC: 121,421
United States
Message 1321083 - Posted: 29 Dec 2012, 1:41:46 UTC - in response to Message 1321082.
Last modified: 29 Dec 2012, 1:43:15 UTC

I have not seen the Seti servers so totally screwed in a long while.
I mean, we have a trifecta going here.
Uploads, downloads, AND scheduler requests all nearly impossible.

Makes me wonder if we have a dang DOS attack on the servers going again.


Perhaps just busy. I had an AP upload and two AP downloads finished within the last 10 minutes.

Not in my experience.
Usually at least downloads remain pretty efficient even during peak loading after outages.
I feel something else is awry here.
Kitties make wonderful traveling companions on your journey through life.

Have made a few friends in this life.
Most were cats.

ID: 1321083 · Report as offensive
Profile Dimly Lit Lightbulb 😀Project Donor
Volunteer tester
Avatar

Send message
Joined: 30 Aug 08
Posts: 14363
Credit: 2,923,362
RAC: 5,480
United Kingdom
Message 1321087 - Posted: 29 Dec 2012, 1:56:23 UTC - in response to Message 1321080.

I have not seen the Seti servers so totally screwed in a long while.
I mean, we have a trifecta going here.
Uploads, downloads, AND scheduler requests all nearly impossible.

Makes me wonder if we have a dang DOS attack on the servers going again.

I dunno, my cache was wiped out due to my lappy having a fit last night resulting in about 15hrs of no crunching. It took a couple of hours before Boinc received the message that the tasks had been reported and since about 18:55:08 UTC to 22:34:42 UTC my cache has been rebuilt.

A lot of timeouts, server returned nothing and so on though :( I'm glad I'm not a mega cruncher, I'd probably be doing a lot of swearing right about now :)

ID: 1321087 · Report as offensive
kittymanProject Donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 45875
Credit: 814,688,572
RAC: 121,421
United States
Message 1321088 - Posted: 29 Dec 2012, 1:58:54 UTC - in response to Message 1321087.
Last modified: 29 Dec 2012, 1:59:17 UTC

I have not seen the Seti servers so totally screwed in a long while.
I mean, we have a trifecta going here.
Uploads, downloads, AND scheduler requests all nearly impossible.

Makes me wonder if we have a dang DOS attack on the servers going again.

I dunno, my cache was wiped out due to my lappy having a fit last night resulting in about 15hrs of no crunching. It took a couple of hours before Boinc received the message that the tasks had been reported and since about 18:55:08 UTC to 22:34:42 UTC my cache has been rebuilt.


And your cache consists of 16 WUs.............
Kitties make wonderful traveling companions on your journey through life.

Have made a few friends in this life.
Most were cats.

ID: 1321088 · Report as offensive
Profile Dimly Lit Lightbulb 😀Project Donor
Volunteer tester
Avatar

Send message
Joined: 30 Aug 08
Posts: 14363
Credit: 2,923,362
RAC: 5,480
United Kingdom
Message 1321096 - Posted: 29 Dec 2012, 2:39:21 UTC - in response to Message 1321088.

I have not seen the Seti servers so totally screwed in a long while.
I mean, we have a trifecta going here.
Uploads, downloads, AND scheduler requests all nearly impossible.

Makes me wonder if we have a dang DOS attack on the servers going again.

I dunno, my cache was wiped out due to my lappy having a fit last night resulting in about 15hrs of no crunching. It took a couple of hours before Boinc received the message that the tasks had been reported and since about 18:55:08 UTC to 22:34:42 UTC my cache has been rebuilt.


And your cache consists of 16 WUs.............

All astro's, the quicker I get 'em the quicker the pipe clears up :)

ID: 1321096 · Report as offensive
kittymanProject Donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 45875
Credit: 814,688,572
RAC: 121,421
United States
Message 1321098 - Posted: 29 Dec 2012, 2:41:35 UTC - in response to Message 1321096.

I have not seen the Seti servers so totally screwed in a long while.
I mean, we have a trifecta going here.
Uploads, downloads, AND scheduler requests all nearly impossible.

Makes me wonder if we have a dang DOS attack on the servers going again.

I dunno, my cache was wiped out due to my lappy having a fit last night resulting in about 15hrs of no crunching. It took a couple of hours before Boinc received the message that the tasks had been reported and since about 18:55:08 UTC to 22:34:42 UTC my cache has been rebuilt.


And your cache consists of 16 WUs.............

All astro's, the quicker I get 'em the quicker the pipe clears up :)

LOL...I think it'll take more than your li'l lappy to clean up that mess.
Kitties make wonderful traveling companions on your journey through life.

Have made a few friends in this life.
Most were cats.

ID: 1321098 · Report as offensive
zoom314
Volunteer tester
Avatar

Send message
Joined: 30 Nov 03
Posts: 56733
Credit: 40,725,595
RAC: 4,936
United States
Message 1321103 - Posted: 29 Dec 2012, 3:09:44 UTC

Upload timeout has been here...


Pluto is still a planet.

Beep! Beep!

ID: 1321103 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 7475
Credit: 90,902,650
RAC: 45,365
Australia
Message 1321117 - Posted: 29 Dec 2012, 4:19:22 UTC - in response to Message 1321103.
Last modified: 29 Dec 2012, 4:36:45 UTC

Scheduler request errors are now occuring within a couple of seconds.
If that doesn't happen then a timeout is more likely than a response. Add to that the shorties & the continuing upload issues.


EDIT- and these shorties are *really* short.
Running 3 at a time on my GTX 560Ti they're being done in just over 4 minutes.


Grant
Darwin NT

ID: 1321117 · Report as offensive
kittymanProject Donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 45875
Credit: 814,688,572
RAC: 121,421
United States
Message 1321119 - Posted: 29 Dec 2012, 4:25:41 UTC - in response to Message 1321117.
Last modified: 29 Dec 2012, 4:26:43 UTC


Scheduler request errors are now occuring within a couple of seconds.
If that doesn't happen then a timeout is more likely than a response. Add to that the shorties & the continuing upload issues.

I have been seeing my caches, such as were stingily allowed recently, diminish greatly.
As is the temperature in the crunching den of the kitties.

When I walked in after work this afternoon and saw the room temperature, I knew something was wrong/. It had already gone to crap hours before, while I was at work.
Multi thousand watts of Seti crunching power not being dissipated by hard working GPUs into the cold winter air.

Meowsigh.
Kitties make wonderful traveling companions on your journey through life.

Have made a few friends in this life.
Most were cats.

ID: 1321119 · Report as offensive
zoom314
Volunteer tester
Avatar

Send message
Joined: 30 Nov 03
Posts: 56733
Credit: 40,725,595
RAC: 4,936
United States
Message 1321122 - Posted: 29 Dec 2012, 4:31:53 UTC - in response to Message 1321119.


Scheduler request errors are now occuring within a couple of seconds.
If that doesn't happen then a timeout is more likely than a response. Add to that the shorties & the continuing upload issues.

I have been seeing my caches, such as were stingily allowed recently, diminish greatly.
As is the temperature in the crunching den of the kitties.

When I walked in after work this afternoon and saw the room temperature, I knew something was wrong/. It had already gone to crap hours before, while I was at work.
Multi thousand watts of Seti crunching power not being dissipated by hard working GPUs into the cold winter air.

Meowsigh.

I had to resort to Einstein and even though My card should be able to handle 6 wu's at a time, problem is, the cpu is stuck at 4, sigh, maybe next month I can fix that... Nothing like 2 months of saving...
Pluto is still a planet.

Beep! Beep!

ID: 1321122 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 7475
Credit: 90,902,650
RAC: 45,365
Australia
Message 1321131 - Posted: 29 Dec 2012, 5:03:24 UTC - in response to Message 1321122.
Last modified: 29 Dec 2012, 5:04:45 UTC

Just ran out of work on one GPU.
Even if i can upload the results, i get nothing but Scheduler errors or timeouts. By then, i can't get new work because of all the uploads that have backed up again. When i finally get them all uploaded, i get a Scheduler error or timeout again.
Etc, etc, etc,


Grant
Darwin NT

ID: 1321131 · Report as offensive
kittymanProject Donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 45875
Credit: 814,688,572
RAC: 121,421
United States
Message 1321133 - Posted: 29 Dec 2012, 5:06:26 UTC - in response to Message 1321131.


Just ran out of work on one GPU.
Even if i can upload the results, i get nothing but Scheduler errors or timeouts. By then, i can't get new work because of all the uploads that have backed up again.

I am ditching soon for the night.......
Hope the kitties and myself don't freeze overnight due to the LACK OF FREAKING CRUNCHING>>>>>>>>>ARRGGGGGGHHHHHHHHHHHHHHHHHHHHHHHHHH.
Kitties make wonderful traveling companions on your journey through life.

Have made a few friends in this life.
Most were cats.

ID: 1321133 · Report as offensive
Profile Wiggo "Socialist"
Avatar

Send message
Joined: 24 Jan 00
Posts: 10499
Credit: 135,206,264
RAC: 37,368
Australia
Message 1321148 - Posted: 29 Dec 2012, 5:58:04 UTC - in response to Message 1321131.

Just ran out of work on one GPU.
Even if i can upload the results, i get nothing but Scheduler errors or timeouts. By then, i can't get new work because of all the uploads that have backed up again. When i finally get them all uploaded, i get a Scheduler error or timeout again.
Etc, etc, etc,

Having 2 or 3 good backup projects at times like these helps (I must see about getting 2 of them added to our Team's participating projects list). ;)

Cheers.

ID: 1321148 · Report as offensive
Profile Wiggo "Socialist"
Avatar

Send message
Joined: 24 Jan 00
Posts: 10499
Credit: 135,206,264
RAC: 37,368
Australia
Message 1321150 - Posted: 29 Dec 2012, 6:13:03 UTC - in response to Message 1321148.

Seems as if something has received a fix as my uploads are suddenly start to clear. :)

Cheers.


ID: 1321150 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 3057
Credit: 122,164,842
RAC: 88,915
United States
Message 1321151 - Posted: 29 Dec 2012, 6:16:04 UTC

It may be about over. My Uploads aren't hanging anymore, most of the 4 minute MB Shorties have been replaced by newly downloaded 20 min Longies, and the Update is working more than not. Maybe...

ID: 1321151 · Report as offensive
Keith White
Avatar

Send message
Joined: 29 May 99
Posts: 384
Credit: 4,626,304
RAC: 1,980
United States
Message 1321153 - Posted: 29 Dec 2012, 6:21:51 UTC - in response to Message 1321133.

Uploads are still working but it takes a while and can be very temperamental. However I can't get a scheduler to not timeout and without that I can't see if downloads are equally temperamental.

Something is stopping up the pipes. Cricket now shows a nasty spike of incoming packets with a similar spike on the outgoing packets. Incoming packets now around 1/3 higher than is was yesterday and outgoing packets at nearly 10 kpkt/s.

Looks as if the chart started to go off "normal" about 24 hours ago. I imagine we are seeing some kind of cascading failure condition, clogging the pipes with retries that then trigger more hosts to fail connecting which then triggers more retires that clogs the pipes even more, etc.

This is what I'm seeing in my log

12/29/2012 1:09:16 AM | SETI@home | Reporting 6 completed tasks, requesting new tasks for CPU and ATI
12/29/2012 1:09:18 AM |  | Project communication failed: attempting access to reference site
12/29/2012 1:09:18 AM | SETI@home | Scheduler request failed: Server returned nothing (no headers, no data)
12/29/2012 1:09:20 AM |  | Internet access OK - project servers may be temporarily down.


"Life is just nature's way of keeping meat fresh." - The Doctor

ID: 1321153 · Report as offensive
Previous · 1 . . . 18 · 19 · 20 · 21 · 22 · 23 · Next

Message boards : Number crunching : Panic Mode On (79) Server Problems?


 
©2016 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.