Panic Mode On (79) Server Problems?

Message boards : Number crunching : Panic Mode On (79) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 18 · 19 · 20 · 21 · 22 · Next

AuthorMessage
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1321083 - Posted: 29 Dec 2012, 1:41:46 UTC - in response to Message 1321082.  
Last modified: 29 Dec 2012, 1:43:15 UTC

I have not seen the Seti servers so totally screwed in a long while.
I mean, we have a trifecta going here.
Uploads, downloads, AND scheduler requests all nearly impossible.

Makes me wonder if we have a dang DOS attack on the servers going again.


Perhaps just busy. I had an AP upload and two AP downloads finished within the last 10 minutes.

Not in my experience.
Usually at least downloads remain pretty efficient even during peak loading after outages.
I feel something else is awry here.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1321083 · Report as offensive
Profile Dimly Lit Lightbulb 😀
Volunteer tester
Avatar

Send message
Joined: 30 Aug 08
Posts: 15399
Credit: 7,423,413
RAC: 1
United Kingdom
Message 1321087 - Posted: 29 Dec 2012, 1:56:23 UTC - in response to Message 1321080.  

I have not seen the Seti servers so totally screwed in a long while.
I mean, we have a trifecta going here.
Uploads, downloads, AND scheduler requests all nearly impossible.

Makes me wonder if we have a dang DOS attack on the servers going again.

I dunno, my cache was wiped out due to my lappy having a fit last night resulting in about 15hrs of no crunching. It took a couple of hours before Boinc received the message that the tasks had been reported and since about 18:55:08 UTC to 22:34:42 UTC my cache has been rebuilt.

A lot of timeouts, server returned nothing and so on though :( I'm glad I'm not a mega cruncher, I'd probably be doing a lot of swearing right about now :)

Member of the People Encouraging Niceness In Society club.

ID: 1321087 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1321088 - Posted: 29 Dec 2012, 1:58:54 UTC - in response to Message 1321087.  
Last modified: 29 Dec 2012, 1:59:17 UTC

I have not seen the Seti servers so totally screwed in a long while.
I mean, we have a trifecta going here.
Uploads, downloads, AND scheduler requests all nearly impossible.

Makes me wonder if we have a dang DOS attack on the servers going again.

I dunno, my cache was wiped out due to my lappy having a fit last night resulting in about 15hrs of no crunching. It took a couple of hours before Boinc received the message that the tasks had been reported and since about 18:55:08 UTC to 22:34:42 UTC my cache has been rebuilt.


And your cache consists of 16 WUs.............
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1321088 · Report as offensive
Profile Dimly Lit Lightbulb 😀
Volunteer tester
Avatar

Send message
Joined: 30 Aug 08
Posts: 15399
Credit: 7,423,413
RAC: 1
United Kingdom
Message 1321096 - Posted: 29 Dec 2012, 2:39:21 UTC - in response to Message 1321088.  

I have not seen the Seti servers so totally screwed in a long while.
I mean, we have a trifecta going here.
Uploads, downloads, AND scheduler requests all nearly impossible.

Makes me wonder if we have a dang DOS attack on the servers going again.

I dunno, my cache was wiped out due to my lappy having a fit last night resulting in about 15hrs of no crunching. It took a couple of hours before Boinc received the message that the tasks had been reported and since about 18:55:08 UTC to 22:34:42 UTC my cache has been rebuilt.


And your cache consists of 16 WUs.............

All astro's, the quicker I get 'em the quicker the pipe clears up :)

Member of the People Encouraging Niceness In Society club.

ID: 1321096 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1321098 - Posted: 29 Dec 2012, 2:41:35 UTC - in response to Message 1321096.  

I have not seen the Seti servers so totally screwed in a long while.
I mean, we have a trifecta going here.
Uploads, downloads, AND scheduler requests all nearly impossible.

Makes me wonder if we have a dang DOS attack on the servers going again.

I dunno, my cache was wiped out due to my lappy having a fit last night resulting in about 15hrs of no crunching. It took a couple of hours before Boinc received the message that the tasks had been reported and since about 18:55:08 UTC to 22:34:42 UTC my cache has been rebuilt.


And your cache consists of 16 WUs.............

All astro's, the quicker I get 'em the quicker the pipe clears up :)

LOL...I think it'll take more than your li'l lappy to clean up that mess.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1321098 · Report as offensive
Profile zoom3+1=4
Volunteer tester
Avatar

Send message
Joined: 30 Nov 03
Posts: 65709
Credit: 55,293,173
RAC: 49
United States
Message 1321103 - Posted: 29 Dec 2012, 3:09:44 UTC

ID: 1321103 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1321117 - Posted: 29 Dec 2012, 4:19:22 UTC - in response to Message 1321103.  
Last modified: 29 Dec 2012, 4:36:45 UTC

Scheduler request errors are now occuring within a couple of seconds.
If that doesn't happen then a timeout is more likely than a response. Add to that the shorties & the continuing upload issues.


EDIT- and these shorties are *really* short.
Running 3 at a time on my GTX 560Ti they're being done in just over 4 minutes.
Grant
Darwin NT
ID: 1321117 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1321119 - Posted: 29 Dec 2012, 4:25:41 UTC - in response to Message 1321117.  
Last modified: 29 Dec 2012, 4:26:43 UTC


Scheduler request errors are now occuring within a couple of seconds.
If that doesn't happen then a timeout is more likely than a response. Add to that the shorties & the continuing upload issues.

I have been seeing my caches, such as were stingily allowed recently, diminish greatly.
As is the temperature in the crunching den of the kitties.

When I walked in after work this afternoon and saw the room temperature, I knew something was wrong/. It had already gone to crap hours before, while I was at work.
Multi thousand watts of Seti crunching power not being dissipated by hard working GPUs into the cold winter air.

Meowsigh.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1321119 · Report as offensive
Profile zoom3+1=4
Volunteer tester
Avatar

Send message
Joined: 30 Nov 03
Posts: 65709
Credit: 55,293,173
RAC: 49
United States
Message 1321122 - Posted: 29 Dec 2012, 4:31:53 UTC - in response to Message 1321119.  


Scheduler request errors are now occuring within a couple of seconds.
If that doesn't happen then a timeout is more likely than a response. Add to that the shorties & the continuing upload issues.

I have been seeing my caches, such as were stingily allowed recently, diminish greatly.
As is the temperature in the crunching den of the kitties.

When I walked in after work this afternoon and saw the room temperature, I knew something was wrong/. It had already gone to crap hours before, while I was at work.
Multi thousand watts of Seti crunching power not being dissipated by hard working GPUs into the cold winter air.

Meowsigh.

I had to resort to Einstein and even though My card should be able to handle 6 wu's at a time, problem is, the cpu is stuck at 4, sigh, maybe next month I can fix that... Nothing like 2 months of saving...
The T1 Trust, PRR T1 Class 4-4-4-4 #5550, 1 of America's First HST's
ID: 1321122 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1321131 - Posted: 29 Dec 2012, 5:03:24 UTC - in response to Message 1321122.  
Last modified: 29 Dec 2012, 5:04:45 UTC

Just ran out of work on one GPU.
Even if i can upload the results, i get nothing but Scheduler errors or timeouts. By then, i can't get new work because of all the uploads that have backed up again. When i finally get them all uploaded, i get a Scheduler error or timeout again.
Etc, etc, etc,
Grant
Darwin NT
ID: 1321131 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1321133 - Posted: 29 Dec 2012, 5:06:26 UTC - in response to Message 1321131.  


Just ran out of work on one GPU.
Even if i can upload the results, i get nothing but Scheduler errors or timeouts. By then, i can't get new work because of all the uploads that have backed up again.

I am ditching soon for the night.......
Hope the kitties and myself don't freeze overnight due to the LACK OF FREAKING CRUNCHING>>>>>>>>>ARRGGGGGGHHHHHHHHHHHHHHHHHHHHHHHHHH.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1321133 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 34744
Credit: 261,360,520
RAC: 489
Australia
Message 1321148 - Posted: 29 Dec 2012, 5:58:04 UTC - in response to Message 1321131.  

Just ran out of work on one GPU.
Even if i can upload the results, i get nothing but Scheduler errors or timeouts. By then, i can't get new work because of all the uploads that have backed up again. When i finally get them all uploaded, i get a Scheduler error or timeout again.
Etc, etc, etc,

Having 2 or 3 good backup projects at times like these helps (I must see about getting 2 of them added to our Team's participating projects list). ;)

Cheers.
ID: 1321148 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 34744
Credit: 261,360,520
RAC: 489
Australia
Message 1321150 - Posted: 29 Dec 2012, 6:13:03 UTC - in response to Message 1321148.  

Seems as if something has received a fix as my uploads are suddenly start to clear. :)

Cheers.
ID: 1321150 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1321151 - Posted: 29 Dec 2012, 6:16:04 UTC

It may be about over. My Uploads aren't hanging anymore, most of the 4 minute MB Shorties have been replaced by newly downloaded 20 min Longies, and the Update is working more than not. Maybe...
ID: 1321151 · Report as offensive
Keith White
Avatar

Send message
Joined: 29 May 99
Posts: 392
Credit: 13,035,233
RAC: 22
United States
Message 1321153 - Posted: 29 Dec 2012, 6:21:51 UTC - in response to Message 1321133.  

Uploads are still working but it takes a while and can be very temperamental. However I can't get a scheduler to not timeout and without that I can't see if downloads are equally temperamental.

Something is stopping up the pipes. Cricket now shows a nasty spike of incoming packets with a similar spike on the outgoing packets. Incoming packets now around 1/3 higher than is was yesterday and outgoing packets at nearly 10 kpkt/s.

Looks as if the chart started to go off "normal" about 24 hours ago. I imagine we are seeing some kind of cascading failure condition, clogging the pipes with retries that then trigger more hosts to fail connecting which then triggers more retires that clogs the pipes even more, etc.

This is what I'm seeing in my log

12/29/2012 1:09:16 AM | SETI@home | Reporting 6 completed tasks, requesting new tasks for CPU and ATI
12/29/2012 1:09:18 AM |  | Project communication failed: attempting access to reference site
12/29/2012 1:09:18 AM | SETI@home | Scheduler request failed: Server returned nothing (no headers, no data)
12/29/2012 1:09:20 AM |  | Internet access OK - project servers may be temporarily down.


"Life is just nature's way of keeping meat fresh." - The Doctor
ID: 1321153 · Report as offensive
Profile zoom3+1=4
Volunteer tester
Avatar

Send message
Joined: 30 Nov 03
Posts: 65709
Credit: 55,293,173
RAC: 49
United States
Message 1321155 - Posted: 29 Dec 2012, 6:23:40 UTC - in response to Message 1321153.  

Uploads are still working but it takes a while and can be very temperamental. However I can't get a scheduler to not timeout and without that I can't see if downloads are equally temperamental.

Something is stopping up the pipes. Cricket now shows a nasty spike of incoming packets with a similar spike on the outgoing packets. Incoming packets now around 1/3 higher than is was yesterday and outgoing packets at nearly 10 kpkt/s.

Looks as if the chart started to go off "normal" about 24 hours ago. I imagine we are seeing some kind of cascading failure condition, clogging the pipes with retries that then trigger more hosts to fail connecting which then triggers more retires that clogs the pipes even more, etc.

This is what I'm seeing in my log

12/29/2012 1:09:16 AM | SETI@home | Reporting 6 completed tasks, requesting new tasks for CPU and ATI
12/29/2012 1:09:18 AM |  | Project communication failed: attempting access to reference site
12/29/2012 1:09:18 AM | SETI@home | Scheduler request failed: Server returned nothing (no headers, no data)
12/29/2012 1:09:20 AM |  | Internet access OK - project servers may be temporarily down.


Question is, can anybody report?
The T1 Trust, PRR T1 Class 4-4-4-4 #5550, 1 of America's First HST's
ID: 1321155 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1321158 - Posted: 29 Dec 2012, 6:30:08 UTC - in response to Message 1321155.  
Last modified: 29 Dec 2012, 6:30:59 UTC

Question is, can anybody report?

Very, very, very occasionally.
Very.


And that's with No New Tasks Set.
Even less often when trying for more work.
Grant
Darwin NT
ID: 1321158 · Report as offensive
Keith White
Avatar

Send message
Joined: 29 May 99
Posts: 392
Credit: 13,035,233
RAC: 22
United States
Message 1321160 - Posted: 29 Dec 2012, 6:31:04 UTC - in response to Message 1321155.  
Last modified: 29 Dec 2012, 6:32:47 UTC

Honestly it's becoming SOP that when I complain it starts working. Well sort of.

12/29/2012 1:23:30 AM | SETI@home | Reporting 7 completed tasks, requesting new tasks for CPU and ATI
12/29/2012 1:24:54 AM | SETI@home | Scheduler request completed: got 7 new tasks
12/29/2012 1:24:56 AM | SETI@home | Started download of 22dc09ac.3555.167212.140733193388041.10.92
12/29/2012 1:24:56 AM | SETI@home | Started download of 08oc12aa.26644.119686.4.10.1.vlar
12/29/2012 1:25:05 AM | SETI@home | Computation for task 26no12ad.24534.17654.140733193388043.10.56_1 finished
12/29/2012 1:25:05 AM | SETI@home | Starting task 22no12ad.3861.7429.140733193388038.10.76.vlar_1 using setiathome_enhanced version 603 in slot 0
12/29/2012 1:25:07 AM | SETI@home | Started upload of 26no12ad.24534.17654.140733193388043.10.56_1_0
12/29/2012 1:25:29 AM |  | Project communication failed: attempting access to reference site
12/29/2012 1:25:29 AM | SETI@home | Temporarily failed upload of 26no12ad.24534.17654.140733193388043.10.56_1_0: connect() failed
12/29/2012 1:25:29 AM | SETI@home | Backing off 2 min 13 sec on upload of 26no12ad.24534.17654.140733193388043.10.56_1_0
12/29/2012 1:25:31 AM |  | Internet access OK - project servers may be temporarily down.
12/29/2012 1:27:43 AM | SETI@home | Started upload of 26no12ad.24534.17654.140733193388043.10.56_1_0
12/29/2012 1:28:58 AM |  | Project communication failed: attempting access to reference site
12/29/2012 1:28:58 AM | SETI@home | Temporarily failed upload of 26no12ad.24534.17654.140733193388043.10.56_1_0: connect() failed
12/29/2012 1:28:58 AM | SETI@home | Backing off 4 min 50 sec on upload of 26no12ad.24534.17654.140733193388043.10.56_1_0
12/29/2012 1:28:59 AM |  | Internet access OK - project servers may be temporarily down.
12/29/2012 1:29:59 AM |  | Project communication failed: attempting access to reference site
12/29/2012 1:29:59 AM | SETI@home | Temporarily failed download of 08oc12aa.26644.119686.4.10.1.vlar: transient HTTP error
12/29/2012 1:29:59 AM | SETI@home | Backing off 3 min 5 sec on download of 08oc12aa.26644.119686.4.10.1.vlar
12/29/2012 1:29:59 AM | SETI@home | Started download of 22dc09ac.3555.167212.140733193388041.10.98
12/29/2012 1:30:00 AM |  | Internet access OK - project servers may be temporarily down.


And the answer to my previous post is yes, downloads are equally cranky.
"Life is just nature's way of keeping meat fresh." - The Doctor
ID: 1321160 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 1321162 - Posted: 29 Dec 2012, 6:32:35 UTC - in response to Message 1321150.  

Seems as if something has received a fix as my uploads are suddenly start to clear. :)

Still takes multiple clicks of the Retry button for me.

Grant
Darwin NT
ID: 1321162 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 34744
Credit: 261,360,520
RAC: 489
Australia
Message 1321163 - Posted: 29 Dec 2012, 6:36:05 UTC - in response to Message 1321150.  

Seems as if something has received a fix as my uploads are suddenly start to clear. :)

Cheers.

Well that only lasted long enough to clear 2 rigs. :(

Cheers.
ID: 1321163 · Report as offensive
Previous · 1 . . . 18 · 19 · 20 · 21 · 22 · Next

Message boards : Number crunching : Panic Mode On (79) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.