Panic Mode On (76) Server Problems?

Message boards : Number crunching : Panic Mode On (76) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 6 · 7 · 8 · 9 · 10 · 11 · 12 . . . 20 · Next

AuthorMessage
.clair.

Send message
Joined: 4 Nov 04
Posts: 1300
Credit: 55,390,408
RAC: 69
United Kingdom
Message 1273255 - Posted: 20 Aug 2012, 11:20:58 UTC

Uploade are slow or take several tries
Scheduler requests time out and try again and time out etc.....
Reporting worn fails for the last six hours (cc_config set at 250)
Three hours of work left
I am not panicing
I am not panicing
I am not panicing
I am not panicing
I_am_not_pan_ic_inn_ggg.
ID: 1273255 · Report as offensive
Profile Link
Avatar

Send message
Joined: 18 Sep 03
Posts: 834
Credit: 1,807,369
RAC: 0
Germany
Message 1273256 - Posted: 20 Aug 2012, 11:23:40 UTC

Uploads and scheduler requests go thru from here OK as far as I can tell from the few I had/needed today. But downloads are quite impossible, with or without a proxy between.
ID: 1273256 · Report as offensive
alephnull
Volunteer tester

Send message
Joined: 16 Mar 03
Posts: 120
Credit: 163,008,396
RAC: 0
United States
Message 1273270 - Posted: 20 Aug 2012, 12:24:29 UTC - in response to Message 1273251.  

Uploads okay here. Scheduler requests okay when I'm on NNT. When I request work, 3 of 4 time out, then I get lost tasks when I do connect.

+1

although ive had two or three occasions where the scheduler requests hung with nnt as well.

lowered cache settings to 2 days but didnt seem to help.

ive noticed these issues only on the machines with gpu work. all my machines with only cpu work schedule fine.

uploads ok with the occasional timeout/retry.

downloads are pretty good depending on the dl server per request (as already known).

rob
ID: 1273270 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 34744
Credit: 261,360,520
RAC: 489
Australia
Message 1273273 - Posted: 20 Aug 2012, 12:34:44 UTC - in response to Message 1273270.  

Other than the usual giving downloads a nudge it's been plain sailing at this end.

Cheers.
ID: 1273273 · Report as offensive
Profile Fred E.
Volunteer tester

Send message
Joined: 22 Jul 99
Posts: 768
Credit: 24,140,697
RAC: 0
United States
Message 1273277 - Posted: 20 Aug 2012, 12:43:05 UTC

Any ideas what's going on this time around? As I said, I understand that post-outage recovery takes a while, but the few days after the outage were actually better than they have been recently. Judging from the countries of the recent posters, the network issues don't seem to be location-specific.


My uneducated guess is database problems, and Scheduler can't get a fast response when work is requested. The startup after the outage was ragged, and they had to disable the replica. So master is running everything - even forums are sluggish at times - I had a lag just starting this post. If it is the database, expect some downtime - hope they don't wait until Tuesday.

That doesn't explain what looks like intermittent upload/download issues. I haven't seen any, but there are plenty of reports. Any other guesses?

Another Fred
Support SETI@home when you search the Web with GoodSearch or shop online with GoodShop.
ID: 1273277 · Report as offensive
Wedge009
Volunteer tester
Avatar

Send message
Joined: 3 Apr 99
Posts: 451
Credit: 431,396,357
RAC: 553
Australia
Message 1273281 - Posted: 20 Aug 2012, 12:54:36 UTC - in response to Message 1273273.  

Other than the usual giving downloads a nudge it's been plain sailing at this end.

Lucky you! Probably not location-specific, then, just luck of the draw.
Soli Deo Gloria
ID: 1273281 · Report as offensive
Profile KWSN Ekky Ekky Ekky
Avatar

Send message
Joined: 25 May 99
Posts: 944
Credit: 52,956,491
RAC: 67
United Kingdom
Message 1273324 - Posted: 20 Aug 2012, 15:21:59 UTC - in response to Message 1273255.  

Uploade are slow or take several tries
Scheduler requests time out and try again and time out etc.....
Reporting worn fails for the last six hours (cc_config set at 250)
Three hours of work left
I am not panicing
I am not panicing
I am not panicing
I am not panicing
I_am_not_pan_ic_inn_ggg.


Is this a transatlantic thing? I am having the same sort of problem. One time work pours in and out at incredible speed, then slows to a crawl, timing out again and again.

ID: 1273324 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 1273328 - Posted: 20 Aug 2012, 15:26:02 UTC

Looks fine now, but I had two failed uploads in the 0730-0800utc time period. They both re-tried about 5 times before making it up to a 2+ hour back-off and then went through around 1100utc.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1273328 · Report as offensive
Profile KWSN Ekky Ekky Ekky
Avatar

Send message
Joined: 25 May 99
Posts: 944
Credit: 52,956,491
RAC: 67
United Kingdom
Message 1273338 - Posted: 20 Aug 2012, 15:42:20 UTC

Just had a look at cricket - bits out has dropped to the bottom line, so perhaps there's a problem after all?


ID: 1273338 · Report as offensive
Terror Australis
Volunteer tester

Send message
Joined: 14 Feb 04
Posts: 1817
Credit: 262,693,308
RAC: 44
Australia
Message 1273341 - Posted: 20 Aug 2012, 15:45:03 UTC

Someone could be kicking something. It's 08:40 Monday Berkeley time, the server status page is blank and the crickets have flatlined.

T.A.
ID: 1273341 · Report as offensive
meijin

Send message
Joined: 24 Mar 12
Posts: 8
Credit: 2,097,274
RAC: 0
United States
Message 1273348 - Posted: 20 Aug 2012, 16:02:05 UTC

Is anyone else having issue reporting/uploading results this morning? All 4 of my boxes can get anything uploaded/reported right now.

Thanks!
ID: 1273348 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1273350 - Posted: 20 Aug 2012, 16:03:43 UTC - in response to Message 1273348.  

Is anyone else having issue reporting/uploading results this morning? All 4 of my boxes can get anything uploaded/reported right now.

Thanks!

The Cricket graphs have just dropped off.
Being Monday morning at about 9:00am in Berkeley, somebody probably just got into the lab and is kicking some tires to reboot things.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1273350 · Report as offensive
Profile Slavac
Volunteer tester
Avatar

Send message
Joined: 27 Apr 11
Posts: 1932
Credit: 17,952,639
RAC: 0
United States
Message 1273357 - Posted: 20 Aug 2012, 16:12:25 UTC - in response to Message 1259773.  

Looking like Bruno barfed.


Executive Director GPU Users Group Inc. -
brad@gpuug.org
ID: 1273357 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1273370 - Posted: 20 Aug 2012, 16:42:44 UTC

And it looks like we are starting to come back up....
Crickets showing some life again, just completed some uploads and reported.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1273370 · Report as offensive
Profile Sunny129
Avatar

Send message
Joined: 7 Nov 00
Posts: 190
Credit: 3,163,755
RAC: 0
United States
Message 1273371 - Posted: 20 Aug 2012, 16:46:15 UTC - in response to Message 1273370.  

And it looks like we are starting to come back up....
Crickets showing some life again, just completed some uploads and reported.

yeah, an hour ago i had 100 or so tasks waiting to upload...they've all uploaded since then. however i'm still having problems reporting...in fact i probably have several hundred tasks waiting to report.
ID: 1273371 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1273373 - Posted: 20 Aug 2012, 16:50:28 UTC - in response to Message 1273371.  
Last modified: 20 Aug 2012, 16:51:18 UTC

And it looks like we are starting to come back up....
Crickets showing some life again, just completed some uploads and reported.

yeah, an hour ago i had 100 or so tasks waiting to upload...they've all uploaded since then. however i'm still having problems reporting...in fact i probably have several hundred tasks waiting to report.

Have you tried <max_tasks_reported>100</max_tasks_reported> in your cc_config.xml file?
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1273373 · Report as offensive
Profile Sunny129
Avatar

Send message
Joined: 7 Nov 00
Posts: 190
Credit: 3,163,755
RAC: 0
United States
Message 1273387 - Posted: 20 Aug 2012, 17:15:55 UTC - in response to Message 1273373.  

And it looks like we are starting to come back up....
Crickets showing some life again, just completed some uploads and reported.

yeah, an hour ago i had 100 or so tasks waiting to upload...they've all uploaded since then. however i'm still having problems reporting...in fact i probably have several hundred tasks waiting to report.

Have you tried <max_tasks_reported>100</max_tasks_reported> in your cc_config.xml file?

thanks...that got my reporting working again.
ID: 1273387 · Report as offensive
kittyman Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 51468
Credit: 1,018,363,574
RAC: 1,004
United States
Message 1273395 - Posted: 20 Aug 2012, 17:21:30 UTC - in response to Message 1273387.  

And it looks like we are starting to come back up....
Crickets showing some life again, just completed some uploads and reported.

yeah, an hour ago i had 100 or so tasks waiting to upload...they've all uploaded since then. however i'm still having problems reporting...in fact i probably have several hundred tasks waiting to report.

Have you tried <max_tasks_reported>100</max_tasks_reported> in your cc_config.xml file?

thanks...that got my reporting working again.

Good to hear.
Reporting should work at higher numbers than that, but the smaller the scheduler request, the easier it seems to be on the servers when they are struggling a bit.
I have this on all of my rigs.
It makes things a little slower getting all work reported after an outage, but unless your rig can crunch perhaps 50 or more WUs every 5 minutes, you should be able to leave it active and have no problems staying current.
"Freedom is just Chaos, with better lighting." Alan Dean Foster

ID: 1273395 · Report as offensive
MikeN

Send message
Joined: 24 Jan 11
Posts: 319
Credit: 64,719,409
RAC: 85
United Kingdom
Message 1273405 - Posted: 20 Aug 2012, 17:32:54 UTC

uploads and reporting seem to be recovering, but now downloads are impossible and the cricket graph is rapidly running out of green paint.
ID: 1273405 · Report as offensive
AndrewM
Volunteer tester

Send message
Joined: 5 Jan 08
Posts: 369
Credit: 34,275,196
RAC: 0
Australia
Message 1273414 - Posted: 20 Aug 2012, 17:43:21 UTC

I've got a lot of Shorties in my dwindling cache.
ID: 1273414 · Report as offensive
Previous · 1 . . . 6 · 7 · 8 · 9 · 10 · 11 · 12 . . . 20 · Next

Message boards : Number crunching : Panic Mode On (76) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.