Panic Mode On (76) Server Problems?

Message boards : Number crunching : Panic Mode On (76) Server Problems?

To post messages, you must log in.

Previous · 1 . . . 6 · 7 · 8 · 9 · 10 · 11 · 12 . . . 21 · Next

AuthorMessage
Profile Gundolf Jahn

Send message
Joined: 19 Sep 00
Posts: 3184
Credit: 386,928
RAC: 40
Germany
Message 1273249 - Posted: 20 Aug 2012, 11:09:13 UTC - in response to Message 1273246.  

For me uploads are fine but totally unable to report, scheduler time out

How many tasks are you trying to report?

Gruß,
Gundolf
ID: 1273249 · Report as offensive
Profile Fred E.
Volunteer tester

Send message
Joined: 22 Jul 99
Posts: 768
Credit: 24,140,697
RAC: 0
United States
Message 1273251 - Posted: 20 Aug 2012, 11:10:29 UTC

Uploads okay here. Scheduler requests okay when I'm on NNT. When I request work, 3 of 4 time out, then I get lost tasks when I do connect.
Another Fred
Support SETI@home when you search the Web with GoodSearch or shop online with GoodShop.
ID: 1273251 · Report as offensive
Wedge009
Volunteer tester
Avatar

Send message
Joined: 3 Apr 99
Posts: 448
Credit: 242,040,991
RAC: 92,752
Australia
Message 1273252 - Posted: 20 Aug 2012, 11:13:37 UTC
Last modified: 20 Aug 2012, 11:14:57 UTC

I understand and expect things to be poor after a multi-day-long outage, but past few days have been atrocious. Normal experience for me is periods of downloads < 1 KiB/s which often stall after a minute or so, alternating with (lesser) periods of downloads ~15 KiB/s. Uploads and scheduler responses consistently speedy.

In contrast, the past few days have been long periods where downloads stall after less than one second and even uploads stalling too, plus scheduler requests hitting time-outs regularly. The only counterpart is brief periods where hosts might get downloads at ~30 KiB/s, but with scheduler responses so rare, those downloads don't do enough anyway.

Any ideas what's going on this time around? As I said, I understand that post-outage recovery takes a while, but the few days after the outage were actually better than they have been recently. Judging from the countries of the recent posters, the network issues don't seem to be location-specific.

Edit: I experience what Fred remarked about the lost tasks as well: when scheduler responses do make it back, it's usually with lost tasks that never made it the first time.
Soli Deo Gloria
ID: 1273252 · Report as offensive
.clair.

Send message
Joined: 4 Nov 04
Posts: 1300
Credit: 39,934,472
RAC: 28,677
United Kingdom
Message 1273255 - Posted: 20 Aug 2012, 11:20:58 UTC

Uploade are slow or take several tries
Scheduler requests time out and try again and time out etc.....
Reporting worn fails for the last six hours (cc_config set at 250)
Three hours of work left
I am not panicing
I am not panicing
I am not panicing
I am not panicing
I_am_not_pan_ic_inn_ggg.
ID: 1273255 · Report as offensive
Profile Link
Avatar

Send message
Joined: 18 Sep 03
Posts: 805
Credit: 1,678,562
RAC: 22
Germany
Message 1273256 - Posted: 20 Aug 2012, 11:23:40 UTC

Uploads and scheduler requests go thru from here OK as far as I can tell from the few I had/needed today. But downloads are quite impossible, with or without a proxy between.
.
ID: 1273256 · Report as offensive
Profile alephnull
Volunteer tester

Send message
Joined: 16 Mar 03
Posts: 120
Credit: 162,964,430
RAC: 0
United States
Message 1273270 - Posted: 20 Aug 2012, 12:24:29 UTC - in response to Message 1273251.  

Uploads okay here. Scheduler requests okay when I'm on NNT. When I request work, 3 of 4 time out, then I get lost tasks when I do connect.

+1

although ive had two or three occasions where the scheduler requests hung with nnt as well.

lowered cache settings to 2 days but didnt seem to help.

ive noticed these issues only on the machines with gpu work. all my machines with only cpu work schedule fine.

uploads ok with the occasional timeout/retry.

downloads are pretty good depending on the dl server per request (as already known).

rob
ID: 1273270 · Report as offensive
Profile Wiggo "Socialist"
Avatar

Send message
Joined: 24 Jan 00
Posts: 10534
Credit: 135,497,071
RAC: 41,937
Australia
Message 1273273 - Posted: 20 Aug 2012, 12:34:44 UTC - in response to Message 1273270.  

Other than the usual giving downloads a nudge it's been plain sailing at this end.

Cheers.
ID: 1273273 · Report as offensive
Profile Fred E.
Volunteer tester

Send message
Joined: 22 Jul 99
Posts: 768
Credit: 24,140,697
RAC: 0
United States
Message 1273277 - Posted: 20 Aug 2012, 12:43:05 UTC

Any ideas what's going on this time around? As I said, I understand that post-outage recovery takes a while, but the few days after the outage were actually better than they have been recently. Judging from the countries of the recent posters, the network issues don't seem to be location-specific.


My uneducated guess is database problems, and Scheduler can't get a fast response when work is requested. The startup after the outage was ragged, and they had to disable the replica. So master is running everything - even forums are sluggish at times - I had a lag just starting this post. If it is the database, expect some downtime - hope they don't wait until Tuesday.

That doesn't explain what looks like intermittent upload/download issues. I haven't seen any, but there are plenty of reports. Any other guesses?

Another Fred
Support SETI@home when you search the Web with GoodSearch or shop online with GoodShop.
ID: 1273277 · Report as offensive
Wedge009
Volunteer tester
Avatar

Send message
Joined: 3 Apr 99
Posts: 448
Credit: 242,040,991
RAC: 92,752
Australia
Message 1273281 - Posted: 20 Aug 2012, 12:54:36 UTC - in response to Message 1273273.  

Other than the usual giving downloads a nudge it's been plain sailing at this end.

Lucky you! Probably not location-specific, then, just luck of the draw.
Soli Deo Gloria
ID: 1273281 · Report as offensive
Profile KWSN Ekky Ekky Ekky
Avatar

Send message
Joined: 25 May 99
Posts: 937
Credit: 20,589,906
RAC: 9,240
United Kingdom
Message 1273324 - Posted: 20 Aug 2012, 15:21:59 UTC - in response to Message 1273255.  

Uploade are slow or take several tries
Scheduler requests time out and try again and time out etc.....
Reporting worn fails for the last six hours (cc_config set at 250)
Three hours of work left
I am not panicing
I am not panicing
I am not panicing
I am not panicing
I_am_not_pan_ic_inn_ggg.


Is this a transatlantic thing? I am having the same sort of problem. One time work pours in and out at incredible speed, then slows to a crawl, timing out again and again.

ID: 1273324 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 2871
Credit: 10,622,605
RAC: 336
United States
Message 1273328 - Posted: 20 Aug 2012, 15:26:02 UTC

Looks fine now, but I had two failed uploads in the 0730-0800utc time period. They both re-tried about 5 times before making it up to a 2+ hour back-off and then went through around 1100utc.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 1273328 · Report as offensive
Profile KWSN Ekky Ekky Ekky
Avatar

Send message
Joined: 25 May 99
Posts: 937
Credit: 20,589,906
RAC: 9,240
United Kingdom
Message 1273338 - Posted: 20 Aug 2012, 15:42:20 UTC

Just had a look at cricket - bits out has dropped to the bottom line, so perhaps there's a problem after all?


ID: 1273338 · Report as offensive
Terror Australis
Volunteer tester

Send message
Joined: 14 Feb 04
Posts: 1791
Credit: 225,341,647
RAC: 10,449
Australia
Message 1273341 - Posted: 20 Aug 2012, 15:45:03 UTC

Someone could be kicking something. It's 08:40 Monday Berkeley time, the server status page is blank and the crickets have flatlined.

T.A.
ID: 1273341 · Report as offensive
meijin

Send message
Joined: 24 Mar 12
Posts: 8
Credit: 2,097,274
RAC: 0
United States
Message 1273348 - Posted: 20 Aug 2012, 16:02:05 UTC

Is anyone else having issue reporting/uploading results this morning? All 4 of my boxes can get anything uploaded/reported right now.

Thanks!
ID: 1273348 · Report as offensive
kittymanProject Donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 45963
Credit: 815,499,288
RAC: 123,976
United States
Message 1273350 - Posted: 20 Aug 2012, 16:03:43 UTC - in response to Message 1273348.  

Is anyone else having issue reporting/uploading results this morning? All 4 of my boxes can get anything uploaded/reported right now.

Thanks!

The Cricket graphs have just dropped off.
Being Monday morning at about 9:00am in Berkeley, somebody probably just got into the lab and is kicking some tires to reboot things.
Always remember.....kitties are all Angels with fur.

Have made friends in this life.
Most were cats.
ID: 1273350 · Report as offensive
Profile Slavac
Volunteer tester
Avatar

Send message
Joined: 27 Apr 11
Posts: 1932
Credit: 17,952,639
RAC: 0
United States
Message 1273357 - Posted: 20 Aug 2012, 16:12:25 UTC - in response to Message 1259773.  

Looking like Bruno barfed.


Executive Director GPU Users Group Inc. -
brad@gpuug.org
ID: 1273357 · Report as offensive
kittymanProject Donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 45963
Credit: 815,499,288
RAC: 123,976
United States
Message 1273370 - Posted: 20 Aug 2012, 16:42:44 UTC

And it looks like we are starting to come back up....
Crickets showing some life again, just completed some uploads and reported.
Always remember.....kitties are all Angels with fur.

Have made friends in this life.
Most were cats.
ID: 1273370 · Report as offensive
Profile Sunny129
Avatar

Send message
Joined: 7 Nov 00
Posts: 190
Credit: 3,163,755
RAC: 0
United States
Message 1273371 - Posted: 20 Aug 2012, 16:46:15 UTC - in response to Message 1273370.  

And it looks like we are starting to come back up....
Crickets showing some life again, just completed some uploads and reported.

yeah, an hour ago i had 100 or so tasks waiting to upload...they've all uploaded since then. however i'm still having problems reporting...in fact i probably have several hundred tasks waiting to report.
ID: 1273371 · Report as offensive
kittymanProject Donor
Volunteer tester
Avatar

Send message
Joined: 9 Jul 00
Posts: 45963
Credit: 815,499,288
RAC: 123,976
United States
Message 1273373 - Posted: 20 Aug 2012, 16:50:28 UTC - in response to Message 1273371.  
Last modified: 20 Aug 2012, 16:51:18 UTC

And it looks like we are starting to come back up....
Crickets showing some life again, just completed some uploads and reported.

yeah, an hour ago i had 100 or so tasks waiting to upload...they've all uploaded since then. however i'm still having problems reporting...in fact i probably have several hundred tasks waiting to report.

Have you tried <max_tasks_reported>100</max_tasks_reported> in your cc_config.xml file?
Always remember.....kitties are all Angels with fur.

Have made friends in this life.
Most were cats.
ID: 1273373 · Report as offensive
Profile Sunny129
Avatar

Send message
Joined: 7 Nov 00
Posts: 190
Credit: 3,163,755
RAC: 0
United States
Message 1273387 - Posted: 20 Aug 2012, 17:15:55 UTC - in response to Message 1273373.  

And it looks like we are starting to come back up....
Crickets showing some life again, just completed some uploads and reported.

yeah, an hour ago i had 100 or so tasks waiting to upload...they've all uploaded since then. however i'm still having problems reporting...in fact i probably have several hundred tasks waiting to report.

Have you tried <max_tasks_reported>100</max_tasks_reported> in your cc_config.xml file?

thanks...that got my reporting working again.
ID: 1273387 · Report as offensive
Previous · 1 . . . 6 · 7 · 8 · 9 · 10 · 11 · 12 . . . 21 · Next

Message boards : Number crunching : Panic Mode On (76) Server Problems?


 
©2016 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.