The Server Issues / Outages Thread - Panic Mode On! (117)

Message boards : Number crunching : The Server Issues / Outages Thread - Panic Mode On! (117)
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 23 · 24 · 25 · 26 · 27 · 28 · 29 . . . 52 · Next

AuthorMessage
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 34744
Credit: 261,360,520
RAC: 489
Australia
Message 2015370 - Posted: 14 Oct 2019, 2:09:54 UTC - in response to Message 2015367.  

Ops... It's pub time. This time I will do the sacrifice.

Will meet you all there while waiting the system returns.
Save a bar stool for me. Need some refreshment from watching football all day.
Did I miss something while I was at the pub having 3 beers?

Cheers.
ID: 2015370 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 2015388 - Posted: 14 Oct 2019, 5:46:15 UTC

. . Hmmm, I don't suppose there is anyone who would care to do something with those 4 ancient tapes that have been hanging about on the splitters since Noah wore shorts? Two very old Blc22 and 2 x very old Blc34 tapes that have shown no sign of starting in the many months they have been loitering on the system. Maybe it is time to either kick start them or toss them to the weeds.

. . Just a thought ...

Stephen

? ?
ID: 2015388 · Report as offensive
Profile Mr. Kevvy Crowdfunding Project Donor*Special Project $250 donor
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 15 May 99
Posts: 3776
Credit: 1,114,826,392
RAC: 3,319
Canada
Message 2015535 - Posted: 15 Oct 2019, 19:55:17 UTC
Last modified: 15 Oct 2019, 19:56:12 UTC

Dr. Korpela indicated that he just may have found and fixed the Sunday unresponsive scheduler issue. Let's hope we're back to merely one weekly outage...
ID: 2015535 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 2015536 - Posted: 15 Oct 2019, 19:58:43 UTC - in response to Message 2015535.  

Dr. Korpela indicated that he just may have found and fixed the Sunday unresponsive scheduler issue. Let's hope we're back to merely one weekly outage...


ID: 2015536 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 2015537 - Posted: 15 Oct 2019, 20:28:14 UTC - in response to Message 2015535.  

Dr. Korpela indicated that he just may have found and fixed the Sunday unresponsive scheduler issue. Let's hope we're back to merely one weekly outage...


YES!! good work Dr. Korpela. and thanks to Mr. Kevvy for bringing us the news.
ID: 2015537 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5124
Credit: 276,046,078
RAC: 462
Message 2015549 - Posted: 15 Oct 2019, 22:34:49 UTC - in response to Message 2015536.  

Dr. Korpela indicated that he just may have found and fixed the Sunday unresponsive scheduler issue. Let's hope we're back to merely one weekly outage...



+1
A proud member of the OFA (Old Farts Association).
ID: 2015549 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1849
Credit: 268,616,081
RAC: 1,349
United States
Message 2015555 - Posted: 15 Oct 2019, 23:24:00 UTC - in response to Message 2015536.  

Dr. Korpela indicated that he just may have found and fixed the Sunday unresponsive scheduler issue. Let's hope we're back to merely one weekly outage...


As Einstein goes into mourning .... :)
ID: 2015555 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 2015557 - Posted: 16 Oct 2019, 0:01:47 UTC - in response to Message 2015535.  

Dr. Korpela indicated that he just may have found and fixed the Sunday unresponsive scheduler issue. Let's hope we're back to merely one weekly outage...


. . Hip, Hip, Hooray!

Stephen

:)
ID: 2015557 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 2015581 - Posted: 16 Oct 2019, 8:34:23 UTC

The file deleters don't appear to be doing much since the outage- their backlog continues to grow , and the splitter output continues to be poor.
Grant
Darwin NT
ID: 2015581 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 2015661 - Posted: 16 Oct 2019, 23:19:01 UTC

. . I haven't been monitoring deletions but since the big Sunday outage nearly two weeks ago my uploads/ downloads and work requests have been really, really slllloooooowwww! Completely stalled now

Stephen

:(
ID: 2015661 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 34744
Credit: 261,360,520
RAC: 489
Australia
Message 2015681 - Posted: 17 Oct 2019, 3:26:46 UTC - in response to Message 2015661.  

. . I haven't been monitoring deletions but since the big Sunday outage nearly two weeks ago my uploads/ downloads and work requests have been really, really slllloooooowwww! Completely stalled now

Stephen :(
No such problems at all like that here, have you done a modem/router/switch reboot since then?

Cheers.
ID: 2015681 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 2015692 - Posted: 17 Oct 2019, 7:42:38 UTC - in response to Message 2015681.  

. . I haven't been monitoring deletions but since the big Sunday outage nearly two weeks ago my uploads/ downloads and work requests have been really, really slllloooooowwww! Completely stalled now

Stephen :(
No such problems at all like that here, have you done a modem/router/switch reboot since then?

Cheers.


. . It's worst on the Windows box, seriously thinking of mutating it into a Linux box as well. It's the only one that stalls. I have done a complete shutdown of boinc/manager and it seems a little better now. I will monitor progress ...

Stephen

:(
ID: 2015692 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1639
Credit: 12,921,799
RAC: 89
New Zealand
Message 2015695 - Posted: 17 Oct 2019, 7:51:10 UTC - in response to Message 2015692.  

. . I haven't been monitoring deletions but since the big Sunday outage nearly two weeks ago my uploads/ downloads and work requests have been really, really slllloooooowwww! Completely stalled now

Stephen :(
No such problems at all like that here, have you done a modem/router/switch reboot since then?

Cheers.


. . It's worst on the Windows box, seriously thinking of mutating it into a Linux box as well. It's the only one that stalls. I have done a complete shutdown of boinc/manager and it seems a little better now. I will monitor progress ...

Stephen

:(

Pleased things are looking on the up for you. If you were to convert the machine to Linux it would certainly boost your RAC.
ID: 2015695 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 2015820 - Posted: 18 Oct 2019, 5:42:31 UTC

The servers have certainly been having issues lately.
After the outage the deleters weren't doing much at all. Once they got their act together and cleared the backlog, the file purger had issues clearing that backlog. After a day or so they managed to clear their backlog, and now the Awaiting validation is becoming backlogged (along with WU awaiting assimilation).
Grant
Darwin NT
ID: 2015820 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 2015822 - Posted: 18 Oct 2019, 5:56:07 UTC - in response to Message 2015820.  

The servers have certainly been having issues lately.
After the outage the deleters weren't doing much at all. Once they got their act together and cleared the backlog, the file purger had issues clearing that backlog. After a day or so they managed to clear their backlog, and now the Awaiting validation is becoming backlogged (along with WU awaiting assimilation).


I've seen this too. I don't see any reason for it though, the hourly return rate is within reason. Something is off.
ID: 2015822 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 2016454 - Posted: 24 Oct 2019, 5:08:27 UTC

Looks like another issue with the Validators again- a huge backlog there.
Grant
Darwin NT
ID: 2016454 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 2016459 - Posted: 24 Oct 2019, 7:41:09 UTC

Validator backlog continues to grow- 500k over it's usual level & rising sharply.
Grant
Darwin NT
ID: 2016459 · Report as offensive
Profile Sebastian M. Bobrecki
Volunteer tester

Send message
Joined: 7 Feb 02
Posts: 23
Credit: 38,375,443
RAC: 0
Poland
Message 2016492 - Posted: 24 Oct 2019, 16:47:54 UTC

If I remember correctly "Results returned and awaiting validation" are results awaiting for quorum. So it is not a server problem. I'm guessing that this could be due to lots of Arecibo shorties, which ends with "-9 result_overflow" and don't match between opt. and normal app., so they requires another wingman.
ID: 2016492 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13720
Credit: 208,696,464
RAC: 304
Australia
Message 2016545 - Posted: 25 Oct 2019, 5:25:05 UTC - in response to Message 2016492.  

If I remember correctly "Results returned and awaiting validation" are results awaiting for quorum.

It's a server issue- there's no way there are almost a million shorties out there waiting to be validated. For as much as we rage against results that take forever to be returned, the vast majority are returned within a few days.
The backlog has started to clear, but it's got a long way to go before it gets back down to it's usual levels.


Arecibo shorties, which ends with "-9 result_overflow"
Those aren't shorties, they are noise bombs, because they bomb out early (in a matter of seconds). Shorties are processed all the way through, but don't take nearly a long as an average mid range WU to process.
Grant
Darwin NT
ID: 2016545 · Report as offensive
Profile Sebastian M. Bobrecki
Volunteer tester

Send message
Joined: 7 Feb 02
Posts: 23
Credit: 38,375,443
RAC: 0
Poland
Message 2016550 - Posted: 25 Oct 2019, 6:15:00 UTC - in response to Message 2016545.  

It's a server issue- there's no way there are almost a million shorties out there waiting to be validated. For as much as we rage against results that take forever to be returned, the vast majority are returned within a few days.
The backlog has started to clear, but it's got a long way to go before it gets back down to it's usual levels.
I think the behavior match with what I think before. I looked at logs of my clients and yesterday was a lot more Arecibo tasks than usual. Actually more Arecibo than blc. According to haveland it was about 500-600k more "awaiting validation" than usual. With over 120k results returned per hour it's not that much. I also saw yesterday that my validation inconclusive increased from usual below 100 to over 150. And I'm not a mega rac cruncher. Today I see it starts dropping with some increase in invalids.


Those aren't shorties, they are noise bombs, because they bomb out early (in a matter of seconds). Shorties are processed all the way through, but don't take nearly a long as an average mid range WU to process.
This is more a matter of naming convention. For me naming those few seconds tasks noise bombs makes a negative feelings (as noise is something we don't want). And actually they could be what we are all looking for. But as I wrote this is just a matter of naming convention.
ID: 2016550 · Report as offensive
Previous · 1 . . . 23 · 24 · 25 · 26 · 27 · 28 · 29 . . . 52 · Next

Message boards : Number crunching : The Server Issues / Outages Thread - Panic Mode On! (117)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.