The Server Issues / Outages Thread - Panic Mode On! (117)

Message boards : Number crunching : The Server Issues / Outages Thread - Panic Mode On! (117)
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 29 · 30 · 31 · 32 · 33 · 34 · 35 . . . 52 · Next

AuthorMessage
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11450
Credit: 29,581,041
RAC: 66
United States
Message 2021687 - Posted: 4 Dec 2019, 17:03:43 UTC

The RTS remains to be alarmingly low.
ID: 2021687 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1646
Credit: 12,921,799
RAC: 89
New Zealand
Message 2021715 - Posted: 4 Dec 2019, 20:47:07 UTC - in response to Message 2021687.  

The RTS remains to be alarmingly low.

It is currently 144,188 I would call that low it is lower than normal granted but there is enough work for about an hour with no output at the current return rate
ID: 2021715 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13904
Credit: 208,696,464
RAC: 304
Australia
Message 2021783 - Posted: 5 Dec 2019, 3:41:27 UTC - in response to Message 2021715.  

It is currently 144,188
Now 477k, so it is recovering, albeit slowly.
Grant
Darwin NT
ID: 2021783 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13904
Credit: 208,696,464
RAC: 304
Australia
Message 2021914 - Posted: 6 Dec 2019, 6:11:09 UTC

Still no graphs.
Grant
Darwin NT
ID: 2021914 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13904
Credit: 208,696,464
RAC: 304
Australia
Message 2021993 - Posted: 7 Dec 2019, 2:21:04 UTC
Last modified: 7 Dec 2019, 2:43:20 UTC

Have we just crashed?
Just had a bunch of stalled downloads on one system (cleared after 3-4min), and now both systems are getting "Project has no tasks available" Scheduler responses, and not a WU in sight.


Edit- eventually got some work, on both systems. And neither can download it; one system's been going for 4min & counting with not a bit transferred (and it's just gone in to timeout). The other keeps timing out on the retries.
Grant
Darwin NT
ID: 2021993 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13904
Credit: 208,696,464
RAC: 304
Australia
Message 2021996 - Posted: 7 Dec 2019, 3:08:28 UTC - in response to Message 2021994.  

Definitely strangeness going on. What I've never seen before is that on my Win box the stalled downloads are in excess to my full cache. e.g. with 2 GPUs I'm limited to 200 GPU tasks, I have 200 tasks in process and it's stalled downloading another 51.
Yeah, that happened on my Windows system that started off with the stalled downloads.
In progress is 346 (but there's only 1 CPU & 2*GPUs).

Everything is now in extended backoff mode, and the website & forums have shifted in to super go slow mode as well.
Grant
Darwin NT
ID: 2021996 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 2021997 - Posted: 7 Dec 2019, 3:18:02 UTC - in response to Message 2021993.  
Last modified: 7 Dec 2019, 3:27:26 UTC

Have we just crashed?
Just had a bunch of stalled downloads on one system (cleared after 3-4min), and now both systems are getting "Project has no tasks available" Scheduler responses, and not a WU in sight.


Edit- eventually got some work, on both systems. And neither can download it; one system's been going for 4min & counting with not a bit transferred (and it's just gone in to timeout). The other keeps timing out on the retries.



. . I have masses of WUs trying to download but it has gone into 'project backoff'.

[edit]
. . I seem to have poked the bear and I am now getting some, very slow, action on the downloads ...

Stephen

:(
ID: 2021997 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1858
Credit: 268,616,081
RAC: 1,349
United States
Message 2021998 - Posted: 7 Dec 2019, 3:25:34 UTC
Last modified: 7 Dec 2019, 3:29:59 UTC

Just cleared, whatever it was ... (somewhat)
ID: 2021998 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13904
Credit: 208,696,464
RAC: 304
Australia
Message 2021999 - Posted: 7 Dec 2019, 3:28:41 UTC - in response to Message 2021997.  

. . I have masses of WUs trying to upload but it has gone into 'projec t backoff'.
Only been getting the odd one or 2 uploads that take a second attempt to go through, been that way for over a month.


My Windows system has managed to get all the downloads to clear (after a lot of retries)- it's set to use George M.
The Linux system is free to choose it's download server, and I've only been able to get a few of it's downloads to clear, till now. The last of the them finally went through.


Scheduler requests either result in "Project has no tasks available" or "Scheduler request failed: Couldn't connect to server" messages. It was "no tasks..." but now it's all "request failed.." responses now.
Grant
Darwin NT
ID: 2021999 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13904
Credit: 208,696,464
RAC: 304
Australia
Message 2022000 - Posted: 7 Dec 2019, 3:30:45 UTC - in response to Message 2021998.  

Just cleared, whatever it was ... (somewhat)
What about Scheduler requests?
Grant
Darwin NT
ID: 2022000 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 2022001 - Posted: 7 Dec 2019, 3:35:20 UTC - in response to Message 2021998.  

Just cleared, whatever it was ... (somewhat)


. . Not quite. The servers are clearly distressed, the downloads finally cleared but I have about 50 WUs waiting to report on just one machine but when I eventually get a response it is 'internal HTTP error'.

. . Not looking pretty ...

Stephen
ID: 2022001 · Report as offensive
Profile Tom M
Volunteer tester

Send message
Joined: 28 Nov 02
Posts: 5126
Credit: 276,046,078
RAC: 462
Message 2022002 - Posted: 7 Dec 2019, 3:36:51 UTC - in response to Message 2022000.  
Last modified: 7 Dec 2019, 3:42:44 UTC

Just cleared, whatever it was ... (somewhat)
What about Scheduler requests?



I just started up my Weekend Warrior machine and it just came back with something like "scheduler request failed, server(s) down"?

Oh, well....
-edit---
Now its reporting no tasks available
--edit--

Tom
A proud member of the OFA (Old Farts Association).
ID: 2022002 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13904
Credit: 208,696,464
RAC: 304
Australia
Message 2022004 - Posted: 7 Dec 2019, 3:44:08 UTC

… and now we're back to "Project has no tasks available" Scheduler responses. At least the Scheduler is back.
Grant
Darwin NT
ID: 2022004 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 2022005 - Posted: 7 Dec 2019, 3:47:16 UTC

. . I am now working on another machine, no CPU crunching, 1 x GPU and running an old stock client but it has 153 WUs cached. Like Jimbocus I am being allocated more WUs than I should be ..............

Stephen

? ?
ID: 2022005 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 2022007 - Posted: 7 Dec 2019, 4:02:36 UTC

What a mess. Everything was fine this morning.
The results out in the field 5,362,638 is high usually we run under 5 million.
I too have been allocated more than my limit.
At least we have WUs to crunch but wow... Tuesdays maintenance is going to have to do extra clean up for this.
ID: 2022007 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13904
Credit: 208,696,464
RAC: 304
Australia
Message 2022009 - Posted: 7 Dec 2019, 4:26:49 UTC - in response to Message 2022007.  
Last modified: 7 Dec 2019, 4:30:30 UTC

At least we have WUs to crunch
Actually, we don't.
My Linux system is out of GPU work, Windows system not too far behind.

Not only is the Scheduler not sending out any work (scratch that- the odd WU here & there on every 4-12 requests), but the Splitters aren't even splitting any new work. Ready-to-send is less than 65k- correction 18.5k. Splitter output, zero.
It's borked.
Grant
Darwin NT
ID: 2022009 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13904
Credit: 208,696,464
RAC: 304
Australia
Message 2022012 - Posted: 7 Dec 2019, 4:45:11 UTC

Yeah, things are rather confused right now.

Looks like the splitters have started producing some work, now hopefully the Scheduler will start allocating them.
Grant
Darwin NT
ID: 2022012 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 2022018 - Posted: 7 Dec 2019, 5:32:05 UTC

I've set myself to no new task for a while, until the situation gets better. I hope everyone who needs WUs can get them.
ID: 2022018 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13904
Credit: 208,696,464
RAC: 304
Australia
Message 2022025 - Posted: 7 Dec 2019, 5:52:11 UTC - in response to Message 2022018.  
Last modified: 7 Dec 2019, 5:55:13 UTC

I've set myself to no new task for a while, until the situation gets better.
Just done that for my Windows system, it's got more than usual to chew on.
Just hoping the Linux system can start picking up some work more frequently before it runs out of work again.

Edit- too late, it's out of GPU work again.
It's odd how one system gets work almost every time, the other almost never.
Grant
Darwin NT
ID: 2022025 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 2022027 - Posted: 7 Dec 2019, 6:12:15 UTC

At what point does the results out in the field begin to be an issue??I'm assuming the db will get too large and the system will crash or slow to a crawl. It is late in California (Friday 10pm ish), so hopefully tomorrow someone can look at the issue.

I'll just add the disclaimer that while no one guarantees us WUs, and I'm certainly not demanding they come in on the weekend and fix things, I'm sure they want to keep the project up.
ID: 2022027 · Report as offensive
Previous · 1 . . . 29 · 30 · 31 · 32 · 33 · 34 · 35 . . . 52 · Next

Message boards : Number crunching : The Server Issues / Outages Thread - Panic Mode On! (117)


 
©2025 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.