Panic Mode On (114) Server Problems?

Message boards : Number crunching : Panic Mode On (114) Server Problems?
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 40 · 41 · 42 · 43 · 44 · 45 · Next

AuthorMessage
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 1979905 - Posted: 11 Feb 2019, 17:10:58 UTC
Last modified: 11 Feb 2019, 17:13:13 UTC

still in panic mode....
10k+ in the AP rts queue. I hope I can get one or two of those when seti decides to hand out WUs again.

edit:
results returned is down to 118k. it is 9am in CA. can someone send the bat signal??
ID: 1979905 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1979907 - Posted: 11 Feb 2019, 17:43:01 UTC

Was out on two hosts and working on other projects. Now refilling again from Seti. They really need to get the file purging done tomorrow.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1979907 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22290
Credit: 416,307,556
RAC: 380
United Kingdom
Message 1979908 - Posted: 11 Feb 2019, 17:52:25 UTC

There is a very substantial proportion of folks who see that the servers are having an iffy fit and stop crunching until the iffy fit is over (or, more likely, a week after the iffy fit finished). Thus we end up with loads of returned tasks to validate.
Well that's my theory, you are welcome to your own.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 1979908 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1979928 - Posted: 11 Feb 2019, 19:36:13 UTC

It's back;
Mon Feb 11 14:31:03 2019 EST | SETI@home | Project has no tasks available...
ID: 1979928 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1979934 - Posted: 11 Feb 2019, 20:11:57 UTC - in response to Message 1979928.  

Yes, out of tasks again. Looks like the replica lag is growing again after being in an unnatural state of being 0 for almost 4 days. See that the WU/results purging has peaked and started trending downwards. Think they are alternating which processes get priority every hour or so.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1979934 · Report as offensive
Profile Freewill Project Donor
Avatar

Send message
Joined: 19 May 99
Posts: 766
Credit: 354,398,348
RAC: 11,693
United States
Message 1979937 - Posted: 11 Feb 2019, 20:41:48 UTC - in response to Message 1979934.  

This crap is starting to get old. I've invested time, hardware, and electricity to support this worthwhile cause only to have cranky software thwart our progress.
ID: 1979937 · Report as offensive
Ben

Send message
Joined: 15 Jun 99
Posts: 54
Credit: 60,003,756
RAC: 150
United States
Message 1979956 - Posted: 11 Feb 2019, 21:46:52 UTC - in response to Message 1979937.  

No task available, all splitters seem to be idle and I'm out of work units.
Sigh.
ID: 1979956 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1643
Credit: 12,921,799
RAC: 89
New Zealand
Message 1979958 - Posted: 11 Feb 2019, 21:57:37 UTC

As I write this 4,587,629 tasks are out in the field
ID: 1979958 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1979967 - Posted: 11 Feb 2019, 22:49:25 UTC - in response to Message 1979958.  
Last modified: 11 Feb 2019, 23:22:14 UTC

. . Well the splitters are marking time because the RTS is 850,000, yet the scheduler/downloaders are not sending them to the field. We have seen this before, very often lately, so it looks like it will be another outage where the rigs get the day off ... :(

<edit>
. . Nothing but "no tasks". Main rig OOW in 15 mins. Others will be within an hour or 2, so it seems I will have to sit out this outage too.

Stephen

:(
ID: 1979967 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1643
Credit: 12,921,799
RAC: 89
New Zealand
Message 1979980 - Posted: 12 Feb 2019, 0:53:41 UTC

I was able to pick up 9 for my GPU with a cache of 0.1
ID: 1979980 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 35233
Credit: 261,360,520
RAC: 489
Australia
Message 1980005 - Posted: 12 Feb 2019, 5:23:30 UTC

I might not be keeping full caches here, but they're not far off either.

Cheers.
ID: 1980005 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13771
Credit: 208,696,464
RAC: 304
Australia
Message 1980010 - Posted: 12 Feb 2019, 5:37:21 UTC

Web site & forums slow, "Project has no tasks available" for random times & random lengths of time each day.
The new normal.
Grant
Darwin NT
ID: 1980010 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1980012 - Posted: 12 Feb 2019, 5:47:41 UTC

I haven't received any work on any host for about 4 hours so far. Just no tasks are available messages.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1980012 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13771
Credit: 208,696,464
RAC: 304
Australia
Message 1980013 - Posted: 12 Feb 2019, 5:54:26 UTC - in response to Message 1980012.  

I haven't received any work on any host for about 4 hours so far. Just no tasks are available messages.

Pretty sure it was about 3-4 hours of that yesterday as well before I started getting work again.
Grant
Darwin NT
ID: 1980013 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 1980017 - Posted: 12 Feb 2019, 6:34:57 UTC - in response to Message 1980013.  

You may be right. One host just got a dozen or so tasks and had troubles with stalled downloads. So maybe the servers are waking up again.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 1980017 · Report as offensive
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 1980019 - Posted: 12 Feb 2019, 6:46:11 UTC

This last time the servers regained contact the RTS was up to 1+ million, shame it won't send them.
This reminds me of the last major problems obtaining work, which was fixed by sending Arecibo VLARs to the GPUs. Things would go along just fine,
then when the Arecibo tasks started to be split you had the same problem as now. Perhaps the solution is to just Stop splitting Arecibo tasks, who knows.
I always said it would be best to have them separate, Arecibos using one App and 1 one cache, then BLCs using a different App with it's own cache.
Something like the APs & MBs. Maybe the Arecibos will run out shortly and no more will be posted for a while...
ID: 1980019 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 35233
Credit: 261,360,520
RAC: 489
Australia
Message 1980020 - Posted: 12 Feb 2019, 6:56:15 UTC
Last modified: 12 Feb 2019, 7:01:12 UTC

My caches have been full for near on the last hour now, but they were never more than 50 short at any time.

Cheers.
ID: 1980020 · Report as offensive
Sirius B Project Donor
Volunteer tester
Avatar

Send message
Joined: 26 Dec 00
Posts: 24884
Credit: 3,081,182
RAC: 7
Ireland
Message 1980039 - Posted: 12 Feb 2019, 10:51:51 UTC

Poodling along nicely in the slow lane. :-)
ID: 1980039 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13771
Credit: 208,696,464
RAC: 304
Australia
Message 1980053 - Posted: 12 Feb 2019, 11:29:57 UTC
Last modified: 12 Feb 2019, 11:33:58 UTC

I see the problems are continuing.

From the looks of the Ready-to-send buffer i'm thinking there's some process that's gotten away & hogging up CPU resources.
The Splitters run on after their usual shut down point, then take ages to restart after going well below their usual start point. Scheduler allocation of work is sporadic, at best. At worst, it's hours between getting any work allocated (regardless of how much is there to be had).

The Deleters have finally cleared their backlog, but now the Purgers are struggling to clear theirs. The amount of work returned per hour is quite low (compared to past times, with much larger volumes & when the system was able to keep up).
A lot of the issues seemed to resurface when the Replica came back on line. Maybe see how the system goes with the Replica off line for 12-24 hours? See if it can recover, or is there something else at play?


Edit- and even viewing my limited number of Tasks is becoming almost impossible (although due to the deleter then purger backlogs those numbers are way higher than usual).
Grant
Darwin NT
ID: 1980053 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 1980055 - Posted: 12 Feb 2019, 11:45:38 UTC - in response to Message 1980053.  

. . I'm surprised I have work on my rigs. Since I got home every work request has been "no tasks" :(

Stephen

:(
ID: 1980055 · Report as offensive
Previous · 1 . . . 40 · 41 · 42 · 43 · 44 · 45 · Next

Message boards : Number crunching : Panic Mode On (114) Server Problems?


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.