The Server Issues / Outages Thread - Panic Mode On! (118)

Message boards : Number crunching : The Server Issues / Outages Thread - Panic Mode On! (118)
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 81 · 82 · 83 · 84 · 85 · 86 · 87 . . . 94 · Next

AuthorMessage
TBar
Volunteer tester

Send message
Joined: 22 May 99
Posts: 5204
Credit: 840,779,836
RAC: 2,768
United States
Message 2032274 - Posted: 13 Feb 2020, 21:27:01 UTC
Last modified: 13 Feb 2020, 21:33:09 UTC

It's getting late, and the Splitters have dropped off to the point there is No RTS. This doesn't look good;
Results ready to send = 3
Current result creation rate ** = 4.0569/sec
Results received in last hour ** = 150,781
ID: 2032274 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 2032277 - Posted: 13 Feb 2020, 22:20:50 UTC - in response to Message 2032263.  

Way. way long before that. https://setiathome.berkeley.edu/forum_thread.php?id=18850&postid=154163#154163
I didn't even consider that event in my resume of changes.


. . And Astropulses became much in demand but are not part of the overall SETI MB throughput of the project. They are not even for SETI in any way, they are a sub-contract for another project in another Uni. (As I understand it).

Stephen

? ?
ID: 2032277 · Report as offensive
AllgoodGuy

Send message
Joined: 29 May 01
Posts: 293
Credit: 16,348,499
RAC: 266
United States
Message 2032312 - Posted: 14 Feb 2020, 0:57:23 UTC - in response to Message 2032274.  

It's getting late, and the Splitters have dropped off to the point there is No RTS. This doesn't look good;
Results ready to send = 3
Current result creation rate ** = 4.0569/sec
Results received in last hour ** = 150,781


Recovering a little, seeing AP being cut and delivered.

Results ready to send 35,768
Current result creation rate ** 35.9130/sec
Results received in last hour ** 149,577

Still nowhere near optimal.
ID: 2032312 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 2032313 - Posted: 14 Feb 2020, 0:57:23 UTC - in response to Message 2032280.  
Last modified: 14 Feb 2020, 0:59:40 UTC

Way. way long before that. https://setiathome.berkeley.edu/forum_thread.php?id=18850&postid=154163#154163
I didn't even consider that event in my resume of changes.


. . And Astropulses became much in demand but are not part of the overall SETI MB throughput of the project. They are not even for SETI in any way, they are a sub-contract for another project in another Uni. (As I understand it).

Stephen

? ?

What?? AP not even for SETI in any way??? Where did you get that idea from??

Astropulse FAQ

And why all of a sudden a more than 14 year old message from a totally different thread here???


. . I was under the impression that Seti@home back end does not look at AP results at all, they are shipped off to another project somewhere. Perhaps I have that wrong. Maybe I was just being too parochial and not recognising that the other project is also part of SETI.

. . As for why it is here, look at the quotes and you will see you need to be asking Nick about that.

Stephen

? ?
ID: 2032313 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13854
Credit: 208,696,464
RAC: 304
Australia
Message 2032341 - Posted: 14 Feb 2020, 5:31:22 UTC

Looks like there've been a few server issues in the last 24hrs or so.
Return rate has died for a while, then Ready-to-send gets pounded when the Scheduler comes back to life, but the splitters don't pick up on the increased load.
Grant
Darwin NT
ID: 2032341 · Report as offensive
Boiler Paul

Send message
Joined: 4 May 00
Posts: 232
Credit: 4,965,771
RAC: 64
United States
Message 2032421 - Posted: 14 Feb 2020, 19:34:06 UTC

For some reason, the server status is unavailable. Just a blank black page. Very odd. I wonder how long this will last
ID: 2032421 · Report as offensive
andybutt
Volunteer tester
Avatar

Send message
Joined: 18 Mar 03
Posts: 262
Credit: 164,205,187
RAC: 516
United Kingdom
Message 2032445 - Posted: 14 Feb 2020, 21:23:34 UTC - in response to Message 2032421.  

back on now but loads of red! problem ready for the weekend?
ID: 2032445 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13854
Credit: 208,696,464
RAC: 304
Australia
Message 2032448 - Posted: 14 Feb 2020, 21:55:09 UTC

A large volume of shorties, with many noise bombs thrown in, going through the system at the moment. In progress climbing, Ready to send falling, and all the past couple of weeks of the Validation & Assimilation backlogs clearing have been undone. Backlogs growing & growing fast.
Grant
Darwin NT
ID: 2032448 · Report as offensive
Speedy
Volunteer tester
Avatar

Send message
Joined: 26 Jun 04
Posts: 1643
Credit: 12,921,799
RAC: 89
New Zealand
Message 2032488 - Posted: 15 Feb 2020, 5:09:18 UTC - in response to Message 2032448.  
Last modified: 15 Feb 2020, 5:10:21 UTC

A large volume of shorties, with many noise bombs thrown in, going through the system at the moment. In progress climbing, Ready to send falling, and all the past couple of weeks of the Validation & Assimilation backlogs clearing have been undone. Backlogs growing & growing fast.

As I type the splitters seem to be coping quite well splitting at over 80 a second. RTS is over 2500, could be higher but it could also be a lot lower
ID: 2032488 · Report as offensive
Ville Saari
Avatar

Send message
Joined: 30 Nov 00
Posts: 1158
Credit: 49,177,052
RAC: 82,530
Finland
Message 2032497 - Posted: 15 Feb 2020, 10:01:09 UTC

The assimilation backlog had been going down at a steady rate for several days but couple of days ago it reversed direction. It was 2.2 mil at the lowest point but is now grown to almost 2.8 mil.

And each workunit in assimilation queue is keeping about 2.2 results stuck in the database. They seem to be shown in the validation queue on the SSP. This has pushed the total number of results to a level that forces their generation to be throttled again by stopping the splitters :(
ID: 2032497 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 2032515 - Posted: 15 Feb 2020, 13:51:57 UTC - in response to Message 2032500.  

And the replica again unable to keep up, as of now 5,478 seconds (1.5 hours) behind.


I’m starting to think this is a weekend problem. 3rd Saturday in a row that it’s happened, and usually catches back up by Monday morning.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 2032515 · Report as offensive
W-K 666 Project Donor
Volunteer tester

Send message
Joined: 18 May 99
Posts: 19401
Credit: 40,757,560
RAC: 67
United Kingdom
Message 2032531 - Posted: 15 Feb 2020, 16:08:00 UTC - in response to Message 2032515.  

And the replica again unable to keep up, as of now 5,478 seconds (1.5 hours) behind.


I’m starting to think this is a weekend problem. 3rd Saturday in a row that it’s happened, and usually catches back up by Monday morning.

Looks like someone has applied a touch of TLC, or a size 10 boot, to the problem. The replica has caught up.
ID: 2032531 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13854
Credit: 208,696,464
RAC: 304
Australia
Message 2032570 - Posted: 15 Feb 2020, 21:57:37 UTC

Validation / Assimilation backlog continues to grow.
Shouldn't be too long and all splitter output will cease again till the backlog starts going down again. The number of shorties & noise bombs has dropped off, so we might get lucky.
Grant
Darwin NT
ID: 2032570 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13854
Credit: 208,696,464
RAC: 304
Australia
Message 2032589 - Posted: 15 Feb 2020, 23:31:37 UTC - in response to Message 2032570.  

The number of shorties & noise bombs has dropped off, so we might get lucky.
Looks like i spoke too soon.
Another batch of shorties going through.
Grant
Darwin NT
ID: 2032589 · Report as offensive
W-K 666 Project Donor
Volunteer tester

Send message
Joined: 18 May 99
Posts: 19401
Credit: 40,757,560
RAC: 67
United Kingdom
Message 2032594 - Posted: 16 Feb 2020, 0:12:37 UTC

Panic Mode; ON

The replica is falling behind again.


As of 16 Feb 2020, 0:00:04 UTC
Replica seconds behind master	  1	0m


LOL

With that it's Good Night.
C U Domani
ID: 2032594 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13854
Credit: 208,696,464
RAC: 304
Australia
Message 2032650 - Posted: 16 Feb 2020, 10:19:41 UTC

Getting long delays in Scheduler response, and it's not supplying as many WUs as are being reported.
Hopefully just a brief passing glitch.
Grant
Darwin NT
ID: 2032650 · Report as offensive
Profile Siran d'Vel'nahr
Volunteer tester
Avatar

Send message
Joined: 23 May 99
Posts: 7379
Credit: 44,181,323
RAC: 238
United States
Message 2032681 - Posted: 16 Feb 2020, 15:53:05 UTC

Greetings,

I do believe that we have finally made it beyond the major slow down we've been experiencing over the past 5 or 6 Sundays. :)

I do hope I didn't just jinx the whole operation! ;)

Have a great day! :)

Siran
CAPT Siran d'Vel'nahr - L L & P _\\//
Winders 11 OS? "What a piece of junk!" - L. Skywalker
"Logic is the cement of our civilization with which we ascend from chaos using reason as our guide." - T'Plana-hath
ID: 2032681 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13854
Credit: 208,696,464
RAC: 304
Australia
Message 2032765 - Posted: 17 Feb 2020, 5:10:13 UTC

Got home form work to find my Linux system out of GPU work. Heaps of work to be downloaded- all in super extended extreme backoff mode.
Grant
Darwin NT
ID: 2032765 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1856
Credit: 268,616,081
RAC: 1,349
United States
Message 2032767 - Posted: 17 Feb 2020, 5:27:28 UTC - in response to Message 2032765.  
Last modified: 17 Feb 2020, 5:33:36 UTC

@Grant:
Got home form work to find my Linux system out of GPU work. Heaps of work to be downloaded- all in super extended extreme backoff mode.

Throw this in a cron job, executed from within the BOINC directory, every 15 minutes or so to solve that:
./boinccmd  --project http://setiathome.berkeley.edu update

Note that you can execute this from other than inside the BOINC directory, but will have to add password from gui_rpc_auth.cfg:
./BOINC/boinccmd  --passwd [password]  --project http://setiathome.berkeley.edu update

ID: 2032767 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13854
Credit: 208,696,464
RAC: 304
Australia
Message 2032769 - Posted: 17 Feb 2020, 5:38:28 UTC - in response to Message 2032767.  

@Grant:
Got home form work to find my Linux system out of GPU work. Heaps of work to be downloaded- all in super extended extreme backoff mode.

Throw this in a cron job, executed from within the BOINC directory, every 15 minutes or so to solve that:
./boinccmd  --project http://setiathome.berkeley.edu update

Note that you can execute this from other than inside the BOINC directory, but will have to add password from gui_rpc_auth.cfg:
./BOINC/boinccmd  --passwd [password]  --project http://setiathome.berkeley.edu update
Thanks, i'll put a copy of this aside for when i feel like wrestling with something new.
After today i need a shower, a bit of a read, then a good lie down.
:-)
Grant
Darwin NT
ID: 2032769 · Report as offensive
Previous · 1 . . . 81 · 82 · 83 · 84 · 85 · 86 · 87 . . . 94 · Next

Message boards : Number crunching : The Server Issues / Outages Thread - Panic Mode On! (118)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.