The Server Issues / Outages Thread - Panic Mode On! (118)

Message boards : Number crunching : The Server Issues / Outages Thread - Panic Mode On! (118)
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 84 · 85 · 86 · 87 · 88 · 89 · 90 . . . 94 · Next

AuthorMessage
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1855
Credit: 268,616,081
RAC: 1,349
United States
Message 2033008 - Posted: 19 Feb 2020, 11:03:22 UTC

No sticky uploads or downloads today.
Log file indicates BoincTasks _seems_ to be handling these.
ID: 2033008 · Report as offensive
Stephen "Heretic" Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 20 Sep 12
Posts: 5557
Credit: 192,787,363
RAC: 628
Australia
Message 2033029 - Posted: 19 Feb 2020, 15:44:06 UTC - in response to Message 2033008.  

No sticky uploads or downloads today.
Log file indicates BoincTasks _seems_ to be handling these.


. . I am not running Boinc Tasks and I am getting plenty of both ...

Stephen

:(
ID: 2033029 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13797
Credit: 208,696,464
RAC: 304
Australia
Message 2033122 - Posted: 20 Feb 2020, 6:10:42 UTC
Last modified: 20 Feb 2020, 6:19:05 UTC

Are we about to crash? Forums are almost non-responsive, web site taking a while to come up.


And all attempted Scheduler contact results in failure.
Grant
Darwin NT
ID: 2033122 · Report as offensive
Profile ravkin
Avatar

Send message
Joined: 14 Aug 09
Posts: 20
Credit: 11,165,042
RAC: 158
United States
Message 2033123 - Posted: 20 Feb 2020, 7:00:16 UTC - in response to Message 2033122.  

I think so but why everything seems normal.
ID: 2033123 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 2033124 - Posted: 20 Feb 2020, 7:01:17 UTC - in response to Message 2033122.  

Are we about to crash? Forums are almost non-responsive, web site taking a while to come up.


And all attempted Scheduler contact results in failure.


Forums are responding slightly better now than 20 minutes ago, but I'm getting nothing from the server...can't upload/download WUs

I'm knocking, but no one is home.
ID: 2033124 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13797
Credit: 208,696,464
RAC: 304
Australia
Message 2033125 - Posted: 20 Feb 2020, 7:08:51 UTC - in response to Message 2033124.  
Last modified: 20 Feb 2020, 7:10:01 UTC

Forums are responding slightly better now than 20 minutes ago, but I'm getting nothing from the server...can't upload/download WUs
No problems here uploading (other than the usual), but still no response at all from the Scheduler.

Graphs show the return rate falling like a stone.
Grant
Darwin NT
ID: 2033125 · Report as offensive
Profile Unixchick Project Donor
Avatar

Send message
Joined: 5 Mar 12
Posts: 815
Credit: 2,361,516
RAC: 22
United States
Message 2033126 - Posted: 20 Feb 2020, 7:29:11 UTC
Last modified: 20 Feb 2020, 7:38:30 UTC

just got some WUs to download... is it fixed?

edit: I'll take a wild guess and say that it had to do with the magic 20 million results issue.
ID: 2033126 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13797
Credit: 208,696,464
RAC: 304
Australia
Message 2033127 - Posted: 20 Feb 2020, 7:37:45 UTC - in response to Message 2033126.  
Last modified: 20 Feb 2020, 7:40:54 UTC

just got some WUs to download... is it fixed?
Yep (at least for now).
The Scheduler is no longer MIA.


Although downloads are an issue on my Linux system (no download server setting in Hosts file)- instant timeouts. Luckily it only took a couple of dozen retries to get them to download.
Grant
Darwin NT
ID: 2033127 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13797
Credit: 208,696,464
RAC: 304
Australia
Message 2033143 - Posted: 20 Feb 2020, 10:49:12 UTC

Ready-to-send buffer is empty, and the splitters aren't splitting.
Looks like all the shorties & noise bombs have built the backlogs back up to the cutoff level again.
Grant
Darwin NT
ID: 2033143 · Report as offensive
W-K 666 Project Donor
Volunteer tester

Send message
Joined: 18 May 99
Posts: 19221
Credit: 40,757,560
RAC: 67
United Kingdom
Message 2033144 - Posted: 20 Feb 2020, 10:57:00 UTC - in response to Message 2033143.  

Ready-to-send buffer is empty, and the splitters aren't splitting.
Looks like all the shorties & noise bombs have built the backlogs back up to the cutoff level again.

Something must be happening as I just got 23 tasks at 10:54
ID: 2033144 · Report as offensive
juan BFP Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 16 Mar 07
Posts: 9786
Credit: 572,710,851
RAC: 3,799
Panama
Message 2033152 - Posted: 20 Feb 2020, 12:00:40 UTC

Lot of DL/UL error and retries. Do we have a problem?
ID: 2033152 · Report as offensive
Ville Saari
Avatar

Send message
Joined: 30 Nov 00
Posts: 1158
Credit: 49,177,052
RAC: 82,530
Finland
Message 2033161 - Posted: 20 Feb 2020, 14:09:34 UTC

I don't think the problem is in shorties or noise bombs but in the assimilator not assimilating. Assimilation queue has skyrocketed after the Tuesday outage and the results it is holding hostage have pushed the database size out of safe zone.
ID: 2033161 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14660
Credit: 200,643,578
RAC: 874
United Kingdom
Message 2033167 - Posted: 20 Feb 2020, 14:35:48 UTC - in response to Message 2033161.  

I don't think the problem is in shorties or noise bombs but in the assimilator not assimilating. Assimilation queue has skyrocketed after the Tuesday outage and the results it is holding hostage have pushed the database size out of safe zone.
The two are related. A lot of shorties and noise bombs reporting quickly after the end of the outage and in the hours that followed will have spiked both the validation queue, and the subsequent assimilation queue, to much higher levels than normal.

To use an illustration all too relevant in this country at the moment: the shorties are like a big thunderstorm in the hills. That creates a surge of water, which moves down the streams and rivers over the following days.
ID: 2033167 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 2033181 - Posted: 20 Feb 2020, 16:18:48 UTC - in response to Message 2032846.  
Last modified: 20 Feb 2020, 16:20:16 UTC

for i in `boinccmd --get_file_transfers | sed -n -e 's/^.*name: //p'`;do boinccmd --file_transfer http://setiathome.berkeley.edu $i retry;done


This command will give the boot to stuck transfers if you don’t want to run additional software.


I was having a heck of a time getting this to work in conjunction with the watch command, but it always seemed to give a syntax error or just didn't work properly. it works as-is if you run it standalone, just not with watch.

but it works if you just drop into a executable bash script

filename update_transfers
#!/bin/bash
for i in `./boinccmd --get_file_transfers | sed -n -e 's/^.*name: //p'`;do ./boinccmd --file_transfer http://setiathome.berkeley.edu $i retry;done

then run it
watch -n 60 ./update_transfers

Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 2033181 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2033213 - Posted: 20 Feb 2020, 21:02:00 UTC

Looks like we are bumping up against the memory limit again. Nothing but no work available for all hosts over the past 20 minutes.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2033213 · Report as offensive
Luc

Send message
Joined: 7 Jan 17
Posts: 4
Credit: 1,840,181
RAC: 5
Canada
Message 2033250 - Posted: 21 Feb 2020, 1:12:23 UTC

more of a whine / observation / OCD thing but why are there 5 'chunks' of data waiting to be processed since late 2019? would flushing everything out of the repositories have a potential cleansing affect? i'm no DB nor systems design guy, just wondering..... it has come close a few times only to have another coupla days of data pushed in front - like today.
ID: 2033250 · Report as offensive
Profile Buckeye4LF Project Donor
Avatar

Send message
Joined: 19 Jun 00
Posts: 173
Credit: 54,916,209
RAC: 833
United States
Message 2033261 - Posted: 21 Feb 2020, 2:36:48 UTC - in response to Message 2033181.  

Thanks, this bash file helps a lot

ID: 2033261 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13164
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2033266 - Posted: 21 Feb 2020, 3:19:07 UTC

Just can't get any or much replacement work. Splitters are throttled because the database is too big.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2033266 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13797
Credit: 208,696,464
RAC: 304
Australia
Message 2033267 - Posted: 21 Feb 2020, 3:24:39 UTC - in response to Message 2033266.  

Just can't get any or much replacement work. Splitters are throttled because the database is too big.
It's backlog is also heading for a new record high. The last time it got this bad we were pretty much out of work for a couple of days.
Grant
Darwin NT
ID: 2033267 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13797
Credit: 208,696,464
RAC: 304
Australia
Message 2033277 - Posted: 21 Feb 2020, 7:15:11 UTC

Forums laggy and Scheduler is MIA/taking ages to respond again. And the response is "Project has no tasks available" if it does respond, even though there are 60k ready-to-send.
Grant
Darwin NT
ID: 2033277 · Report as offensive
Previous · 1 . . . 84 · 85 · 86 · 87 · 88 · 89 · 90 . . . 94 · Next

Message boards : Number crunching : The Server Issues / Outages Thread - Panic Mode On! (118)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.