The Server Issues / Outages Thread - Panic Mode On! (117)

Message boards : Number crunching : The Server Issues / Outages Thread - Panic Mode On! (117)
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 27 · 28 · 29 · 30 · 31 · 32 · 33 . . . 52 · Next

AuthorMessage
Cherokee150

Send message
Joined: 11 Nov 99
Posts: 192
Credit: 58,513,758
RAC: 74
United States
Message 2018879 - Posted: 14 Nov 2019, 1:32:18 UTC

Has anybody noticed that the Server Status clock is exactly 24 hours fast?

I point this out because I do not know how the server it runs on, and any other software running on that server, interface with SETI and, perhaps, even BOINC itself. However, as a DT person, I know that a clock error like this has the potential of causing serious problems, not only with the proper triggering of tasks, but even serious data corruption. An error like this can be especially serious in scientific research, where incorrect timestamps can destroy the credibility and validity of data.

I hope this is just a minor problem. Does anyone know the exact interface of this server with everything else? If so, please evaluate the possible impact. If no one knows for sure, then I recommend someone with immediate access to the staff notify them as soon as possible in case this situation is serious.

Thanks.
ID: 2018879 · Report as offensive
Ian&Steve C.
Avatar

Send message
Joined: 28 Sep 99
Posts: 4267
Credit: 1,282,604,591
RAC: 6,640
United States
Message 2018886 - Posted: 14 Nov 2019, 2:37:01 UTC - in response to Message 2018879.  

Which clock do you mean?

When I look at the Server Status page, at the top it says “ As of 14 Nov 2019, 2:20:05 UTC”. Which was about 15 minutes ago.
Seti@Home classic workunits: 29,492 CPU time: 134,419 hours

ID: 2018886 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13161
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2018887 - Posted: 14 Nov 2019, 2:37:11 UTC - in response to Message 2018879.  

The server status page shows UTC time. It is correct.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2018887 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14649
Credit: 200,643,578
RAC: 874
United Kingdom
Message 2018938 - Posted: 14 Nov 2019, 14:20:30 UTC

The SETI servers are all individually synced with an NTP server. I know that, because I once noticed that one of them started drifting, and claiming that messages here had been posted at some time in the future. I pointed it out to Matt Lebofsky, who was running things at the time: he acknowledged that he'd forgotten that particular configuration on a new server, corrected it, and I've never seen any time error since then.
ID: 2018938 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 34744
Credit: 261,360,520
RAC: 489
Australia
Message 2019011 - Posted: 14 Nov 2019, 21:52:59 UTC

I don't know if it's time for some to panic, but the replica database has been falling behind master for the last hour or so and is now 1,515 sec's behind the master.

Cheers.
ID: 2019011 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 34744
Credit: 261,360,520
RAC: 489
Australia
Message 2019034 - Posted: 14 Nov 2019, 23:15:42 UTC - in response to Message 2019011.  

I don't know if it's time for some to panic, but the replica database has been falling behind master for the last hour or so and is now 1,515 sec's behind the master.
It's slowly recovering now, down to 753sec's behind.

Cheers.
ID: 2019034 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13722
Credit: 208,696,464
RAC: 304
Australia
Message 2019216 - Posted: 16 Nov 2019, 3:03:09 UTC

Has anyone heard any more on the new upload server?

Over the last week or so I've noticed the odd upload sitting there counting down for a retry. So far it's just the odd one, and it always goes through on the second attempt, but prior to this there haven't been any such issues.
Grant
Darwin NT
ID: 2019216 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13161
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2019222 - Posted: 16 Nov 2019, 3:46:27 UTC - in response to Message 2019216.  

Has anyone heard any more on the new upload server?

Over the last week or so I've noticed the odd upload sitting there counting down for a retry. So far it's just the odd one, and it always goes through on the second attempt, but prior to this there haven't been any such issues.


I haven't heard anything other than I think it is in use at Beta. I am constantly plagued with a couple of stuck downloads on every computer every day. I have to use the Downloads tab in BT to notice them and then go around and do the Suspend Network Activity trick to get them to download. Downloads are going on all the time but instead of having 8 active download threads running at each connection, each stalled download for however many hours that are sitting there running up the counter, steal away from the download bandwidth for the rest of the downloads on the machine. A nuisance.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2019222 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13722
Credit: 208,696,464
RAC: 304
Australia
Message 2019224 - Posted: 16 Nov 2019, 4:30:56 UTC - in response to Message 2019222.  

I am constantly plagued with a couple of stuck downloads on every computer every day.
Have you got the download server set in your Hosts file?
I noticed after the weekly outage on my recently converted Linux system that the downloads were incredibly slow to start downloading, then actually download, but no issues on my Windows system which still has the download server set from the last time the download servers had major issues.
Grant
Darwin NT
ID: 2019224 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13161
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2019225 - Posted: 16 Nov 2019, 4:46:27 UTC - in response to Message 2019224.  

No, don't have the servers in my hosts file. I thought that caused more issues than solved.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2019225 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13722
Credit: 208,696,464
RAC: 304
Australia
Message 2019226 - Posted: 16 Nov 2019, 5:17:18 UTC - in response to Message 2019225.  

No, don't have the servers in my hosts file. I thought that caused more issues than solved.
If you're spending a fair chunk of your day sorting out download issues, it might be worth giving it a go.

My Win10 system Seti hosts entry
# 208.68.240.118 setiboincdata.ssl.berkeley.edu # upload server Oct 2016
208.68.240.119 boinc2.ssl.berkeley.edu # Georgem download server Oct 2016
# 208.68.240.126 setiboinc.ssl.berkeley.edu # scheduler Oct 2016
# 208.68.240.127 vader.ssl.berkeley.edu # Vader download server Oct 2016 

Grant
Darwin NT
ID: 2019226 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13161
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2019228 - Posted: 16 Nov 2019, 6:24:45 UTC - in response to Message 2019226.  

Yes, I have a text file with the hosts snippet saved for the servers that I can paste into the hosts file when necessary. But I really only have to unstick tasks once a day, once I notice them. Only time the counter gets up to a day or more is when I forget to check the Downloads tab. Then a few steps to each computer to unstick the stalled downloads. Only takes a minute or so. I could throw Georgem back into the hosts file and see if anything changes.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2019228 · Report as offensive
Profile Jimbocous Project Donor
Volunteer tester
Avatar

Send message
Joined: 1 Apr 13
Posts: 1849
Credit: 268,616,081
RAC: 1,349
United States
Message 2019240 - Posted: 16 Nov 2019, 10:01:21 UTC

fwiw, haven't seen any stuck downloads here other than during outage recovery, and definitely haven't done anything with a hosts file on either Linux box here (don't even recall where it might live).
ID: 2019240 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13161
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2019275 - Posted: 16 Nov 2019, 16:45:09 UTC - in response to Message 2019240.  
Last modified: 16 Nov 2019, 16:45:49 UTC

fwiw, haven't seen any stuck downloads here other than during outage recovery, and definitely haven't done anything with a hosts file on either Linux box here (don't even recall where it might live).

The last stuck download looks like it may have coincided with a reboot of the host to install a new kernel after a month of uptime and ignorance. MIght have caught the downloads mid-download or something. The timer count in hours matched closely with when I rebooted. But I have had other instances where nothing has happened on a host and I still get stuck downloads that never start and just keep running the count timer.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2019275 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13722
Credit: 208,696,464
RAC: 304
Australia
Message 2019314 - Posted: 16 Nov 2019, 23:42:13 UTC

Just spotted a stuck download on an AP WU on my Windows system (so set to use the better download server). It was about halfway done, just sitting there.
Disabled & re-enabled networking, and it finished off at around 1MB/s.

Still getting the odd upload not starting on it's first attempt, no problems on it's first retry.
Grant
Darwin NT
ID: 2019314 · Report as offensive
Profile Keith Myers Special Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 29 Apr 01
Posts: 13161
Credit: 1,160,866,277
RAC: 1,873
United States
Message 2019319 - Posted: 17 Nov 2019, 0:04:29 UTC - in response to Message 2019314.  

Most of mine never even get one byte downloaded, just the timer counting.
Seti@Home classic workunits:20,676 CPU time:74,226 hours

A proud member of the OFA (Old Farts Association)
ID: 2019319 · Report as offensive
Sirius B Project Donor
Volunteer tester
Avatar

Send message
Joined: 26 Dec 00
Posts: 24877
Credit: 3,081,182
RAC: 7
Ireland
Message 2019647 - Posted: 19 Nov 2019, 16:31:16 UTC

Hmm, that was a short one.
ID: 2019647 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13722
Credit: 208,696,464
RAC: 304
Australia
Message 2020297 - Posted: 23 Nov 2019, 21:57:09 UTC

For the last few weeks the Seti servers have been somewhat lethargic- always behind with their cleaning up, but not enough to have a noticeable effect.
But now both the MB & AP deleters appear to have given up, and the splitter output has fallen away & the Ready-to-send buffer has been emptying out. Luckily with all the Arecibo AP & VLAR work and generally longer running GBT WUs the return rate has been low enough that it hasn't gotten to the point of running out completely.
Hopefully the deleters will get back on the job, and the splitters return to full capabilities before that does occur.
Grant
Darwin NT
ID: 2020297 · Report as offensive
Profile Wiggo
Avatar

Send message
Joined: 24 Jan 00
Posts: 34744
Credit: 261,360,520
RAC: 489
Australia
Message 2020369 - Posted: 24 Nov 2019, 8:51:54 UTC

Is it just me or are downloads just slow ATM?

Cheers.
ID: 2020369 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13722
Credit: 208,696,464
RAC: 304
Australia
Message 2020371 - Posted: 24 Nov 2019, 8:56:16 UTC - in response to Message 2020369.  

Is it just me or are downloads just slow ATM?
Good speeds on my Windows machine- but it has the download server set in the Hosts file. On the Linux system with no Hosts entries, the download speed is very variable- some download straight away at high speed, others can take a while to even start & are slow to download once they finally get going.
Grant
Darwin NT
ID: 2020371 · Report as offensive
Previous · 1 . . . 27 · 28 · 29 · 30 · 31 · 32 · 33 . . . 52 · Next

Message boards : Number crunching : The Server Issues / Outages Thread - Panic Mode On! (117)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.