The Server Issues / Outages Thread - Panic Mode On! (117)

Message boards : Number crunching : The Server Issues / Outages Thread - Panic Mode On! (117)
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 11 · 12 · 13 · 14 · 15 · 16 · 17 . . . 52 · Next

AuthorMessage
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 2012292 - Posted: 17 Sep 2019, 7:48:50 UTC - in response to Message 2012290.  

So now what?
I had a problem very similar to this on my Mint hosts. That problem went away when I installed the libnsspem library.
sudo apt-get install libnsspem
As I recall it, the http_debug output in my case complained that it couldn't find libnsspem.so, so it may not have been exactly the same as your problem...

Tried that just now, as well. No change.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 2012292 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 2012293 - Posted: 17 Sep 2019, 8:02:59 UTC - in response to Message 2012289.  

I don't know if it means anything, but in my googling, I found a boinc forum post from 2009 where someone was asking about this very thing for WCG, and one of the devs replied and said that the cert WCG relies-on is good until "2019".

Maybe we've reached the point in "2019" for that particular cert, perhaps? I know we're not WCG, but it seems to line up.
BOINC grasped that particular nettle in January 2018, and updated the certificate bundle ('ca-bundle.crt') sent to Windows users - and, I think, Android users too. It was a developer from WCG who took the lead on #2326, so I don't think we're expecting any equivalent of the millenium bug in 2019.

But that doesn't solve the problem of bringing Linux system-level certificates up to 2019 standards.
ID: 2012293 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 2012294 - Posted: 17 Sep 2019, 8:19:12 UTC - in response to Message 2012293.  
Last modified: 17 Sep 2019, 8:19:33 UTC

BOINC grasped that particular nettle in January 2018, and updated the certificate bundle ('ca-bundle.crt') sent to Windows users - and, I think, Android users too. It was a developer from WCG who took the lead on #2326, so I don't think we're expecting any equivalent of the millenium bug in 2019.

But that doesn't solve the problem of bringing Linux system-level certificates up to 2019 standards.

Well I copied the text from https://github.com/BOINC/boinc/blob/master/curl/ca-bundle.crt and saved it as ca-bundle.crt and dropped it into the data directory on the linux machine. Same error:

[http_debug] [ID#1] Info: error:0D0C50A1:asn1 encoding routines:ASN1_item_verify:unknown message digest algorithm

What else can I do/try?
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 2012294 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 2012295 - Posted: 17 Sep 2019, 8:35:47 UTC - in response to Message 2012294.  

Well I copied the text from https://github.com/BOINC/boinc/blob/master/curl/ca-bundle.crt and saved it as ca-bundle.crt and dropped it into the data directory on the linux machine. Same error:

[http_debug] [ID#1] Info: error:0D0C50A1:asn1 encoding routines:ASN1_item_verify:unknown message digest algorithm

What else can I do/try?
I can't help with a magic script, but I can try to help narrow down the problem.

ca-bundle.crt is actually - I was surprised to find out - a plain text file. You can open it in any text editor and read the description, and the names of each included certificate. It starts:

## Certificate data from Mozilla as of: Fri Jan 26 21:30:21 2018 GMT
##
## This is a bundle of X.509 certificates of public Certificate Authorities
## (CA). These were automatically extracted from Mozilla's root certificates
## file (certdata.txt). This file can be found in the mozilla source tree:
## https://hg.mozilla.org/releases/mozilla-release/raw-file/default/security/nss/lib/ckfw/builtins/certdata.txt
##
## It contains the certificates in PEM format and therefore
## can be directly used with curl / libcurl / php_curl, or with
## an Apache+mod_ssl webserver for SSL client authentication.
## Just configure this file as the SSLCACertificateFile.
##
## Conversion done with mk-ca-bundle.pl version 1.27.
## SHA256: a3ac15b98179dd2f3c5de076d10b1d53048754372f7207c2f327510cdd78fbd8
The individual certificates themselves are, of course, encrypted. It rather sounds as if Cosmic_Ocean's problem doesn't lie with the certificates themselves, but with the tool needed to decrypt them.
ID: 2012295 · Report as offensive
Oddbjornik Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 15 May 99
Posts: 220
Credit: 349,610,548
RAC: 1,728
Norway
Message 2012329 - Posted: 17 Sep 2019, 19:18:12 UTC - in response to Message 2012295.  

The individual certificates themselves are, of course, encrypted. It rather sounds as if Cosmic_Ocean's problem doesn't lie with the certificates themselves, but with the tool needed to decrypt them.
He runs Boinc version 6.10.58, that may just be too old...?
ID: 2012329 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 2012330 - Posted: 17 Sep 2019, 19:20:59 UTC - in response to Message 2012329.  

He runs Boinc version 6.10.58, that may just be too old...?
I'm sure I've advised users of older BOINCs to update certificate bundles, and it's worked with the new files. This one is different.
ID: 2012330 · Report as offensive
Oddbjornik Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 15 May 99
Posts: 220
Credit: 349,610,548
RAC: 1,728
Norway
Message 2012331 - Posted: 17 Sep 2019, 19:27:12 UTC - in response to Message 2012330.  
Last modified: 17 Sep 2019, 19:29:35 UTC

He runs Boinc version 6.10.58, that may just be too old...?
I'm sure I've advised users of older BOINCs to update certificate bundles, and it's worked with the new files. This one is different.
Yes, I've looked around, and it seems the standard fix for the ASN1_item_verify error is to update openssl to version 0.9.8o.
Have you tried that, Cosmic?

[Edit: I checked on one of my Mint hosts, and 'openssl version' says 1.1.1]
ID: 2012331 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 2012332 - Posted: 17 Sep 2019, 19:46:17 UTC - in response to Message 2012331.  
Last modified: 17 Sep 2019, 19:48:09 UTC

He runs Boinc version 6.10.58, that may just be too old...?
I'm sure I've advised users of older BOINCs to update certificate bundles, and it's worked with the new files. This one is different.
Yes, I've looked around, and it seems the standard fix for the ASN1_item_verify error is to update openssl to version 0.9.8o.
Have you tried that, Cosmic?

[Edit: I checked on one of my Mint hosts, and 'openssl version' says 1.1.1]

root@taurus:~# openssl version
OpenSSL 1.1.1  11 Sep 2018
root@taurus:~#


but... this line from the startup of boinc:

2019-09-17 04:15:58 Libraries: libcurl/7.18.0 OpenSSL/0.9.8g zlib/1.2.11 c-ares/1.5.1

So how do I get boinc to use openssl/1.1.1 instead? I remember getting a Windows client to use a newer version of curl before by doing something, but I don't remember what I did to get it to do that.


also, for reference..

2019-09-17 04:15:58 Starting BOINC client version 6.10.58 for x86_64-pc-linux-gnu
2019-09-17 04:15:58 Data directory: /var/lib/boinc-client
2019-09-17 04:15:58 OS: Linux: 4.15.0-62-generic
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 2012332 · Report as offensive
Oddbjornik Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer tester
Avatar

Send message
Joined: 15 May 99
Posts: 220
Credit: 349,610,548
RAC: 1,728
Norway
Message 2012333 - Posted: 17 Sep 2019, 19:58:34 UTC - in response to Message 2012332.  

but... this line from the startup of boinc:

2019-09-17 04:15:58 Libraries: libcurl/7.18.0 OpenSSL/0.9.8g zlib/1.2.11 c-ares/1.5.1

So how do I get boinc to use openssl/1.1.1 instead? I remember getting a Windows client to use a newer version of curl before by doing something, but I don't remember what I did to get it to do that.

So there's the core of the problem. Boinc uses 0.9.8g where it should use something newer. Now all we need is a good solution!
ID: 2012333 · Report as offensive
Cosmic_Ocean
Avatar

Send message
Joined: 23 Dec 00
Posts: 3027
Credit: 13,516,867
RAC: 13
United States
Message 2012334 - Posted: 17 Sep 2019, 20:10:01 UTC - in response to Message 2012332.  

So how do I get boinc to use openssl/1.1.1 instead? I remember getting a Windows client to use a newer version of curl before by doing something, but I don't remember what I did to get it to do that.

I think I remember how I did it with Windows.. in c:\program files\boinc (where the BOINC software itself is installed), there are DLLs for those libraries.

But in linux.. the core software is in the data directory where all the xml files are and it relies on system variables and references for the libraries.

So.. I need to try to figure out why it is falling-back to 0.9.8g and fix that reference, I guess.
Linux laptop:
record uptime: 1511d 20h 19m (ended due to the power brick giving-up)
ID: 2012334 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 2012401 - Posted: 18 Sep 2019, 11:05:54 UTC
Last modified: 18 Sep 2019, 11:06:14 UTC

Not a good sign, on the server status page
Database/file status

Warning: number_format() expects parameter 1 to be double, string given in /disks/carolyn/b/home/boincadm/projects/sah/html/seti_boinc_html/sah_status.php on line 604 Warning: number_format() expects parameter 1 to be double, string given in /disks/carolyn/b/home/boincadm/projects/sah/html/seti_boinc_html/sah_status.php on line 606 Warning: number_format() expects parameter 1 to be double, string given in /disks/carolyn/b/home/boincadm/projects/sah/html/seti_boinc_html/sah_status.php on line 608
And the "Results received in last hour " line is blank.
Grant
Darwin NT
ID: 2012401 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 2012402 - Posted: 18 Sep 2019, 11:09:00 UTC - in response to Message 2012401.  

And the "Results received in last hour " line is blank.
As are the result turnround times, as a result of the number formatting failures in the warning messages.
ID: 2012402 · Report as offensive
Grant (SSSF)
Volunteer tester

Send message
Joined: 19 Aug 99
Posts: 13736
Credit: 208,696,464
RAC: 304
Australia
Message 2012403 - Posted: 18 Sep 2019, 11:32:11 UTC

And now it's been posted about, it's Ok again.
Grant
Darwin NT
ID: 2012403 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 2012404 - Posted: 18 Sep 2019, 11:38:28 UTC - in response to Message 2012403.  

It seems that the phantom tinkerer has been tinkering again. The BOINC website (housed in the same rack cabinets as the SETI servers) has been intermittently 'down for maintenance' since about midnight, Berkeley time - affecting only the database of messages, not downloads, wiki pages, etc.
ID: 2012404 · Report as offensive
rob smith Crowdfunding Project Donor*Special Project $75 donorSpecial Project $250 donor
Volunteer moderator
Volunteer tester

Send message
Joined: 7 Mar 03
Posts: 22205
Credit: 416,307,556
RAC: 380
United Kingdom
Message 2012414 - Posted: 18 Sep 2019, 13:22:26 UTC

The "randomness" of the BOINC message board must be perfectly synchronised with me trying to access the BOINC message boards as they have been saying "down for maintenance" since I first looked at about 07:00 (BST) today.
The Phantom Phiddler must has done his/her thing then gone home for the night without checking the thing had actually worked, or rolling back to a known working state.
Bob Smith
Member of Seti PIPPS (Pluto is a Planet Protest Society)
Somewhere in the (un)known Universe?
ID: 2012414 · Report as offensive
Profile Jord
Volunteer tester
Avatar

Send message
Joined: 9 Jun 99
Posts: 15184
Credit: 4,362,181
RAC: 3
Netherlands
Message 2012417 - Posted: 18 Sep 2019, 13:38:50 UTC - in response to Message 2012414.  

Since no one appears to have his email address, I have emailed him from my Eype beach. Let's see if we get a reaction.
ID: 2012417 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 2012418 - Posted: 18 Sep 2019, 14:07:13 UTC - in response to Message 2012417.  

It did come back online briefly somewhere around 3am PST, because I retrieved a PM sent at 0:33:53 UTC (but still no email notification). The PM answered a question I had asked on the open message board, but the sender said he was using a PM to reply because "Tried to answer your question but the program did not allow". I'm being very careful not to refresh the PM page because I need to tie back that answer with the original post which prompted my question.
ID: 2012418 · Report as offensive
Richard Haselgrove Project Donor
Volunteer tester

Send message
Joined: 4 Jul 99
Posts: 14650
Credit: 200,643,578
RAC: 874
United Kingdom
Message 2012432 - Posted: 18 Sep 2019, 15:58:53 UTC - in response to Message 2012418.  

For PST, read PDT.
ID: 2012432 · Report as offensive
Dr Who Fan
Volunteer tester
Avatar

Send message
Joined: 8 Jan 01
Posts: 3214
Credit: 715,342
RAC: 4
United States
Message 2012449 - Posted: 18 Sep 2019, 19:59:46 UTC

BOINC message boards still "down for maintenance"... Guess there has been too much hair burning and howling over there that the servers have come to an early death.
ID: 2012449 · Report as offensive
Profile betreger Project Donor
Avatar

Send message
Joined: 29 Jun 99
Posts: 11361
Credit: 29,581,041
RAC: 66
United States
Message 2012454 - Posted: 18 Sep 2019, 20:47:26 UTC - in response to Message 2012449.  

BOINC message boards still "down for maintenance"... Guess there has been too much hair burning and howling over there that the servers have come to an early death.

What else would you suggest for us to do in order to bring it back?
ID: 2012454 · Report as offensive
Previous · 1 . . . 11 · 12 · 13 · 14 · 15 · 16 · 17 . . . 52 · Next

Message boards : Number crunching : The Server Issues / Outages Thread - Panic Mode On! (117)


 
©2024 University of California
 
SETI@home and Astropulse are funded by grants from the National Science Foundation, NASA, and donations from SETI@home volunteers. AstroPulse is funded in part by the NSF through grant AST-0307956.