View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0037977 | FPC | Packages | public | 2020-10-24 05:57 | 2020-10-26 06:21 |
Reporter | Trevor Roydhouse | Assigned To | Michael Van Canneyt | ||
Priority | normal | Severity | minor | Reproducibility | always |
Status | resolved | Resolution | fixed | ||
Platform | x86_64 | OS | macOS | ||
Product Version | 3.2.0 | ||||
Fixed in Version | 3.3.1 | ||||
Summary | 0037977: MacOS: Unable to use more modern LibreSSL libraries for TLS 1.2/1.3 only sites | ||||
Description | The fix in https://bugs.freepascal.org/view.php?id=36484 breaks macOS ability to connect to web sites using HTTPS with TLSv1.2 or TLSv1.3 protocols only. The result of the fix is that while FPC 3.0.4 can connect to such sites successfully using the LibreSSL library (using the unversioned /usr/lib/libssl.dylib which is linked to /usr/lib/libssl.35.dylib -- there are also other LibreSSL library versions 43 and 44) FPC 3.2.0 (and trunk) fall back to using the OpenSSL library (specifically /usr/lib/openssl.0.9.8.dylib) which causes the failure. For a full discussion of the issue: see https://forum.lazarus.freepascal.org/index.php/topic,50350.0.html especially the last page. | ||||
Steps To Reproduce | unit Unit1; {$mode objfpc}{$H+} interface uses Classes, SysUtils, Forms, Controls, Graphics, Dialogs , fphttpclient, LCLIntf, LCLType, StdCtrls; //, OpenSSLSockets; -- needed for 3.3.1 type { TForm1 } TForm1 = class(TForm) Button1: TButton; procedure Button1Click(Sender: TObject); private public end; var Form1: TForm1; implementation {$R *.lfm} function GetWebPage(const URL: string): string; var Client: TFPHttpClient; begin Client := TFPHttpClient.Create(nil); Try Client.AllowRedirect := true; Client.AddHeader('User-Agent', 'Mozilla/5.0(compatible; fpweb)'); Result := Client.Get(URL); except on E: Exception do ShowMessage('Retrieval of: ' + URL + LineEnding + 'Failed with error: ' + E.Message + LineEnding + 'HTTP code: ' + IntToSTr(Client.ResponseStatusCode)+ LineEnding); end; end; { TForm1 } procedure TForm1.Button1Click(Sender: TObject); begin ShowMessage(GetWebPage('https://www.ariva.de/')); end; end. | ||||
Additional Information | FPC 3.2.0 reports using OpenSSL 0.9.8zh 14 January 2016 FPC 3.0.4 reports using LibreSSL 2.2.7 (The default unversioned /usr/lib/libssl.dylib or /usr/lib/libssl.35.dylib ). Reverting the "fix" for Bug ID 36484 enables FPC 3.3.0 (and trunk) to use the LibreSSL library and connections are again successful. However, the better fix is to: svn diff Index: openssl.pas =================================================================== --- openssl.pas (revision 45778) +++ openssl.pas (working copy) @@ -111,7 +111,7 @@ { ADD NEW ONES WHEN THEY APPEAR! Always make .so/dylib first, then versions, in descending order! Add "." .before the version, first is always just "" } - DLLVersions: array[1..19] of string = ('', '.1.1', '.11', '.10', '.1.0.6', '.1.0.5', '.1.0.4', '.1.0.3', + DLLVersions: array[1..24] of string = ('', '.46', '.45', '.44', '.43', '.35', '.1.1', '.11', '.10', '.1.0.6', '.1.0.5', '.1.0.4', '.1.0.3', '.1.0.2', '.1.0.1','.1.0.0','.0.9.8', '.0.9.7', '.0.9.6', '.0.9.5', '.0.9.4', '.0.9.3', '.0.9.2', '.0.9.1'); which allows the original fix to work. It will then find the highest of /usr/lib/libssl.[35|43|44|45|46].dylib etc. Beware that older versions of macOS only include old OpenSSL libraries (libssl.0.9.7.dylib and libssl.0.9.8.dylib ) and not LibreSSL . Also note that while OpenSSL's libssl and libcrypto appear to always use the same version, LibreSSL does not necessarily (eg libssl.46.dylib is paired with libcrypto.44.dylib for LibreSSL v2.8.3. ) which also means that -- at least for macOS -- the array also needs .38, .41 and .42. I don't know about Linux or OpenBSD. Here's macOS by version: 10.11 - OpenSSL 0.9.7, 0.9.8 - LibreSSL libssh.35 and libcrypto.35 10.12 - OpenSSL 0.9.7, 0.9.8 - LibreSSL libssh.35, libssh.39; libcrypto.35, libcrypto.38 10.13 - OpenSSL 0.9.7, 0.9.8 - LibreSSL libssh.35, libssh.43; libcrypto.35, libcrypto.41 10.14 - OpenSSL 0.9.7, 0.9.8 - LibreSSL libssh.35, libssh.43, libssl.44; libcrypto.35, libcrypto.41, libcrypto.42 10.15 - OpenSSL 0.9.7, 0.9.8 - LibreSSL libssh.35, libssh.43, libssl.44, libssl.46; libcrypto.35, libcrypto.41, libcrypto.42, libcrypto.44 11.0 beta 4 - OpenSSL 0.9.7, 0.9.8 - LibreSSL libssh.35, libssh.43, libssl.44, libssl.46; libcrypto.35, libcrypto.41, libcrypto.42, libcrypto.44 | ||||
Tags | MacOS, SSL | ||||
Fixed in Revision | 47174 | ||||
FPCOldBugId | |||||
FPCTarget | 3.2.2 | ||||
Attached Files |
|
|
If you create a patch, please create a proper patch for trunk. The one you sent is useless. The versions for openssl and libressl have been split in trunk. Your report is so long and full of unnecessary info, that I don't understand what the actual problem is. So please confirm: The problem is that for libressl the libssl and libcrypto dlls are not necessary using the same version ? |
|
When I test on mac with trunk, the version reported is LibreSSL 2.8.3, as expected, since I have .44 installed. It loads the page from your example correctly. |
|
Apologies. I only noticed your comment in the previous bug report to start a new report today while I was checking something for the forum. Add to that the fact that I hadn't updated my source since you added the separate LibreSSL array recently and, in the absence of any comment on the previous related bug, this is where we ended up. I have tested the current trunk tonight with macOS 10.14.6 Mojave and it fails to load the best (most recent) version of the LibreSSL libraries. After setting DYLD_PRINT_LIBRARIES and running the program, this is what happens: dyld: loaded: /usr/lib/libssl.44.dylib dyld: loaded: /usr/lib/libcrypto.42.dylib dyld: loaded: /System/Library/PrivateFrameworks/TrustEvaluationAgent.framework/Versions/A/TrustEvaluationAgent dyld: loaded: /usr/lib/libssl.43.dylib dyld: loaded: /usr/lib/libcrypto.41.dylib dyld: loaded: /usr/lib/libssl.35.dylib dyld: loaded: /usr/lib/libcrypto.35.dylib It ends up loading v35 (LibreSSL 2.2.7) instead of v44 (LibreSSL 2.6.5). This is because the version numbers of the libssl and libcrypto libraries are out of sync until v35. As you can see from the above, libssl.44 goes with libcrypto.42, and so on until v35 at which point the FPC code is happy because both library versions finally match. |
|
It works on Catalina. I changed openssl to print the pair being loaded, set DYLD_PRINT_LIBRARIES, here is the relevant output: Loading pair: libssl.48.dylib+libcrypto.48.dylib Loading pair: libssl.47.dylib+libcrypto.47.dylib Loading pair: libssl.46.dylib+libcrypto.46.dylib Loading pair: libssl.45.dylib+libcrypto.45.dylib Loading pair: libssl.44.dylib+libcrypto.44.dylib dyld: loaded: <CB6B188A-1ADC-3C5D-AFFC-2ACD86E235EE> /usr/lib/libcrypto.44.dylib dyld: loaded: <10E56F70-E234-31E0-9286-96D93A8ED17E> /System/Library/PrivateFrameworks/TrustEvaluationAgent.framework/Versions/A/TrustEvaluationAgent dyld: loaded: <5DB7F9FC-031E-3F63-869E-60D8826723A9> /usr/lib/libssl.44.dylib dyld: loaded: <C8AC7DA9-4DDF-3E05-85DC-F4D434325235> /usr/lib/libcrypto.42.dylib So it works on MacOS Catalina, and I am inclined to think it is not an error in the algorithm. Remains the question why it does not work on your system. |
|
I just tried it on a new install of Mojave 10.14.6 in a VM (as opposed to real hardware) with the same result. When I tried it with Catalina, I get: dyld: loaded: <CB6B188A-1ADC-3C5D-AFFC-2ACD86E235EE> /usr/lib/libcrypto.44.dylib dyld: loaded: <10E56F70-E234-31E0-9286-96D93A8ED17E> /System/Library/PrivateFrameworks/TrustEvaluationAgent.framework/Versions/A/TrustEvaluationAgent dyld: loaded: <5DB7F9FC-031E-3F63-869E-60D8826723A9> /usr/lib/libssl.44.dylib dyld: loaded: <C8AC7DA9-4DDF-3E05-85DC-F4D434325235> /usr/lib/libcrypto.42.dylib So it used LibreSSL 2.6.5 (ssl 44 and crypto 42 - same as you) instead of LibreSSL 2.8.3 (ssl 46 and crypto 44) which is again not the wanted result. |
|
I don't agree that this is not the wanted result. There is simply no telling what the desired result is: Clearly, Apple does some tricks behind the scenes. Look at my sample output: I am asking to load version .44, and it loads .42 but all seems well from the application point of view, as it has actually loaded .44 as far as it can tell. Why it does not want to load 46/46 (with possible fallback to 44) is unknown, and I don't think it has anything to do with the FPC algorithm, which just tries to load matching versions. I do not intend to work around or cope with Apple's quircks in library versions, who knows what will happen in future versions. So, to remedy the situation and provide people with an algorithm which can cope with all eventualities, I have added an overloaded version of InitSSLInterface in which you can name the library pair to be loaded, so you can completely control how the library is loaded & initialized. |
|
I add sequential version numbers to the LibreSSL array 48-35 because LibreSSL is used in other than macOS who knows what versions other OS (eg Linux) are using. I then added "Trying..." and "Unloading..." in TryLoadLibPair() and reversed the order of loading the libraries so that: SSLUtilHandle := LoadLibrary(Crypto_DLL_Name); if (SSLUtilHandle <> 0) then SSLLibHandle := LoadLibrary(SSL_DLL_Name); became: SSLLibHandle := LoadLibrary(SSL_DLL_Name); if (SSLLibHandle <> 0) then SSLUtilHandle := LoadLibrary(Crypto_DLL_Name); and I think the new output below demonstrates the issue on macOS 10.14.6 and 10.15.6:: First 10.14.6: Trying libssl.48.dylib Trying libcrypto.48.dylib Unloading libssl.48.dylib Unloading libcrypto.48.dylib Trying libssl.47.dylib Trying libcrypto.47.dylib Unloading libssl.47.dylib Unloading libcrypto.47.dylib Trying libssl.46.dylib Trying libcrypto.46.dylib Unloading libssl.46.dylib Unloading libcrypto.46.dylib Trying libssl.45.dylib Trying libcrypto.45.dylib Unloading libssl.45.dylib Unloading libcrypto.45.dylib Trying libssl.44.dylib Trying libcrypto.44.dylib >> dyld: loaded: /usr/lib/libssl.44.dylib >> dyld: loaded: /usr/lib/libcrypto.42.dylib dyld: loaded: /System/Library/PrivateFrameworks/TrustEvaluationAgent.framework/Versions/A/TrustEvaluationAgent Unloading libssl.44.dylib Unloading libcrypto.44.dylib Trying libssl.43.dylib Trying libcrypto.43.dylib dyld: loaded: /usr/lib/libssl.43.dylib dyld: loaded: /usr/lib/libcrypto.41.dylib Unloading libssl.43.dylib Unloading libcrypto.43.dylib Trying libssl.42.dylib Trying libcrypto.42.dylib Unloading libssl.42.dylib Unloading libcrypto.42.dylib Trying libssl.41.dylib Trying libcrypto.41.dylib Unloading libssl.41.dylib Unloading libcrypto.41.dylib Trying libssl.40.dylib Trying libcrypto.40.dylib Unloading libssl.40.dylib Unloading libcrypto.40.dylib Trying libssl.39.dylib Trying libcrypto.39.dylib Unloading libssl.39.dylib Unloading libcrypto.39.dylib Trying libssl.38.dylib Trying libcrypto.38.dylib Unloading libssl.38.dylib Unloading libcrypto.38.dylib Trying libssl.37.dylib Trying libcrypto.37.dylib Unloading libssl.37.dylib Unloading libcrypto.37.dylib Trying libssl.36.dylib Trying libcrypto.36.dylib Unloading libssl.36.dylib Unloading libcrypto.36.dylib Trying libssl.35.dylib Trying libcrypto.35.dylib >> dyld: loaded: /usr/lib/libssl.35.dylib >> dyld: loaded: /usr/lib/libcrypto.35.dylib 10.15.6 Trying libssl.48.dylib Trying libcrypto.48.dylib Unloading libssl.48.dylib Unloading libcrypto.48.dylib Trying libssl.47.dylib Trying libcrypto.47.dylib Unloading libssl.47.dylib Unloading libcrypto.47.dylib Trying libssl.46.dylib Trying libcrypto.46.dylib >> dyld: loaded: <06932872-13DA-33E3-8C28-7B49FC582039> /usr/lib/libssl.46.dylib >> dyld: loaded: <CB6B188A-1ADC-3C5D-AFFC-2ACD86E235EE> /usr/lib/libcrypto.44.dylib dyld: loaded: <10E56F70-E234-31E0-9286-96D93A8ED17E> /System/Library/PrivateFrameworks/TrustEvaluationAgent.framework/Versions/A/TrustEvaluationAgent Unloading libssl.46.dylib Unloading libcrypto.46.dylib Trying libssl.45.dylib Trying libcrypto.45.dylib Unloading libssl.45.dylib Unloading libcrypto.45.dylib Trying libssl.44.dylib Trying libcrypto.44.dylib >> dyld: loaded: <5DB7F9FC-031E-3F63-869E-60D8826723A9> /usr/lib/libssl.44.dylib >> dyld: loaded: <C8AC7DA9-4DDF-3E05-85DC-F4D434325235> /usr/lib/libcrypto.42.dylib It looks to me like the first library load is correct (ie the highest version number) and that libssl itself loads the correct libcrypto, but the current openssl.pas code then unloads libssl because there is no matching libcrypto with the same version. I'm not sure Apple is doing anything behind the scenes. |
|
Yes indeed, otool -l on the libssl.44 library reveals: Load command 11 cmd LC_LOAD_DYLIB cmdsize 56 name /usr/lib/libcrypto.42.dylib (offset 24) time stamp 2 Thu Jan 1 10:00:02 1970 current version 43.0.0 compatibility version 43.0.0 |
|
I agree with your conclusion. The dynamic loader probably loads the correct version. I dug a little deeper, and it seems the libraries are distributed by the libressl team with mismatching versions, also on linux. The short term solution is then to split the libssl/crypto version arrays with matching versions. I have 46-44, 44-42, 43-41, 35-35 It seems you have access to several MacOS versions, any additional list of matching versions you can provide is most welcome. In the long term, we'll have to split libressl from openssl. The openbsd team have added a tls layer (much like gnutls provides) that allows to set up a TLS connection in a safer way, and that should be used instead of the low-level functions. |
|
found meanwhile 48-46,47-45,45-43,39-38,38-37,37-36,34-33,33-32,32-32 |
|
Here's the list from 10.1 through 11 beta 10: 10.8, 10.9, 10.10 libssl.0.9.7.dylib libssl.0.9.8.dylib 10.11 libssl.0.9.7.dylib libssl.0.9.8.dylib libssl.35.dylib - libcrypto.35.dylib 10.12 libssl.0.9.7.dylib libssl.0.9.8.dylib libssl.35.dylib - libcrypto.35.dylib libssl.39.dylib - libcrypto.38.dylib 10.13 libssl.0.9.7.dylib libssl.0.9.8.dylib libssl.35.dylib - libcrypto.35.dylib libssl.39.dylib - libcrypto.38.dylib libssl.43.dylib - libcrypto.41.dylib 10.14 libssl.0.9.7.dylib libssl.0.9.8.dylib libssl.35.dylib - libcrypto.35.dylib libssl.43.dylib - libcrypto.41.dylib libssl.44.dylib - libcrypto.42.dylib 10.15 libssl.0.9.7.dylib libssl.0.9.8.dylib libssl.35.dylib - libcrypto.35.dylib libssl.43.dylib - libcrypto.41.dylib libssl.44.dylib - libcrypto.42.dylib libssl.46.dylib - libcrypto.44.dylib 11.0 (beta 10) libssl.35.dylib - libcrypto.35.dylib libssl.43.dylib - libcrypto.41.dylib libssl.44.dylib - libcrypto.42.dylib libssl.46.dylib - libcrypto.44.dylib |
Date Modified | Username | Field | Change |
---|---|---|---|
2020-10-24 05:57 | Trevor Roydhouse | New Issue | |
2020-10-24 05:57 | Trevor Roydhouse | Issue generated from: 0036484 | |
2020-10-24 05:57 | Trevor Roydhouse | Tag Attached: MacOS | |
2020-10-24 05:57 | Trevor Roydhouse | Tag Attached: SSL | |
2020-10-24 10:10 | Michael Van Canneyt | Assigned To | => Michael Van Canneyt |
2020-10-24 10:10 | Michael Van Canneyt | Status | new => assigned |
2020-10-24 10:15 | Michael Van Canneyt | Status | assigned => feedback |
2020-10-24 10:15 | Michael Van Canneyt | FPCTarget | => - |
2020-10-24 10:15 | Michael Van Canneyt | Note Added: 0126502 | |
2020-10-24 10:54 | Michael Van Canneyt | Note Added: 0126504 | |
2020-10-24 15:42 | Trevor Roydhouse | Note Added: 0126512 | |
2020-10-24 15:42 | Trevor Roydhouse | Status | feedback => assigned |
2020-10-24 16:24 | Michael Van Canneyt | Note Added: 0126515 | |
2020-10-24 17:26 | Trevor Roydhouse | Note Added: 0126516 | |
2020-10-24 18:32 | Michael Van Canneyt | Status | assigned => resolved |
2020-10-24 18:32 | Michael Van Canneyt | Resolution | open => fixed |
2020-10-24 18:32 | Michael Van Canneyt | Fixed in Version | => 3.3.1 |
2020-10-24 18:32 | Michael Van Canneyt | Fixed in Revision | => 47174 |
2020-10-24 18:32 | Michael Van Canneyt | FPCTarget | - => 3.2.2 |
2020-10-24 18:32 | Michael Van Canneyt | Note Added: 0126517 | |
2020-10-24 18:42 | Michael Van Canneyt | Note Edited: 0126517 | View Revisions |
2020-10-25 02:15 | Trevor Roydhouse | Note Added: 0126528 | |
2020-10-25 05:49 | Trevor Roydhouse | Note Added: 0126530 | |
2020-10-25 10:45 | Michael Van Canneyt | Note Added: 0126536 | |
2020-10-25 15:04 | Michael Van Canneyt | Note Added: 0126545 | |
2020-10-25 15:11 | Michael Van Canneyt | Note Edited: 0126545 | View Revisions |
2020-10-26 06:21 | Trevor Roydhouse | Note Added: 0126561 |