Why switch to LibreSSL (was: BoringSSL)?

[jay]

Why switch the default SSL lib to BoringSSL, I would think we'd want to keep OpenSSL the default because like 90% of curl users or something (I forget the exact stats) use it. Is it really that similar to OpenSSL? The path vulnerability you cite was fixed long ago wasn't it?

[vszakats]

No, the vulnerability was never fixed. It was merely mitigated by moving the settings under C:\Program Files\ (from /usr/local/ and similar paths that were used before by default), which remains vulnerable on all localized Windows versions that use a different folder name. I could mitigate it more by moving the settings under C:\Windows\System32\, but even this is not safe as the OS might be installed in a different directory. Even for this I must patch OpenSSL's Configure, because there is a cross-build bug which prevents overriding the settings directory, saying that the directory isn't absolute when it is. For this bug I provided a patch, which they refused to merge (asking for personal data to sign another CLA, because someone deemed the two-liner fix, that I finally contributed myself, as "non-trivial"). They also refuse to fix the vulnerability itself because "they do not provide Windows binaries".

The correct fix is to either use a runtime-detected Windows OS location as a base directory for its settings, or even better would be to allow disabling the loading of settings from disk altogether (as BoringSSL does for example). LibreSSL has the C:\Windows\ mitigation in place, out of the box. [Aside: libssh suffers from similar issues on Windows]

Other reasons:

• curl-for-win already doesn't use OpenSSL, but a fork of it, see next:
• No HTTP/3 support. So one must rely on a fork, such as quictls. Which in turn isn't GPG signed (or even checksummed, or formally released).
• Large binaries. (BoringSSL is significantly smaller. Exact figures are in the commits [1] [2] from the time when I was dealing with this)
• Regressions (from 2022: 3.0.3 requiring rollback due case-sensitive comparison bug, 3.0.6 withdrawn for a CRITICAL vulnerability fix.)
• Rejected Windows vulnerability fix with loading configuration from hardcoded paths. (see above)
• Unfixed cross-build bug, that prevents mitigation of the above. (see above)
• QUIC API support not planned. A different solution is planned and even that is years away.
• Speaking of the planned "different solution": It seems risky to rely a newly-designed, supported-by-no-app, networked QUIC component added to an already large and fragile OpenSSL codebase. That was (and still is) the general sentiment after these plans were revealed.
• Missing Windows ARM64 ASM support. (to be fair this also applies to the alternatives). Fixed with OpenSSL v3.1.0.
• Uses proprietary build system.
• Requires a patch for reproducible builds. (otherwise certain build settings/properties are included in the final binaries)
• Requires build tricks to disable dynamic loading of code.
• Windows support feels like an afterthought.
• Locking/performance issues with 3.x:
  • https://news.ycombinator.com/item?id=33831967
  • 3.0 performance degraded due to locking openssl/openssl#20286
  • HAProxy 2.9.5 performance  haproxy/haproxy#2454 (comment)

Since the end of 2022, both LibreSSL 3.7.0 and BoringSSL are fully on par with OpenSSL (or rather with the quictls fork!), with the exception of TLS-SRP, which is missing from both. SRP is old and not recommended nowadays. Also since a couple of weeks, curl started using a function which is missing from LibreSSL. This is going to affect 7.88.0, though it's unclear in what ways [UPDATE: it's an internal detail, with no user-facing effect]. LibreSSL 3.8.1 will drop loading of engines, conf modules and disables DSO altogether.

I shall add, that the cited vulnerability shouldn't affect curl itself, because we build it with -DCURL_DISABLE_OPENSSL_AUTO_LOAD_CONFIG. I haven't tested what else may still be loaded from the settings directories, plus we're distributing OpenSSL static libs, which remain affected by the vulnerability.

Overall, I think most curl-for-win users are using curl.exe, for them, the choice of TLS-backend is transparent for the most part. It may matter for static lib users, but I'd guesstimate their numbers not too high. And by that I mean very low.

[vszakats]

Another opinion on the state of OpenSSL:
haproxy/haproxy#680 (comment)

[jay]

Interesting read. I hope things improve.

[vszakats]

Since LibreSSL (with v3.7.x) has caught up in features with BoringSSL, and BoringSSL takes quite a bit of effort to adjust for our purposes, and also doesn't seem that its pthread requirement is going away anytime soon (hitting a toolchain bug, which also seems to stay with us), I plan to switch to LibreSSL instead.

This will result in significantly smaller binaries, faster builds, hack- and patch-free build process and a proven development path with versioned releases. Also worth a note that the latest OpenSSL release 3.1.1 from 2 months ago, still did not get a matching release from the quictls fork. Instead it got a second set of patches for performance. Meanwhile mainline OpenSSL turned to focus almost exclusively on its own in-house QUIC implementation.

[vszakats]

@arm64-v9a That QUIC is OpenSSL's home-grown implementation. It isn't compatible with curl or any software out there and it isn't released yet. A missing quictls-compatble API is just one in the list of concerns with OpenSSL.

There is no perfect choice out there, including the mainline. E.g. OpenSSL itself suffers from a significant performance drop since 3.x, due to locking.

[vszakats]

A quictls QUIC API is required for HTTP/3 in curl. OpenSSL decided that it will not provide one.

[vszakats]

After playing with LibreSSL for a while today, found some caveats:

LibreSSL offers ASM support for x64 only (which is OK). ASM is enabled by default in autotools builds, but due to bugs, it's never enabled in CMake MinGW builds (MSVC seems OK, but haven't tested). The next problem is that 3.8.0 builds are consistently broken with ASM enabled (they do not even start, as if the exe was corrupted). 3.7.3 works fine.

[vszakats]

Tried it, but llvm doesn't support this option, so failed at ./configure: https://ci.appveyor.com/project/curlorg/curl-for-win/builds/47725397#L4569

[vszakats]

PR for the LibreSSL CMake/MinGW/ASM bugs: libressl/portable#894
Related Issue also for the 3.8.0 crashes: libressl/portable#893

[vszakats]

Raw crypto performance is important, but not the most important property of a TLS-backend in our primary use-case of transferring data over networks either manually or via shell scripts.

OpenSSL is unbeatable in raw crypto perf as they've been focusing on this heavily. It's also one reason why the library grew so huge in size. E.g. libcrypto-lib-aes-gcm-avx512.obj in 3.1.2, a single function, at 729KB, is larger than libcurl itself in a perfectly usable configuration. This to me seems overkill for our purposes.

It'd be interesting to see performance comparison on typical hardware/network with typical workloads and typical, secure servers using curl built against various backends. If performance turns out to be remarkably worse, it's probably better to raise these issues directly with LibreSSL.

curl-for-win also offers the Schannel backend by default. It's an option for cases the default backend performs worse than desired. Custom builds also remain possible with all popular backends.

[vszakats]

Status updates:

• LibreSSL 3.8.2 stable is out. It includes some necessary build patches. Also fixes the Windows x64 crash.
  cmake: fix to enable ASM support in MinGW builds libressl/portable#894
  Fix processor detection in CMakeLists.txt libressl/portable#909
• Still looking forward for some last-minute LibreSSL build patches coming up in the next release. No showstoppers.
  amd64/bn_arch.h: avoid redefinition of OPENSSL_NO_ASM libressl/portable#940
  cmake: simplify if expressions, accept CPU values more consistently libressl/portable#938
  cmake: stop passing unused C macros libressl/portable#937
  cmake: fix arm64 to not trigger armv4 ASM libressl/portable#936
  slim tls-static library (used in tests) libressl/portable#932
  slim down libtls libressl/portable#931
  cmake: fix default TLS_DEFAULT_CA_FILE value in libtls for Windows libressl/portable#930
  delete unused savsig variable on Windows libressl/portable#927
• LibreSSL CMake builds are ready for use and have been tested extensively.
• There is also a short list of LibreSSL caveats, e.g. missing support for SSL_set0_wbio(). But also, no showstoppers here.
• Pushed a series of curl patches to enable certain OpenSSL codepaths also for LibreSSL. Some of these were out with 8.4.0, some more coming with 8.5.0.
  openssl: enable infof_certstack for 1.1 and LibreSSL 3.6 curl#12385
  rand: fix build error with autotools + LibreSSL curl#12274
  openssl: use SSL_CTX_set_ciphersuites with LibreSSL 3.4.1 curl#11616
  openssl: use SSL_CTX_set_keylog_callback with LibreSSL 3.5.0 curl#11615
  openssl: switch to modern init for LibreSSL 2.7.0+ curl#11611
  cmake: detect SSL_set0_wbio in OpenSSL curl#11555
• and to libssh2:
  openssl: use automatic initialization with LibreSSL 2.7.0+ libssh2/libssh2#1146
• and to ngtcp2:
  compiler warnings with LibreSSL 3.7.3 ngtcp2/ngtcp2#935
  Fix compile error with libressl ngtcp2/ngtcp2#936
• OpenSSL 3.2.0 was released today.
• One long-awaited 3.2.0 feature for Windows is support
  for the system certificate store (not yet enabled by default
  at runtime). I believe curl supports that also for LibreSSL
  via --ca-native and --proxy-ca-native and via
  CURLSSLOPT_NATIVE_CA in libcurl.
• It seems that quictls stops rebasing on top of 3.2.0.
  Meaning stuck with 3.1.x bugfix releases.
  https://github.com/quictls/openssl/issues/138
• Path for future quictls updates is unclear. Current best option
  may be coming with OpenSSL 3.3, pending questions:
  OpenSSL 3.3, and moving the QUIC API to a separate project? quictls/openssl#124

I don't expect much to change short/mid-term with the quictls
(or OpenSSL QUIC) situation. So, unless I missed something here
or a last-minute showstopper comes up, it means I'm making the
switch to LibreSSL on the next curl release, 8.5.0. Preceded by
a final revision of 8.4.0 built with LibreSSL, to be able to sort out
if a possible regression is due to the TLS backend change or due
to the curl version update.