brew: MacOS system curl broken after letsencrypt CA cert expiry

`brew config` output

HOMEBREW_VERSION: 3.2.14
ORIGIN: https://github.com/Homebrew/brew
HEAD: 4e6919b73444e9f9e02ad81d8afc4bbd97533567
Last commit: 2 days ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: c437fbe02752c09d3bad41eb2ab84832ea15968f
Core tap last commit: 20 minutes ago
Core tap branch: master
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 4
Homebrew Ruby: 2.6.3 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.3_2/bin/ruby
CPU: quad-core 64-bit kabylake
Clang: 11.0.0 build 1100
Git: 2.33.0 => /usr/local/bin/git
Curl: 7.54.0 => /usr/bin/curl
macOS: 10.14.6-x86_64
CLT: 10.3.0.0.1.1562985497
Xcode: 11.3.1

`brew doctor` output

Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: A newer Command Line Tools release is available.
Update them from Software Update in System Preferences or run:
  softwareupdate --all --install --force

If that doesn't show you any updates, run:
  sudo rm -rf /Library/Developer/CommandLineTools
  sudo xcode-select --install

Alternatively, manually download them from:
  https://developer.apple.com/download/more/.
You should download the Command Line Tools for Xcode 11.3.1.


Warning: Homebrew's "sbin" was not found in your PATH but you have installed
formulae that put executables in /usr/local/sbin.
Consider setting your PATH for example like so:
  echo 'export PATH="/usr/local/sbin:$PATH"' >> /Users/me/.bash_profile

I ran brew update and am still able to reproduce my issue.
I have resolved all warnings from brew doctor and that did not fix my problem.

What were you trying to do (and why)?

I was trying to reinstall openconnect from HEAD using brew reinstall openconnect. I had originally installed openconnect using brew install openconnect --HEAD.

What happened (include all command output)?

$ brew reinstall openconnect
==> Downloading https://git.infradead.org/users/dwmw2/vpnc-scripts.git/blob_plain/c0122e891f7e033f35f047dad963702199d5cb9e:/vpnc-script

curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.
HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure.
Error: Failed to download resource "openconnect--vpnc-script"
Download failed: https://git.infradead.org/users/dwmw2/vpnc-scripts.git/blob_plain/c0122e891f7e033f35f047dad963702199d5cb9e:/vpnc-script

What did you expect to happen?

openconnect should be installed from latest HEAD.

Step-by-step reproduction instructions (by running `brew` commands)

brew reinstall openconnect

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 24 (15 by maintainers)

Commits related to this issue

lib/test_bot: set `HOMEBREW_FORCE_BREWED_CURL` on Mojave Mojave Curl currently doesn't work (cf. Homebrew/brew#12161). Until we have a fix in `brew`, let's set this in `test-bot` so that we can get C... — committed to carlocab/homebrew-test-bot by carlocab 3 years ago
lib/test_bot: set `HOMEBREW_FORCE_BREWED_CURL` on Mojave Mojave Curl currently doesn't work (cf. Homebrew/brew#12161). Until we have a fix in `brew`, let's set this in `test-bot` so that we can get C... — committed to carlocab/homebrew-test-bot by carlocab 3 years ago

Most upvoted comments

We’ve released 3.2.15 with a full fix for this.

Short summary: You don’t need to set any envs or do any workarounds anymore (though you can keep them if you want) - it should all work automatically now.

Long changelog:

ca-certificates has been added to homebrew-core (https://github.com/Homebrew/homebrew-core/pull/86304)
- Brew will now install this automatically when running macOS <= 10.15.5 (https://github.com/Homebrew/brew/pull/12167), and will use it instead of the system CA store when working with system curl and git. This behaviour can be forced on other platforms (like Linux) by HOMEBREW_FORCE_BREWED_CA_CERTIFICATES=1.
  - If building from source, we make sure even a curl with broken certificates can still install this (https://github.com/Homebrew/brew/pull/12172).
- For Yosemite and El Capitan, a full install of Homebrew curl and git are required and will be performed automatically.
- If you are using our openssl@1.1, our CA store used by default (other TLS libraries coming soon), but it can also be used externally should whatever you are using allow swapping CA certificate stores.
We’ve made the installation of Homebrew’s curl easier
- When building from source (i.e. not using bottles), plain HTTP mirrors have been added (https://github.com/Homebrew/homebrew-core/pull/86292, https://github.com/Homebrew/homebrew-core/pull/86293, https://github.com/Homebrew/homebrew-core/pull/86294, https://github.com/Homebrew/homebrew-core/pull/86295, https://github.com/Homebrew/homebrew-core/pull/86296, https://github.com/Homebrew/homebrew-core/pull/86297, https://github.com/Homebrew/homebrew-core/pull/86298, https://github.com/Homebrew/homebrew-core/pull/86299, https://github.com/Homebrew/homebrew-core/pull/86300, https://github.com/Homebrew/homebrew-core/pull/86301)
  - We will make sure these continue to be offered (https://github.com/Homebrew/brew/pull/12166).
  - Note that on Linux, this doesn’t cover the full dependency tree yet (but if you are experiencing SSL cert issues there then HOMEBREW_FORCE_BREWED_CA_CERTIFICATES=1 should be enough)
- Fewer dependencies are installed, without losing functionality:
  - cmake (needed if you are building from source) no longer installs documentation - this is now in a separate cmake-docs formula (https://github.com/Homebrew/homebrew-core/pull/86249).
  - c-ares now has only one build-time dependency (https://github.com/Homebrew/homebrew-core/pull/86250/)
  - The libraries from nghttp2 have been split into a separate libnghttp2 formula (https://github.com/Homebrew/homebrew-core/pull/86291).
Outside of brew’s core operations, we’re fixing formula-specific issues on a case-by-case basis. If you spot a SSL certificate error with a specific formula, open a homebrew-core issue and let us know.
- Homebrew’s git now uses our own libcurl on <= Mojave (https://github.com/Homebrew/homebrew-core/pull/86439), so if you currently using system git and are experiencing issues, you can switch to our git. Alternatively, you can swap out the default SSL store used by system git to point to our CA store (via GIT_SSL_CAINFO), which is installed by default on affected macOS systems.

There’s more but I think that covers most of noticeable things.

+15

Bo98 on Oct 5, 2021

This answer at Stack Overflow shows which entry to delete from /etc/ssl/cert.pem to get things working on your machine.

EricFromCanada on Oct 2, 2021

Thanks for the report – we’ve encountered this problem in CI in quite a few PRs already.

In the meantime, as a workaround, you can set HOMEBREW_FORCE_BREWED_CURL=1.

carlocab on Oct 1, 2021

due to this https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/.

Yes it’s exactly that. In terms of LibreSSL (what Apple use for curl), it was an bug up to LibreSSL 3.2.0. But Apple did backport a fix to 2.8.3 in macOS 10.15.6.

So the bug affects 10.13-10.15.5 (curl uses SecureTransport before 10.13). 10.10 and 10.11 are also affected because they don’t contain the ISRG Root X1 certificate in their default cert store at all. 10.12 weirdly enough should work fine as far as I know.

Bo98 on Oct 2, 2021

Yes it’s exactly that. In terms of LibreSSL (what Apple use for curl), it was an bug up to LibreSSL 3.2.0. But Apple did backport a fix to 2.8.3 in macOS 10.15.6.

So the bug affects 10.13-10.15.5 (curl uses SecureTransport before 10.13). 10.10 and 10.11 are also affected because they don’t contain the ISRG Root X1 certificate in their default cert store at all. 10.12 weirdly enough should work fine as far as I know.

I can confirm the above. Where I work we have Mac buildbots for each major macOS version. None have brew. This table shows where I’ve had issues with svn and/or git failing, and the version of openssl and curl on each.

macOS	problem?	openssl version	curl --version
10.10.5	fail	OpenSSL 0.9.8zh	curl 7.43.0 SecureTransport
10.11.6	fail	OpenSSL 0.9.8zh	curl 7.43.0 SecureTransport
10.12.6	ok	OpenSSL 0.9.8zh	curl 7.54.0 SecureTransport
10.13.6	fail	LibreSSL 2.2.7	curl 7.54.0 LibreSSL/2.0.20
10.14.6	fail	LibreSSL 2.6.5	curl 7.54.0 LibreSSL/2.6.5
10.15.7	ok	LibreSSL 2.8.3	curl 7.64.1 SecureTransport
11.6	ok	LibreSSL 2.8.3	curl 7.64.1 SecureTransport
12.0	ok	LibreSSL 2.8.3	curl 7.77.0 SecureTransport

10.10 and 10.11 I fixed by just adding the ‘ISRG Root X1’ certificate in Keychain.app, per this:

https://community.letsencrypt.org/t/connection-errors-on-apple-devices/161107/25?u=seanm

But 10.13 and 10.14 still have issues, and indeed I suspect curl is the issue.

I thought updating my svn and git binaries would help, and for that I’ve turned to brew, but it uses curl too, so I’m still stuck. 😃

seanm on Oct 4, 2021

As far as I know you should just be able to install the bottles Homebrew provides as well, it should only be building from source that’s broken.

SMillerDev on Oct 2, 2021

I believe this is due to the expiry of a LetsEncrypt CA certificate.

https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

If I try to download the openconnect file using system curl, it fails.

$ /usr/bin/curl https://git.infradead.org/users/dwmw2/vpnc-scripts.git/blob_plain/c0122e891f7e033f35f047dad963702199d5cb9e:/vpnc-script
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.
HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure.

If I use brew-installed curl, then it succeeds.

$ /usr/local/opt/curl/bin/curl https://git.infradead.org/users/dwmw2/vpnc-scripts.git/blob_plain/c0122e891f7e033f35f047dad963702199d5cb9e:/vpnc-script
#!/bin/sh
#
# Originally part of vpnc source code:
# © 2005-2012 Maurice Massar, Jörg Mayer, Antonio Borneo et al.
# © 2009-2012 David Woodhouse <dwmw2@infradead.org>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
#
################
#
# List of parameters passed through environment
...

Homebrew should be forcing the use of homebrew-installed curl for brew install on systems that don’t have a system curl compatible with the new LetsEncrypt root CA certificates.

wwuck on Oct 1, 2021