grpc: Python gRPC with TLS no longer seems to find the correct credentials by default

What version of gRPC and what language are you using?

Python gRPC 1.32.0

What operating system (Linux, Windows,…) and version?

macOS 11.6 (Big Sur)

What runtime / compiler are you using (e.g. python version or version of gcc)

Python 3.8.3

What did you do?

Attempt to establish a secure channel with a remote host using valid SSL certificate.

What did you expect to see?

A secure channel established. Example:

import grpc

host = "hostname"
port = 443

credentials = grpc.ssl_channel_credentials()
channel = grpc.secure_channel('{}:{}'.format(host, port), credentials)

try:
    grpc.channel_ready_future(channel).result(timeout=5)
    print(f"Successfully established secure channel to {host}")
except:
    print(f"Timeout. Failed to establish secure channel to {host}")

Should return:

Successfully established secure channel to hostname

What did you see instead?

E1014 14:19:37.997415000 123145637335040 ssl_transport_security.cc:1446] Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.

This has previously worked for everyone at my current organization using the gRPC Python client. Instead, on all hosts (macOS, centos) and Python versions (>=3.6) we suddenly need to explicitly pass the credentials location. The following works on macOs:

import grpc

host = "hostname"
port = 443

with open('/etc/ssl/cert.pem', 'rb') as f:
    certificate_chain = f.read()

credentials = grpc.ssl_channel_credentials(root_certificates=certificate_chain)
channel = grpc.secure_channel('{}:{}'.format(host, port), credentials)

try:
    grpc.channel_ready_future(channel).result(timeout=5)
    print(f"Successfully established secure channel to {host}")
except:
    print(f"Timeout. Failed to establish secure channel to {host}")

Can verify curl finds the proper SSL credentials.

Anything else we should know about your project / environment?

The SSL cert is valid, here’s is some information from curl -v hostname on the host we’re trying to reach.

* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=hostname
*  start date: Sep  1 20:41:14 2021 GMT
*  expire date: Nov 30 20:41:13 2021 GMT
*  subjectAltName: host "hostname" matched cert's "hostname"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.