python: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate | Rancher EKS Cluster

What happened (please include outputs or screenshots):

Here’s me using kubectl. I’m just expecting the equiv. Python to work.

→ export KUBECONFIG=~/.kube/config
→ kubectl get no
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-122-0-141.us-west-2.compute.internal   Ready    <none>   23h   v1.20.11-eks-f17b81
ip-10-122-16-82.us-west-2.compute.internal   Ready    <none>   23h   v1.20.11-eks-f17b81

My attempt at doing the same thing in Python and getting different results.

>>> import kubernetes
>> kubernetes.config.load_config()
>>> v1 = kubernetes.client.CoreV1Api()
>>> v1.list_node()
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(                                                                                                                                           File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)                                                                                                                                                        File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
    conn.connect()                                                                                                                                                                   File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(                                                                                                                                                     File "/usr/local/lib/python3.9/dist-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(                                                                                                                                                File "/usr/local/lib/python3.9/dist-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)                                                                                                            File "/usr/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(                                                                                                                                             File "/usr/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()                                                                                                                                                    ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1123)
                                                                                                                                                                                   During handling of the above exception, another exception occurred:

OK, doesn’t work. Let’s try loading the kube_config directly.

>>> import os
>>> os.path.exists(os.environ["KUBECONFIG"])
True
>> kubernetes.config.load_kube_config(os.environ["KUBECONFIG"])
>> v1 = kubernetes.client.CoreV1Api()
>>> v1.list_node()  
# SAME ERROR AS ABOVE

OK, digging through Github issues I came across #1622 which leads to #36 I can try that…

>>> from kubernetes import client
>>> from kubernetes import config                                                                                                                                                 
>>> from kubernetes.client.api import core_v1_api
>>> config.load_config()
>>> configuration = client.Configuration()
>>> configuration.assert_hostname = False
>>> configuration.verify_ssl = True
>>> client.Configuration.set_default(configuration)
>>>
>>> v1 = core_v1_api.CoreV1Api()
>>> v1.list_node()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/dist-packages/kubernetes/client/api/core_v1_api.py", line 16844, in list_node
    return self.list_node_with_http_info(**kwargs)  # noqa: E501
  File "/usr/local/lib/python3.9/dist-packages/kubernetes/client/api/core_v1_api.py", line 16951, in list_node_with_http_info
    return self.api_client.call_api(
  File "/usr/local/lib/python3.9/dist-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.9/dist-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/usr/local/lib/python3.9/dist-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "/usr/local/lib/python3.9/dist-packages/kubernetes/client/rest.py", line 240, in GET
    return self.request("GET", url,
  File "/usr/local/lib/python3.9/dist-packages/kubernetes/client/rest.py", line 213, in request
    r = self.pool_manager.request(method, url,
  File "/usr/local/lib/python3.9/dist-packages/urllib3/request.py", line 74, in request
    return self.request_encode_url(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/request.py", line 96, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/poolmanager.py", line 376, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 692, in urlopen
    conn = self._get_conn(timeout=pool_timeout)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 281, in _get_conn
    return conn or self._new_conn()
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 235, in _new_conn
    conn = self.ConnectionCls(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 130, in __init__
    _HTTPConnection.__init__(self, *args, **kw)
TypeError: __init__() got an unexpected keyword argument 'assert_hostname'

OK, what if we just disable SSL verify instead?

>>> from kubernetes import client
>>> from kubernetes import config
>>> from kubernetes.client.api import core_v1_api
>>> config.load_config()
>>> configuration = client.Configuration()
>>> configuration.verify_ssl = False
>>> client.Configuration.set_default(configuration)
>>> v1 = core_v1_api.CoreV1Api()
>>> v1.list_node()
...
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f245a5701f0>: Failed to establish a new connection: [Errno 111] Connection refused'))

I can’t actually get this client to work in any way that I can try…

What you expected to happen:

Anything to work.

How to reproduce it (as minimally and precisely as possible):

Many examples above.

Anything else we need to know?:

Environment:

  • Kubernetes version (kubectl version):
→ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:30:48Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.11-eks-f17b81", GitCommit:"f17b810c9e5a82200d28b6210b458497ddfcf31b", GitTreeState:"clean", BuildDate:"2021-10-15T21:46:21Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.23) and server (1.20) exceeds the supported minor version skew of +/-1
  • OS (e.g., MacOS 10.13.6):
→ uname -a
Linux 7CWL5Y2 5.10.16.3-microsoft-standard-WSL2 #1 SMP Fri Apr 2 22:23:49 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
  • Python version (python --version)
# python3 --version
Python 3.9.2
  • Python client version (pip list | grep kubernetes)
# pip list | grep kubernetes
kubernetes          20.13.0

I originally tried the very latest of this library, but I decided to test a downgrade to see if v23 would work too. I downgraded to have the client version match the server version.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 3
  • Comments: 26 (3 by maintainers)

Most upvoted comments

Thanks, I had the same problem as well and wanted to keep kubectl working.

I generate my client now like this until this is fixed:

import os
import yaml
import tempfile
from pathlib import Path
from kubernetes import client, config
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)


def gen_client():
    kube_config_orig = f'{Path.home()}/.kube/config'
    tmp_config = tempfile.NamedTemporaryFile().name

    with open(kube_config_orig, "r") as fd:
        kubeconfig = yaml.load(fd, Loader=yaml.FullLoader)
    for cluster in kubeconfig["clusters"]:
        cluster["cluster"]["insecure-skip-tls-verify"] = True
    with open(tmp_config, "w") as fd:
        yaml.dump(kubeconfig, fd, default_flow_style=False)

    config.load_kube_config(tmp_config)
    os.remove(tmp_config)

    return client.CoreV1Api()


v1 = gen_client()
+9
dheeg on Aug 17, 2022

I also encountered this problem, following is some debug info which might be useful, seems like a problem with urllib3

Reproduction steps:

  • set env vars
    • USERNAME=<YOUR KUBECTL CONFIG USER NAME>
    • CLUSTER=<YOUR KUBECTL CONFIG CLUSTER NAME>
    • CACERT=<FILE TO STORE THE CA CERTIFICATE IN>
    • TOKEN="$(kubectl config view --raw "-ojsonpath={.users[?(@.name=='${USERNAME}')].user.token}")"
    • kubectl config view --raw "-ojsonpath={.clusters[?(@.name=='${CLUSTER}')].cluster['certificate-authority-data']}" | base64 -d > $CACERT
    • SERVER="$(kubectl config view --raw "-ojsonpath={.clusters[?(@.name=='${CLUSTER}')].cluster.server}")"
  • make a request to Kubernetes using curl
    • curl -H "authorization: Bearer ${TOKEN}" "${SERVER}/api/v1/nodes" --cacert $CACERT
    • request succeeds
  • make the same request using urllib3 (same way the kubernetes python client does it)
    • python3 -c "import urllib3; urllib3.PoolManager(ca_certs='${CACERT}').request('GET', '${SERVER}/api/v1/nodes', headers={'authorization': 'Bearer ${TOKEN}'})"

Expected

  • Request using urllib3 should succeed

Actual

  • Request using urllib3 fails with following error:
  • urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='***', port=443): Max retries exceeded with url: ****/api/v1/nodes (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1131)')))
+1
OriHoch on Dec 13, 2022

Just to help any others who come across this, I had a similar error message while using ansible with the kubernetes.core modules.

Likely not the same cause as above, but the certificate that I provided in certificate-authority-data was not self-signed. Our internal setup used:

  • a per-worker certificate (herinafter worker)
  • signed by an cluster-specific intermediary CA (herinafter cluster)
  • signed by a department-level intermediary CA (herinafter department).
  • signed by a root CA (herinafter company).
  • signed by company (self).

The value of certificate-authority-data in my ~/.kube/config was effectively set to:

Issuer: department
Subject: cluster
...

This doesn’t make valid certificate chain, but usage of an intermediary as your certificate-authority-data does work with kubectl and any tooling based on kubernetes/client-go

I resolved my issue by setting certificate-authority-data to something more like:

Issuer: company
Subject: company
...

this wouldn’t be ideal for inter-worker communication (broadens the trust scope a great deal), but it will be fine for my local config.

I wouldn’t be surprised if other tooling (a la kubeadm) will generate a similar config if configured to get its certificates from an issuer like vault.

Only thing left that is curious to me is why it didn’t resolve the root certificate from the operating store (since company is present in my trust store), but that’s an investigation for another day.

Hope this helps someone 😃

+1
DanInProgress on Dec 9, 2022

The client just does not seem to read certificate-authority-data at all and then (obviously) fails to connect.

EDIT: It does seem to read it correctly but somehow it’s broken for some clusters. With a remote cluster on AKS it works just fine. The broken cluster is an in LAN cluster created with kubeadm.

The one difference I can detect is that AKS uses 4096 bits RSA and kubeadm uses 2048 and the kubeadm CA cert has the following extra extensions:

X509v3 Subject Key Identifier:
    D7:FB:F3<snip>
X509v3 Subject Alternative Name:
    DNS:kubernetes

And it connects to a nonstandard port, which might also cause the issue.

configuration.verify_ssl = False
configuration.client_side_validation = False

Has no effect which is curious

+1
EraYaN on Dec 7, 2022

there seem to be issues in your use of the client library, could you try the given example in https://github.com/kubernetes-client/python#examples?

+1
yliaog on Apr 12, 2022
Read more comments on GitHub
← python: [RFC] Major breaking changes in the upcoming release
python: SSL verification on python2.7 failing →