kubernetes: Failed to pull image with "x509: certificate signed by unknown authority" error

You would think that this is solved by now.

CA Certificates

Actual recorded cases of preventing unauthorized access : ZERO Amount of countless of developer time wasted because of tooling that don’t integrate CA certs into their tooling properly: gasmillions of man hours.

Moral of the story. Ditch CA certs. Such a ballache every time you have try to get tooling to work together. Nobody knows how it works. Nobody. Software that use it never work. In the end you just copy all the certs to every machine and your toaster just so you don’t get that god damn x509: certificate signed by unknown authority bullshit error every bloody time you try to do any tooling.

Now I have to go and drill right to the core of this cluster to get those certs installed, because kubernetes’s secrets for handling docker is just plain useless.

Just use the money that would have been spent trying to get the bloody CA certs to work and hire a henchman with an axe that cut the hardlines when the hacker comes. This CA certs it’s not security if it does not let authorized people in because the entire field is just one giant BUG that will JAM your tooling.

@srossross-tableau

As far as I remember this was a docker issue, not a kubernetes one. Docker does not use linux’s ca certs. Nobody knows why.

You have to install those certs manually (on every node that could spawn those pods) so that docker can use them:

/etc/docker/certs.d/mydomain.com:1234/ca.crt

This is a highly annoying issue as you have to butcher your nodes after bootstrapping to get those certs in there. And kubernetes spawns nodes all the time. How this issue has not been solved yet is a mystery to me. It’s a complete showstopper IMO.

This should really be solved using the secrets mechanism of kubernetes. But somehow it is not. Who knows!?

should anyone face it while using directly the gcr.io, one possible situation is that CA certificates on your machine are too old.

docker pull gcr.io/google_containers/kube-apiserver-amd64:v1.7.2
Trying to pull repository gcr.io/google_containers/kube-apiserver-amd64 ...
Get https://gcr.io/v1/_ping: x509: certificate signed by unknown authority '

solution that worked for me on RH/CentOS:

yum check-update ca-certificates; (($?==100)) && yum update ca-certificates || yum reinstall ca-certificates
update-ca-trust extract

@rushilpaul

• First --insecure-skip-tls-verify is not a valid argument for kubectl create;
• Actually x509 error is on docker side. The daemon failed to pull image from that insecure registry. You can refer to insecure docker registry on how to trust/skip the registry security.

2020 and still the same issue. private harbor registry. docker pull works with no problem. ls /etc/docker/certs.d/registry.myharbor.com/ shows up the certificate. kubernetes fails to pull images with imagepullbackoff error. It is 3 years and kubernetes still has this issue. Very disappointing.

@unixoff THANKS.

I did this a gazzilion times:

systemctl restart docker

when I did this it worked:

systemctl restart docker
systemctl restart containerd

Don’t forget to execute after changes

systemctl restart docker
systemctl restart containerd

working solution for pulling docker image on ubuntu from Artifactory (certificate is selfsigned):

• put all used (if there is more than one root ca) ca certs in /usr/local/share/ca-certificates
• run update-ca-certificates
• restart docker daemon (sudo service docker restart)

That is a valid arg to kubectl create but just controls trust between kubectl and the API server

The pull error is between the node and the docker registry. The node either needs to trust the certificate or treat that registry as an untrusted registry (which makes the node tolerate TLS verification errors)

These commands work for me, run them on all nodes include master node

sudo cp /opt/certs/registry.crt /usr/local/share/ca-certificates/docker-registry.crt
sudo update-ca-certificates

sudo systemctl restart docker
sudo systemctl restart containerd

Ref: https://docs.docker.com/registry/insecure/#docker-still-complains-about-the-certificate-when-using-authentication

for x509: certificate signed by unknown authority

https://gitlab.com/stirumala/kubernetes/-/blob/main/findings.txt

I am working in my local dev environment

    OS:
       Ubuntu (bionic) 18.0.4 LTS    
    Minikube Version:
       v1.11.0
    Docker Version:
       19.03.10

I am using Jfrog Container Registry as registry to my minikube. I am able do the following:

1. docker login localhost:443 | or | ip-add:443
2. docker push ip-add:443/docker-local/test:latest
3. docker pull ip-add:443/docker-local/test:latest

I have configured Jfrog Container Registry to run behind Nginx Reverse Proxy listening on port 443. Created self-signed certs and Jfrog is using these certs.

Configured docker to use the self-signed certs as follows.

1. Create certs, copy it to /usr/local/share/ca-certificates/
2. sudo update-ca-certificates
3. copy the certificate to /etc/docker/cert.d/192.168.0.114:443/ca.crt
4. restarted the docker, just be sure

Configure K8 to use the docker login secret by .yaml file as following:

1. base64 encode ~/.docker/config.json
2. use it in the following template

apiVersion: v1
kind: Secret
metadata:
 name: myregistrykey
 namespace: awesomeapps
data:
 .dockerconfigjson: UmVhbGx5IHJlYWxseSByZWVlZWVlZWVlZWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGx5eXl5eXl5eXl5eXl5eXl5eXl5eSBsbGxsbGxsbGxsbGxsbG9vb29vb29vb29vb29vb29vb29vb29vb29vb25ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubmdnZ2dnZ2dnZ2dnZ2dnZ2dnZ2cgYXV0aCBrZXlzCg==
type: kubernetes.io/dockerconfigjson

In the deployment.yaml, I use ImagePullSecrets and the name flag.

Now after all this setup where the docker pull is working on terminal, I get error on the pods saying x509 IP Sans.

I went through lot of documentation and K8 issues, replicated the step of https://github.com/kubernetes/kubernetes/issues/43924#issuecomment-631533150

replicated the steps didn’t work out. Can anyone let me know what I am doing wrong? and how can I correct it.

Paste the entire folder name in ‘/etc/docker/certs.d/’ please. And the filenames of the certs.

It should work if all your nodes have that cert installed.

--insecure-skip-tls-verify just skips the server’s certificate verification, not docker registry, so it can not solve the problem. The error is from Docker daemon while pulling image.

I installed the server certificate globally on this kubernetes master node and then restarted the docker service running on it.

Maybe you should try the command docker pull 10.78.0.228:5000/monitorms on the k8s nodes which hold the pod, not the k8s master.

No one really spelled out how to fix this. Here is what I did

My env:

• Private docker registry on separate host. No password, using port 443
• Private openssl CA.

1. Generate certs for each node (workers and masters). need public key, private key (unencrypted), and your root ca’s public key (Openssl CA public key). I created a Subject Alternative Name (SAN) value for short hostname, FQDN, and IP address for each.
2. On each node create /etc/docker/certs.d/ (If it does not exist create it)
3. cd /etc/docker/certs.d/, create a directory in here that uses the SAME EXACT name used when you communicate with the registry in your manifest. So if you are referecing “docker-reg.company.domain” directory must say that. Or if you use “docker-reg.company.domain:443”, need to make the directory name include the port. Must be exact as you reference in your mainfest.
4. Copy your RootCA, public key, and private key to this directory. They must be this name structure.

• root ca = ca.crt
• public key = client.cert
• private key = client.key

1. Copy the root ca public key also to the cert store for the OS

Debian / Ubuntu cp certs/domain.crt /usr/local/share/ca-certificates/myregistrydomain.com.crt update-ca-certificates Rhel / Centos cp certs/domain.crt /etc/pki/ca-trust/source/anchors/myregistrydomain.com.crt update-ca-trust

1. Test to make sure you can pull directly from the node with docker pull docker-reg.company.domain:443/my-image:latest (Or however you have it in your manifest, must be exact)

If all this is done correctly try doing a deployment with kubectl again. Be careful of namespaces.

I run kubernetes on RancherOS as part of Rancher 2.x setup and have private registry that is not internet facing, so I have to use self-signed certificate on it, resulting in x509 error. I read this thread and few others and the solved the issue - sharing in case it may help someone, if not directly then by suggesting possible path.

This worked for me - https://www.ctrl-alt-del.cc/2018/11/solution-rancher-2-k8s-private-registry.html