kubernetes: External docker registry, can't pull images

Hello everyone, I’m trying since a while to pull images from my external private docker registry, and I don’t understand why it doesn’t work.

I’ve read a lot a things around the web and almost tested everything I’ve found, without any success.

Finally, here I am to ask for some help…

My private registry is a v2 version, so I use .docker/config.json files instead of .dockercfg I’ve tryed to types of config.json: one with auth and one without.

{"auths":{"https://my-private-registry.fr:5000":{"auth":"XXXXX","email":"my@mail.fr"}}}
{"https://my-private-registry.fr:5000":{"auth":"XXXXX","email":"my@mail.fr"}}

I’ve created a secret.yaml file, and then create the secret:

apiVersion: v1
kind: Secret
metadata:
  name: my-registry-key
data:
  .dockerconfigjson: BASE64XXXXXXXXXXXXXXXXXXXXX
type: kubernetes.io/dockerconfigjson

And finally I use this yaml file to test if I can pull or not the image:

apiVersion: v1
kind: Pod
metadata:
  name: apache-example
spec:
  containers:
    - name: apache-example
      image: "my-private-registry.fr:5000/apache_test:latest"
  imagePullSecrets:
    - name: my-registry-key

I must precise that I can manually pull the images from all nodes, since I have the same config.json files in all of them.

Everytime that I try to create my pod, I get this error:

Failed to pull image "my-private-registry.fr:5000/apache_test:latest" from pod "apache-example_default" and container "apache-example": Error: image apache_test:latest not found

What am I doing wrong?

Thank you all in advance for your help

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 19 (2 by maintainers)

Most upvoted comments

This cost WAY more time that it should have taken. Here is what I did:

Create the secret. The secret needs to live in the same namespace as where it will get used. Here we use the secret name “regcred”. Note the “/v2/” at the end of the registry URL

kubectl --namespace NAMESPACE create secret docker-registry regcred --docker-server=“https://myregistry.com/v2/” --docker-username=USERNAME --docker-password=PASSWORD --docker-email=EMAIL

You need to tell the deployment where to get the image from using “imagePullSecrets”. Note that if you are trying out an example using “helm create” and helm 2.8.1, you will need to augment template/deployment.yaml. Add to spec.template.spec.containers there: imagePullSecrets: -name: regcred This is in line with what https://raw.githubusercontent.com/kubernetes/website/master/docs/tasks/configure-pod-container/private-reg-pod.yaml has, as explained in “Create a Pod that Uses Your Secret” in https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/.
You should now be able to create your pod (helm install …)

rajamony on Mar 2, 2018

Is there any reason this is not documented anywhere? I’ve lost quite a bit of time trying to figure out why it was broken with a private repo. We assumed it was an issue on our end rather than a built-in command that basically is useless.

Aaron3 on Dec 26, 2017

I guess it is not solved yet. I spent half a day reading github issues related to this subject and trying to make it work in AWS with Ubuntu Vivid , but it fails to pull images if you serialize config.json with a new docker format.

In fact I created 2 clusters in AWS: one based on kubernetes from release v1.1.7 and second one based on 1.2.0-alpha.8. In both cases the new format does not work.

So after struggling with it for a few hours I make it work on v1.1.7 with older format (dockercfg), procedure below:

Delete wrapping lines “auths”:{ }
Add https:// prefix at the begining of the host URL
Serialize it via following command: echo $(cat ~/.docker/config.json) | base64 -w 0
Copy and paste result to secret YAML based on the old format:

apiVersion:  v1
kind: Secret
metadata:
  name: docker-secret
type: kubernetes.io/dockercfg
data:
  .dockercfg: <YOUR_BASE64_JSON_HERE>

There is also 2nd issue I encountered with Kubernetes installed from release v1.1.7 (latest stable). I have 2 docker private registries, configured with authentication + TLS certificate. The registry docker image pulled from Docker Hub around 3 months ago worked fine with secret based on dockercfg format:

registry            2                   2717e28b728b        3 months ago        223.4 MB

but the latest version of the registry (I tagged it as new below):

registry            new                 c3c093b5becb        6 days ago          165.7 MB

did not work at all, even with “dockercfg” old format ! I was getting authentication failures when I tried to login there from Docker 1.7:

 docker version 
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/amd64

And “Failed to pull image” from Kubernetes.

However the new Docker Registry worked fine with Kubernetes v1.2.0-alpha.8. I noticed this version is using a new AMI Ubuntu Vivid image and the docker inside minions is udpated to version 1.8.3:

Client:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   f4bf5c7
 Built:        Mon Oct 12 05:39:44 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   f4bf5c7
 Built:        Mon Oct 12 05:39:44 UTC 2015
 OS/Arch:      linux/amd64

So it might be related to a docker version, even if according to the official Docker Registry docs it should be supported with docker version 1.6+ - it does not seem to be a case.

Last word of advice. The best way to verify whether custom secret is working with your environment is by trying to login first via docker login docker_registry_host:docker_registry_port and try to pull some image (busybox for example). If it does not work - it won’t work with Kubernetes as well.

Patryk

kpatryk on Feb 23, 2016

@Aaron3 adding-imagepullsecrets-to-a-service-account

fdr2 on Jan 10, 2018