origin: Pod unable to pull image from private registry on Docker Hub

We also encountered problems when trying to pull a private builder image from Dockerhub, which is why I’m here. Our setup works fine on a 3.7 cluster, but fails on 3.10 and 3.11. We did find a solution, however, I do have questions why our setup actually works.

First of all, our OpenShift Version:

openshift v3.11.146
kubernetes v1.11.0+d4cacc0

We’re setting up the following resources (which works perfectly on 3.7):

oc create secret docker-registry dockerhub-secret \
    --docker-server=docker.io \
    --docker-username=${DOCKER_USERNAME} \
    --docker-password=${DOCKER_PASSWORD} \
    --docker-email=unused

oc secrets link builder dockerhub-secret

oc import-image ${IMAGE_STREAM} \
    --from=docker.io/our-org/our-image \
    --confirm \
    --scheduled=true \
    --all

Unfortunately, on 3.10 and 3.11 every build fails:

pulling image error : repository docker.io/our-org/our-image not found: does not exist or no pull access
error: build error: unable to get docker.io/our-org/our-image@sha256:...

Possible solutions mentioned above did not work for us. Lots of trial and error later we created our image stream with --reference-policy=local, which resolved the error and our build runs just fine.

oc import-image ${IMAGE_STREAM} \
    --from=docker.io/our-org/our-image \
    --confirm \
    --scheduled=true \
    --all \
    --reference-policy=local

Problem is, we do not really understand why the change from reference policy ‘source’ to ‘local’ makes it possible to pull the image from Dockerhub. Maybe someone with a deeper understanding of OpenShift can shed some light on this behaviour?

@StefanScherer – there are two places where the docker image secret get used:

1. Image stream creation (oc import-image or in the UI: Add to Project -> Image Name -> 🔍)
2. Pod creation (triggered by oc new-app per my original description or in the UI: Add to Project -> Image Name -> 🔍 -> Deploy)

In this issue I described how (1) works fine but (2) fails. In your two comments it sounds like you had an issue with (1) regardless of using oc or the UI, then you fixed it with the workaround of using oc create secret generic. However, I don’t see any indication that (2) works for you, and suspect you’ll run into a problem there. Can verify whether you can successfully deploy the private docker.io image and have the pod created?

@bparees – Though I believe @StefanScherer’s observation is not related to this issue (see above), I tried the workaround for posterity and can confirm that the deployment still fails at pod creation, step (2). And yes, here are all of the permutations of image pull secret creation I’ve tried:

• From disk with oc secret create ... (.dockercfg format)
• Using oc create secret docker-registry ... (.dockerconfigjson format)
• oc create secret generic --type=kubernetes.io/dockerconfigjson ... (.dockerconfigjson format)
• oc create secret generic --type=kubernetes.io/dockercfg ... (.dockercfg format)
• And now also in the UI, copy-pasting my ~/.docker/config.json (.dockerconfigjson format)
• On top of these, I also manually edited the secret data by duplicating auths for multiple domains involved in docker.io private registry authentication…

  {
    "auths": {
      "https://index.docker.io/v1/": {
        "auth": REDACTED
      },
      "auth.docker.io": {
        "auth": SAME AS ABOVE
      },
      "registry.docker.io": {
        "auth": SAME AS ABOVE
      },
      "docker.io": {
        "auth": SAME AS ABOVE
      }
    }
  }
  

All of the above permutations produce valid, correctly-formatted docker image pull secrets, and I can verify this because step (1) image stream creation succeeds and imports image metadata. But all fail at step (2) 😢

I can confirm (for OpenShift Online) that pods cannot pull private images from docker.io. I tried like every instruction I could find on the internet but none of them worked 😞 .

Interesting though is the fact that at some point it worked (around 6th of February) but a few days later it again stopped working. It also works on a dedicated instance that I have access to, which has the version v3.6.173.0.49, although it didn’t work there a few days ago neither (maybe because a different version was deployed then, didn’t check that).

For anyone else that winds up here due to image pull issues when running a build that is based on an ImageStream tied to a private repo – I struggled getting a builder pod to be able to pull from a private repo because I had only linked the secret to builder with --for=pull. I could import the image to an ImageStream just fine but the build would fail with the error:

error: build error: failed to pull image: unauthorized: access to the requested resource is not authorized

Once I linked with --for=mount the pull worked within the build pod:

oc secrets link builder my-docker-secret --for=pull,mount

Incidentally, the default value for --for if it is not specified is mount … by specifying --for=pull I caused myself some trouble 😃

Thanks @php-coder, @bparees. I definitely tried both docker secret formats, and at one point embedded more than one entry (i.e. multiple hostnames) in both .dockerconfigjson and .dockercfg formats to see if it would help.

Ultimately I determined the problem to not be with docker secret formats, since I was able to pull images from a private registry hosted on gitlab.com. It’s docker.io private registries which were having issues. It would be good to repeat the test with a third private docker registry host (I’m thinking codefresh.io), to have another data point.

I also spent hours trying to get an image from private registries (github then gitlab), until I found this thread. I just added a new secret for gitlab.com (I already had one for registry.gitlab.com) and now the image gets pulled and deployed. This is quite frustrating but now I am able to try out openshift a bit further! Thanks.

If you use service account in our deployment config in my case I used spserviceuser so the spserviceuser have to have access to pull your private registry.

oc secrets link spserviceuser dockerauth --for=pull

If you don’t specify any service user then please link the secrete default service user oc secrets link default dockerauth --for=pull

In my case oc create secret docker-registry worked fine.

apiVersion: v1
kind: DeploymentConfig
*********
spec:
  replicas: 1
  selector:
    app: splessons-abc
  strategy:    
    type: Rolling
  template:
    metadata:
      labels:
        app: splessons-abc
    spec:
      containers:
      -name: splessons-abc
      serviceAccountName: spserviceuser

This is problem is occuring only when we don’t use the default service account in deployment config.

Thanks, Sree

Found the fix. Based on this https://stackoverflow.com/questions/47993222/can-not-pull-image-from-gitlab-private-registryopenshift

So gitlab make the authentification in two times, first gitlab.com then registry.gitlab.com. Actually the error we got was the first one that was being dropped.

Just duplicate what you've done for registry.gitlab.com, but for gitlab.com```

i had create another secret that is referring to  `gitlab.com`   similar to  `egistry.gitlab.com` then it fixed the issue

OK, thanks @bparees that command works for me

$ oc create secret generic dockerhub-secret \
  --from-file=.dockerconfigjson=$HOME/.docker/config.json \
  --type=kubernetes.io/dockerconfigjson
$ oc secrets link default dockerhub-secret --for=pull
$ oc import-image myimage --from=docker.io/myorg/myimage --confirm

But as a Docker4Mac user I had to disable the preferences in General tab -> Securely store docker logins in macOS keychain and run docker login to create a config.json with auth entries.

Okay, for that third data point, I just confirmed that OpenShift is able to pull from a private registry on codefresh.io. I did not test for multiple types of secrets, and instead used the newrer .dockerconfigjson format (rather than .dockercfg).

Here’s a little more information about how I found out I needed two secrets in order to get the gitlab.com repo to work… Since I pushed my image to registry.gitlab.com, I used curl to inspect what the authentication with the registry might look like:

curl -i https://registry.gitlab.com/v2/
HTTP/1.1 401 Unauthorized
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Bearer realm="https://gitlab.com/jwt/auth",service="container_registry"
X-Content-Type-Options: nosniff
Date: Tue, 13 Mar 2018 17:58:23 GMT
Content-Length: 87

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

What caught my attention is the realm="https://gitlab.com/jwt/auth" bit and how it uses the gitlab.com domain instead of registry.gitlab.com. So I created a new image pull secret with the same credentials as registry.gitlab.com, except for gitlab.com, and added both image pull secrets to my deployment config. (What will also work is to manually shove multiple secrets in whatever JSON format you end up using, .dockercfg or .dockerconfigjson, into the same image pull secret and add that one secret to the deployment config.)

docker.io is confusing because their actual domain name for registry pulls is index.docker.io (can anyone confirm?), and their curl output looks like this:

curl -i https://index.docker.io/v2/
HTTP/1.1 401 Unauthorized
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io"
Date: Tue, 13 Mar 2018 18:07:22 GMT
Content-Length: 87
Strict-Transport-Security: max-age=31536000

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

Looks like they use auth.docker.io for their auth realm. In spite of all this, adding secrets to OpenShift for three domain names simultaneously (docker.io, index.docker.io, auth.docker.io), did not make for a successful deployment 😞