portainer: Updating a service fails when the image is from a private registry

For anyone finding this issue, it’s possible the target node is out of disk space.

If you cannot pull the image to the particular node with the Portainer Images UI, check the free disk space.

Same here, using Portainer 1.20.2 I have to pull new images manually before update services. The webhook is not working also

Reproduced on 1.24.1 with below steps:

1. With image registry.portainer.io:443/test on machine, deploy a stack referencing it in portainer and see it deploys and runs
2. Add private registry to portainer
3. Push new image to registry from different host
4. Update service and see it goes to a failed state with no such image
5. Remove and re-add stack and it now works fine

We experience the same problem with:

Docker version 19.03.6, build 369ce74a3c Portainer: 1.23.2 Portainer Agent: 1.5.1

I tried a simple test of simply deployng a container from a password protected Artifactory repo without success.

cat ~/.docker/config.json
{
    "auths": {
      "https://docker-artifactory.XXX.com": {
          "auth": "c2t5bGFiLmFkbWluhOZnFUYXlSeUZaUeGsxM3kyaTJraXpaNXY1c1Bv",
          "email": "admin@XXX.com"
      },

We use artifactory for our registry I cut and pasted the credentials exactly out of the above config file.

Those credentials work fine in the docker cli but not with portainer, not even to deploy an individual container.

I also receive the ‘no Such Image’ error. If I then go on to the CLI and pull the container, I can then deploy the container just fine using the registry I have defined in Portainer.

This issue also prevents being able do deploy stacks using the Portainer UI as has been reported elsewhere, there is no portainer equivalent to the CLI docker stack deploy --with-registry-auth

FWIW:

We also have a an insecure registries stanza in our /etc/docker/daemon.json

cat /etc/docker/daemon.json
{
  "data-root": "/data/var/lib/docker",
  "insecure-registries": [
    "artifactory.services.xxx:8080"
  ],
  "default-address-pools": [
    {
      "scope": "local",
      "base": "172.16.0.0/12",
      "size": 23
    }
  ]
}

I am facing this issue as well. I had to update the auth credentials for the registry, and after doing so, any restarts to the stack, service, or container failed to pull the latest image with a “Not found” error. But I am able to pull the image from the Docker host machine using those credentials as well as from the “Images” tab of the Portainer UI. Here’s a screenshot showing the container’s failed attemps to pull the image followed by the successful attempt. I did not change any credentials during this time period. I simply manually pulled the image from the “Images” tab, and it was then able to pull it from cache.

@diegovilar Agreed. for the new users the message “No such image: xxxxxxx” , it takes time figure out what went wrong. If the error diagnosis can be better then, one can come straight to the issue and fix it. for example. "Image cannot be accessed. Add your image registry with the accesses credentials in the registry tab " or a similar message.

@thnk2wn thanks for your repro steps, i’ll need to have another go at reproducing this as it sounds like somewhere the registry credentials are out-of-sync with the stack and aren’t used

Additional notes on Artifactory custom registries.

In my posts to this issue above, i noted that I was unable to get Portainer custom registries to work with Portainer when using credentials copied from the ~/.docker/config.json file.

When I instead entered a UserID + API Key as credentials, everything started working.

This Artifactory Knowledge Base article was my clue.

Update: Because I had some trouble with my Gitlab instance in the last few days, I pushed the Image to Docker Hub (private repo) and encounter the exact same problem. I have entered my DockerHub credentials into Portainer but when updating the service I see “authorization required” errors in the journal of Docker and Portainer fails to update the servie with “No such image”.

But again when I manually go to the “Images” section of Portainer, he pulls the image just fine.

@pushpdeep we solved the problem. However, the issue was a bit different for us.

We are doing the initial deployment via Gitlab CI and used Gitlab CI’s temporary token to log in to the Docker registry. So the problem was that, once the token is expired (it expires in 30 min I believe) it was not possible to pull new images or update services from Portainer because it was using the temporary token and therefore throwing no such image error.

Now we are using the permanent token and Portainer is able to pull new images and updating services.

I want to thank @itsconquest for his time and effort

Glad we could resolve this!

@pushpdeep we solved the problem. However, the issue was a bit different for us.

We are doing the initial deployment via Gitlab CI and used Gitlab CI’s temporary token to log in to the Docker registry. So the problem was that, once the token is expired (it expires in 30 min I believe) it was not possible to pull new images or update services from Portainer because it was using the temporary token and therefore throwing no such image error.

Now we are using the permanent token and Portainer is able to pull new images and updating services.

I want to thank @itsconquest for his time and effort

FYI if its easier for you, you can chat to me on our slack server here while we debug this

I can currently reproduce this on 1.23.2. If I try to deploy a service to a Swarm it fails with “No such image: <image name>” unless I manually pull the image down from a private registry ahead of time with a docker pull <image name>.