moby: Docker service update --image "could not accessed on a registry to record its digest"

Hello,

I use docker swarm with

docker -v
Docker version 17.06.0-ce, build 02c1d87

My swarm is composed only of one node : docker node ls 2c9zrmt995p3kxn0qi4sg2xyu * ${nodeName} Ready Active Leader

When i run docker service update --image i have this error :

docker service update --image=${REGISTRY}:5000/${IMAGE}:latest ${serviceName}
image ${REGISTRY}:5000/${IMAGE}:latest could not be accessed on a registry to record
its digest. Each node will access ${REGISTRY}:5000/${IMAGE}:latest independently,
possibly leading to different nodes running different
versions of the image.
{serviceName}
Since --detach=false was not specified, tasks will be updated in the background.
In a future release, --detach=false will become the default. 

The file /home/${USER}/.docker/config.json exists and contains the auth for the container. When i run docker images, i see docker has been able to download images automatically for others services. I think the difference is for this service i have downloaded manually the image and after i have create the service.

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 26
  • Comments: 28 (6 by maintainers)

Commits related to this issue

Most upvoted comments

When updating services that need credentials to pull the image, you need to pass --with-registry-auth. Images pulled for a service take a different path than a regular docker pull, because the actual pull is performed on each node in the swarm where an instance is deployed. To pull an image, the swarm cluster needs to have the credentials stored (so that the credentials can be passed on to the node it performs the pull on).

Even though the “node” in this case is your local node, swarm takes the same approach (otherwise it would only be able to pull the image on the local node, but not on any of the other nodes).

Setting the --with-registry-auth option passes your locally stored credentials to the daemon, and stores them in the raft store. After that, the image digest is resolved (using those credentials), and the image is pulled on the node that the task is scheduled on (again, using the credentials that were stored in the service).

Can you confirm if passing --with-registry-auth makes the problem go away?

I still get the same error, i.e.:

image dockersamples/visualiser:stable could not be accessed on a registry to record its digest. Each node will access dockersamples/visualiser:stable independently, possibly leading to different nodes running different versions of the image.

Even when using --resolve-image=always and --with-registry-auth. I’m literally following the instructions from https://docs.docker.com/get-started/part5/#add-a-new-service-and-redeploy. Everything was working perfectly before. Does someone have any other idea on how to solve this?

So in case someone still struggles with this issue despite using --with-registry-auth, it’s probably because you are using sudo -E to preserve environmental variables. This leads to Docker not having access to root’s home folder with docker credentials, but your non-root user’s home directory. The solution is to use sudo su and do everything as root. Otherwise, for every env variable you have to use sudo --preserve-env=YOUR_VARIABLE1 --preserve-env=YOUR_VARIABLE2..., which is tedious.

Yes it’s true i have already gone this scenarios but this time i have installed our application on a new virtual machine. We have a script which is in charge to deploy the application. It runs 4 docker stacks with the command line docker stack deploy --with-registry-auth.... I use this script on another environnement which deploy every night our application and for this environnement i have one manager and 2 workers. The manager does not host containers. With this same script i have deployed the same application but this time the images was not the latest. Docker service has downloaded images but not the latest version. I am sure that the image has been downloaded by docker service because the version is defined to <none>.

I use docker stack deploy and the option --force is not present to this command. https://docs.docker.com/engine/reference/commandline/stack_deploy/ Do you have a solution to force the downloading of the latest version every time with docker stack deploy?

@majstrooker I’m having the same issue.

@thaJeztah thanks a lot adding --with-registry-auth solves my issue

that’s a known issue with GitHub’s registry. The GitHub image registry currently isn’t compliant with the registry specification and doesn’t implement all options

FYI, for those that find this via search. The GitHub Container Registry https://ghcr.io/ is compliant with the spec and this should work correctly when connecting to ghcr.io but won’t work when trying with docker.pkg.github.com.

master node needs same auth to determine the image downloaded by the worker node

I’ve double-checked auth in the config.json (file is identical on both nodes) and yet I have the same warning during the following command

docker service update --with-registry-auth --force --image my_gitlab_private_registry/img service_name
image my_gitlab_private_registry/img:latest could not be accessed on a registry to record
its digest. Each node will access my_gitlab_private_registry/img:latest independently,
possibly leading to different nodes running different
versions of the image.

service_name
overall progress: 1 out of 1 tasks 
1/1: running   [==================================================>] 
verify: Service converged 

To make things happen as expected I have to log in to each worker node and issue docker image pull my_gitlab_private_registry/img command beforehand.

Pulling images manually makes me really sad. 😦

Any Solution yet ? –with-registry-auth doesn’t solve my problem.

Thanks for the update @clarkbw !

but won’t work when trying with docker.pkg.github.com

Update to this. The GitHub team is in the process of migrating all docker.pkg.github.com users in the background. When the migration is finished docker.pkg.github.com and ghcr.io will be the same service with the same type of responses and only different domains.