moby: Docker service update --image "could not accessed on a registry to record its digest"
Hello,
I use docker swarm with
docker -v
Docker version 17.06.0-ce, build 02c1d87
My swarm is composed only of one node :
docker node ls 2c9zrmt995p3kxn0qi4sg2xyu * ${nodeName} Ready Active Leader
When i run docker service update --image i have this error :
docker service update --image=${REGISTRY}:5000/${IMAGE}:latest ${serviceName}
image ${REGISTRY}:5000/${IMAGE}:latest could not be accessed on a registry to record
its digest. Each node will access ${REGISTRY}:5000/${IMAGE}:latest independently,
possibly leading to different nodes running different
versions of the image.
{serviceName}
Since --detach=false was not specified, tasks will be updated in the background.
In a future release, --detach=false will become the default.
The file /home/${USER}/.docker/config.json exists and contains the auth for the container. When i run docker images, i see docker has been able to download images automatically for others services. I think the difference is for this service i have downloaded manually the image and after i have create the service.
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 26
- Comments: 28 (6 by maintainers)
Commits related to this issue
- Add flag to docker stack deploy to allow nodes access to registry https://github.com/moby/moby/issues/34153#issuecomment-316047924 — committed to vision-it/vision-roles by jacksgt 5 years ago
- Pass registry auth: https://github.com/moby/moby/issues/34153 — committed to tyler71/OwlChirp by deleted user a year ago
When updating services that need credentials to pull the image, you need to pass
--with-registry-auth
. Images pulled for a service take a different path than a regulardocker pull
, because the actualpull
is performed on each node in the swarm where an instance is deployed. To pull an image, the swarm cluster needs to have the credentials stored (so that the credentials can be passed on to the node it performs the pull on).Even though the “node” in this case is your local node, swarm takes the same approach (otherwise it would only be able to pull the image on the local node, but not on any of the other nodes).
Setting the
--with-registry-auth
option passes your locally stored credentials to the daemon, and stores them in the raft store. After that, the image digest is resolved (using those credentials), and the image is pulled on the node that the task is scheduled on (again, using the credentials that were stored in the service).Can you confirm if passing
--with-registry-auth
makes the problem go away?I still get the same error, i.e.:
image dockersamples/visualiser:stable could not be accessed on a registry to record its digest. Each node will access dockersamples/visualiser:stable independently, possibly leading to different nodes running different versions of the image.
Even when using --resolve-image=always and --with-registry-auth. I’m literally following the instructions from https://docs.docker.com/get-started/part5/#add-a-new-service-and-redeploy. Everything was working perfectly before. Does someone have any other idea on how to solve this?
So in case someone still struggles with this issue despite using
--with-registry-auth
, it’s probably because you are usingsudo -E
to preserve environmental variables. This leads to Docker not having access to root’s home folder with docker credentials, but your non-root user’s home directory. The solution is to usesudo su
and do everything as root. Otherwise, for every env variable you have to usesudo --preserve-env=YOUR_VARIABLE1 --preserve-env=YOUR_VARIABLE2...
, which is tedious.Yes it’s true i have already gone this scenarios but this time i have installed our application on a new virtual machine. We have a script which is in charge to deploy the application. It runs 4 docker stacks with the command line
docker stack deploy --with-registry-auth...
. I use this script on another environnement which deploy every night our application and for this environnement i have one manager and 2 workers. The manager does not host containers. With this same script i have deployed the same application but this time the images was not the latest. Docker service has downloaded images but not the latest version. I am sure that the image has been downloaded by docker service because the version is defined to<none>
.I use docker stack deploy and the option
--force
is not present to this command. https://docs.docker.com/engine/reference/commandline/stack_deploy/ Do you have a solution to force the downloading of the latest version every time withdocker stack deploy
?@majstrooker I’m having the same issue.
@thaJeztah thanks a lot adding --with-registry-auth solves my issue
FYI, for those that find this via search. The GitHub Container Registry https://ghcr.io/ is compliant with the spec and this should work correctly when connecting to
ghcr.io
but won’t work when trying withdocker.pkg.github.com
.I’ve double-checked auth in the config.json (file is identical on both nodes) and yet I have the same warning during the following command
To make things happen as expected I have to log in to each worker node and issue
docker image pull my_gitlab_private_registry/img
command beforehand.Pulling images manually makes me really sad. 😦
Any Solution yet ? –with-registry-auth doesn’t solve my problem.
Thanks for the update @clarkbw !
Update to this. The GitHub team is in the process of migrating all
docker.pkg.github.com
users in the background. When the migration is finisheddocker.pkg.github.com
andghcr.io
will be the same service with the same type of responses and only different domains.