portainer: Portainer fails to pull images with authenticated registry and breaks deployments
Bug description When using private images, deployments via webhook fail with “pull access denied for mycontainer, repository does not exist or may require ‘docker login’: denied: requested access to the resource is denied”
Expected behavior Redeployment via webhook should succees is properly authenticated
Portainer Logs This is the response I receive
{
"message": "Failed to update the stack",
"details": "failed to deploy a docker compose stack 25: failed to pull images of the stack: IMAGE_NAME Pulling IMAGE_NAME Error Error response from daemon: pull access denied for USERNAME/IMAGE_NAME, repository does not exist or may require 'docker login': denied: requested access to the resource is denied"
}
Steps to reproduce the issue:
- Setup portainer CE on host A
- Setup portainer environment on host B (docker standalone)
- Add docker hub as an authenticated registry in portainer (expectation is that this will always be used to pull anything from docker hub)
- Login to docker hub on both hosts
- Create a git stack on host B environment and setup a webhook
- Make changes to the repository and push, then call webhook
Result: webhook will fail with message below:
{
"message": "Failed to update the stack",
"details": "failed to deploy a docker compose stack 25: failed to pull images of the stack: IMAGE_NAME Pulling IMAGE_NAME Error Error response from daemon: pull access denied for USERNAME/IMAGE_NAME, repository does not exist or may require 'docker login': denied: requested access to the resource is denied"
}
Technical details:
- Portainer version: 2.18
- Docker version: 20.10.12
- Kubernetes version (managed by Portainer):
- Platform: Ubuntu 20.04
- Command used to start Portainer (
docker run -p 9443:9443 portainer/portainer):
version: "3.5"
services:
portainer:
container_name: portainer
image: portainer/portainer-ce:latest
restart: always
security_opt:
- no-new-privileges:true
tmpfs:
- /tmp
volumes:
- /etc/localtime:/etc/localtime:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./.portainer:/data
- Browser: chrome
- Use Case (delete as appropriate): Using Portainer at Home.
- Have you reviewed our technical documentation and knowledge base? Yes
Additional context Looks like portainer server always tries to pull the image locally (which makes no sense if it’s trying to deploy remotely), and somehow never uses the authenticated docker hub registry, it uses the anonymous one and fails. Solution would be to just leave the remote agent deal with this, but there is no option to get around this because the local pull is hardcoded. I tested the changes below with a custom image and it just works.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 1
- Comments: 17 (2 by maintainers)
for some reason, if i delete “latest” tag in my image definition, it works ok for me
I can confirm the same problem after upgrading Portainer to 2.20.3 BE.
We have GitLab server with mix of public and private projects with Docker registries. Credentials (username nad PAT) with access to all projects and registeries are configured in Portainer Registeries UI. I can see and pull both public and private images trough Portainer Images UI. And via
docker loginthe same credentials are saved on all Swarm nodes where I candocker pullboth public and private images.When I try to Update a Service in Portainer UI with option “Re-pull image” checked, then it does not pull private images. The same issue happens when using Portainer API
PUT /endpoints/{id}/forceupdateservice. Both methods worked fine in older Portainer versions. When it happens, syslog contains errors like:In that case Portainer fails to start Tasks with status Rejected and Error message: “No such image: …”. And they are getting rejected until they start on some Node with image from previous deployment. So instead of redeploy with new images it just restarts service using old images.
Partial workaround is to stop and start a Stack. In that case it pulls images fine. However only to nodes where it just started new Tasks. On other Nodes are left outdated images which may be used if Service for some reason migrates to other Nodes.
Only safe workaround is to manually pull images on all Swarm Nodes before updating a service with Portainer.
I can confirm this on a clean server (just installed from scratch). Running 2.19.1 CE.