portainer: pulling from custom registry credentials are ignored

Before you start please confirm the following.

Problem Description

When pulling from a custom registry a fake error message is shown and pulling is refused.

Expected Behavior

The images should be pulled.

Actual Behavior

After upgrading to Portainer BE 2.27.0 LTS when attempting to pull from a custom registry the fake error Failed to pull images of the stack: compose pull operation failed: Error response from daemon: Head "https://registry.something.com/v2/someproject/backend/manifests/latest": no basic auth credentials is shown. It is not being attempted to pull from the registry.

When doing docker login and docker pull on the server Portainer is running on it works as expected. The credentials are valid, the login is done and the image is pulled.

Unsuccessful attempts to fix:

  • recreate Portainer container
  • reboot server
  • delete registry from Portainer configuration and re-adding it
  • trying with different registry user/password

Steps to Reproduce

Error cannot be reliably reproduced.

Portainer logs or screenshots

2025/02/23 01:53PM ??? github.com/docker/cli@v27.4.0+incompatible/cli/streams/out.go:20 > backend Error Head "https://registry.something.com/v2/someproject/backend/manifests/latest": no basic auth credentials |

Portainer version

2.27.0

Portainer Edition

Business Edition (BE/EE) with 5NF / 3NF license

Platform and Version

Docker version 27.3.1, build 3ab4256

OS and Architecture

VMware Photon OS 5.0

Browser

No response

What command did you use to deploy Portainer?


Additional Information

No response

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 2
  • Comments: 39

Most upvoted comments

Thanks all. I have raised this internally for investigation.

internal: BE-11674

Same in 2.27.6 as obviously at this point Portainer doesn’t want to fix this severe bug.

@Nick-Portainer What the hell is going on? Why is this bug ignored?

Issue still there in 2.27.5 and I guess we have to life with this beeing permanently broken now.

I also have the same issue than everybody else. I can deploy a stack from a github repo docker-compose.yml, using a AWS ECR repo , it works fine which means my credentials are fine, but I cant pull and redeploy. I am using 2.27.3 LTs BE

I am experiecing similar issue after updating to Community Edition 2.27.3 LTS today. This upgrade broke the re-pull image and redeploy feature for certain images.

Failed to pull images of the stack: compose pull operation failed: Error response from daemon: Head “https://registry-1.docker.io/v2/kometateam/kometa/manifests/nightly”: unauthorized: incorrect username or password

I’m having a similar issue on 2.27.2 LTS

  1. It only happens when redeploying a stack.
  2. Pulling the image directly from the images menu, works fine
  3. I do have multiple registries configured
  4. I’m using an admin user

Failed to pull images of the stack: compose pull operation failed: Error response from daemon: Head "https://registryv2.motorsync.site/v2/motorsync/website/manifests/release": no basic auth credentials

@Nick-Portainer @oscarzhou-portainer The issue I reported is absolutely not resolved in 2.27.9 LTS, which is as of 04.07.2025 8:30 UTC the latest version. I have also tested by manually going forward to 2.31.3 STS. The issue is still there. I only have one custom registry. The password for it never changed. DockerHub has no credentials. Even after stopping all Stacks and attempting to pull and redeploy fails.

@Nick-Portainer Please fix this bug. This issue is not almost 2 months old and nothing was done to fix this. It is a horrible problem to our company that Portainer is fundamentally broken. The “Images” view can only be accessed by admins and as “pull and redeploy” is broken, only admins can effectifely redeploy stacks. So every time someone needs to redeploy a stack, the admin must be contacted.

ok, maybe this isn’t a bug… I had to give additional access to the custom registry to my user (i’m not operating as admin at all cases!). As admin go to ‘environment’ -> ‘Host’ -> Registries -> ‘Manage access’

This solved my problem. I’m using CE 2.27.2 LTS.

It does not work even as an admin in BE 2.27.2 LTS. This issue has nothing to do with permissions.

I can confirm the issue. I run portainer BE 2.27.1 LTS. The issue only occurs in the context of images that are pulled from an registry that requires authorization AND if the pull is triggered on the stack level. Triggering a re-pull using a stacks webhook gives no error (answer is “204 No Content”), but images are not re-pulled.

This makes GitOps for stacks with images from non-public registries unusable at the moment.

Note/Workaround: A re-pull of the image through the container UI (“Recreate”->“Re-pull”) or a “Container webhook” does work. So the registry with authorization is correctly configured.

Same for my instances. Exact behavior as described as I reported ages ago in this thread.

Em sex., 4 de jul. de 2025 às 08:01, WingZeroh @.***> escreveu:

WingZeroh left a comment (portainer/portainer#12551) https://github.com/portainer/portainer/issues/12551#issuecomment-3035609486

I echo this. It’s 100% broken with the latest versions starting with 2.27.9.It’s starting to seem like the development team does not know how to triage a fix.

For the time being, I ended up just running an alternative open source product and mixing usage with Synology’s Container Manager. I’m starting to like it.

@Nick-Portainer https://github.com/Nick-Portainer @oscarzhou-portainer https://github.com/oscarzhou-portainer The issue I reported is absolutely not resolved in 2.27.9 LTS, which is as of 04.07.2025 8:30 UTC the latest version. I have also tested by manually going forward to 2.31.3 STS. The issue is still there. I only have one custom registry. The password for it never changed. DockerHub has no credentials. Even after stopping all Stacks and attempting to pull and redeploy fails.

— Reply to this email directly, view it on GitHub https://github.com/portainer/portainer/issues/12551#issuecomment-3035609486, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANWUIYNXV3NDI5NVHQDPBTD3GZNHTAVCNFSM6AAAAABXWKPGMGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTAMZVGYYDSNBYGY . You are receiving this because you were mentioned.Message ID: @.***>

I echo this. It’s 100% broken with the latest versions starting with 2.27.9.It’s starting to seem like the development team does not know how to triage a fix.

For the time being, I ended up just running an alternative open source product and mixing usage with Synology’s Container Manager. I’m starting to like it.

@Nick-Portainer @oscarzhou-portainer The issue I reported is absolutely not resolved in 2.27.9 LTS, which is as of 04.07.2025 8:30 UTC the latest version. I have also tested by manually going forward to 2.31.3 STS. The issue is still there. I only have one custom registry. The password for it never changed. DockerHub has no credentials. Even after stopping all Stacks and attempting to pull and redeploy fails.

Hi @pmharris77,

We’ve tested with a single custom registry instance and have been unable to reproduce the issue. I’ve personally been using a custom private repository across multiple releases without encountering any problems.

There was a separate issue mentioned in this thread involving two registries having their authentication mixed up, which we’ve been working on. However, we have not been able to reproduce any issues when using a single registry.

Is there anything extra inside your environment outside of the typical one?

@Nick-Portainer I do have two custom registries on my account (dockerhub.io and ghcr.io), but only one registry type is used on any given stack, no mixing.

We’ve managed to solve the issue.

The problem started after we changed the password for our private registry. Although we updated the credentials in Portainer, all existing stacks that were previously deployed using the old password were no longer able to pull updated images.

What worked for us was: Stop the stack and then Update the stack. This process removes all services in the stack and recreates them from scratch. After this, image updates started working again from within each service, without issues.

Environment:

  • Docker version 26.1.4, build 5650f9b
  • Debian GNU/Linux 10 (buster)
  • Portainer 2.27.8 LTS

@rafaelreis-r Thanks for the bump. I have been keeping an eye on this and yes we are planning a fix for this in a future release. I can not provide an ETA just yet

I just tested with the latest 2.27.2 version, but no change. I didn’t have the courage to downgrade…

@pmharris77 , I first thought the issue was #9040, but then I found this one which is exactly my issue, it was working correctly with previous ( LTS) version, but the 2.27.X broke this.

@Nick-Portainer Add a docker registry for docker hub and put your credentials in (username / token). I only have this authorized registry for docker hub, so no possibility for portainer to download unauthorized.

I use this stack for watchtower:

version: "3.9"
services:
  watchtower:
    image: containrrr/watchtower:latest
    container_name: WATCHTOWER
    hostname: watchtower
    mem_limit: 512m
    mem_reservation: 128m
    cpu_shares: 512
    security_opt:
      - no-new-privileges=true
    read_only: true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      TZ: Europe/Zurich
      WATCHTOWER_CLEANUP: true # Remove old images after updating
      WATCHTOWER_REMOVE_VOLUMES: false # Remove attached volumes after updating
      DOCKER_API_VERSION: 1.47 # SSH docker version 1.41 for Docker engine version 20.10 - 1.43 for Docker engine version 24 - 1.45 for Docker engine version 26.1
      WATCHTOWER_INCLUDE_RESTARTING: true # Restart containers after update
      WATCHTOWER_INCLUDE_STOPPED: false # Update stopped containers
      WATCHTOWER_SCHEDULE: "0 0 */2 * * *" # Update & Scan containers every 2 hours
      WATCHTOWER_LABEL_ENABLE: false
      WATCHTOWER_ROLLING_RESTART: true
      WATCHTOWER_TIMEOUT: 30s
      WATCHTOWER_LOG_FORMAT: pretty
    network_mode: bridge
    restart: on-failure:5

When I know try to deploy this stack and select that it should re-pull the image, I get the following error:

Failed  to pull images of the stack: compose pull operation failed: Error  response from daemon: Head  "https://registry-1.docker.io/v2/containrrr/watchtower/manifests/latest":  unauthorized: incorrect username or password

Hope this helps to reproduce the issue.