moby: docker stack deploy does not pull images from private registry on 17.03.0-ce-mac1

Description

I use “docker stack deploy -c deploy.yaml test” to deploy set of containers. All is working fine before. But after upgrading to 17.0.3.0-ce-mac1, docker would not pull the images that are in my private registry. So all services replied on private images failed with “No such image”. While the ones relying on public images are working fine.

Steps to reproduce the issue:

Have a private registry to host some docker images.
Define a docker compose file of containers using private images.
run docker stack deploy

Describe the results you received: The services replying on private images failed to start because of image not found.

Describe the results you expected: All services deployed correctly

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Version:      17.03.0-ce
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 10:40:59 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.03.0-ce
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   3a232c8
 Built:        Tue Feb 28 07:52:04 2017
 OS/Arch:      linux/amd64
 Experimental: true

Output of docker info:

Containers: 4
 Running: 4
 Paused: 0
 Stopped: 0
Images: 4
Server Version: 17.03.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
Swarm: active
 NodeID: 3fbs8nc8dr3qil5519touhj1l
 Is Manager: true
 ClusterID: tw5nbfyhim0dqccz3ioofh829
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 192.168.65.2
 Manager Addresses:
  192.168.65.2:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 977c511eda0925a723debdc94d09459af49d082a
runc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.12-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.855 GiB
Name: moby
ID: MV2W:LPMG:LE4Z:IC7P:53WM:LZDD:Q77E:JDHR:GELH:3W7O:5SKV:JWYI
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 75
 Goroutines: 213
 System Time: 2017-03-03T23:01:05.213716496Z
 EventsListeners: 5
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.): Docker On Mac

About this issue

Original URL
State: open
Created 7 years ago
Reactions: 38
Comments: 62 (9 by maintainers)

Links to this issue

Docker/ECR - image could not be accessed on a registry to record its digest

Most upvoted comments

Had a similar issue on 1.13.0, but it’s working if I add --with-registry-auth at the end of the “docker stack deploy” command.

+68

JnMik on Mar 9, 2017

would it make sense to make --with-registry-auth a default to docker stack deploy, I mean under which scenario we don’t want the workers/managers to not have the creds?

+14

jmarcos-cano on Sep 27, 2018

I’ve been running production swarms for over 2 years (that is probably more than 6 docker engine version upgrades) and have had no problem at all, this is my sweet spot

docker stack deploy --prune --with-registry-auth --resolve-image=always  -c docker-compose.yml MY_STACK

jmarcos-cano on Oct 4, 2019

Broken again in 17.09.0-ce specifying --with-registry-auth no longer working. I’ve tried with and without and receive the error “No such image …”.

Client: Version: 17.09.0-ce API version: 1.32 Go version: go1.8.3 Git commit: afdb6d4 Built: Tue Sep 26 22:41:23 2017 OS/Arch: linux/amd64

Server: Version: 17.09.0-ce API version: 1.32 (minimum version 1.12) Go version: go1.8.3 Git commit: afdb6d4 Built: Tue Sep 26 22:42:49 2017 OS/Arch: linux/amd64 Experimental: false

Insanityisnice on Nov 16, 2017

Looks like fixed in 17.06.0

soar on Jul 11, 2017

I’m seeing what I believe is the same problem on a cloud formation deployed swarm cluster on AWS. I have:

Docker version 17.03.0-ce, build 60ccb22

When i do docker stack ps, I see:

No such image: my_username/my_private_repository

I got past this with:

--with-registry-auth

davidthewatson on Mar 21, 2017

Yes @soar you are correct. I just found that you have to specify --with-registry-auth to make it work with docker stack command in 17.06.0

ChadCorsentino on Jul 11, 2017

Still see this issue With versions 17.10 & 18.06 --with-registry-auth does not work Pulling locally from these workers work config.json authes looks ok with “https://” prefix

haimari on May 16, 2018

Working for me only with –with-registry-auth even in a just_one_manager_node.

You need to be logged in (docker login) or it fails silently. I’m not sure if intended behaviour or not, as the documentation doesn’t mention.

Client:
 Version:	17.12.0-ce-rc4
 API version:	1.35
 Go version:	go1.9.2
 Git commit:	6a2c058
 Built:	Wed Dec 20 15:53:52 2017
 OS/Arch:	darwin/amd64

Server:
 Engine:
  Version:	17.12.0-ce-rc4
  API version:	1.35 (minimum version 1.12)
  Go version:	go1.9.2
  Git commit:	6a2c058
  Built:	Wed Dec 20 15:59:49 2017
  OS/Arch:	linux/amd64
  Experimental:	true

aaparmeggiani on Dec 29, 2017

This did work for me with the following version. Had to use –with-registry-auth though.

~ $ docker version Client: Version: 17.09.0-ce API version: 1.32 Go version: go1.8.3 Git commit: afdb6d4 Built: Tue Sep 26 22:39:28 2017 OS/Arch: linux/amd64

Server: Version: 17.09.0-ce API version: 1.32 (minimum version 1.12) Go version: go1.8.3 Git commit: afdb6d4 Built: Tue Sep 26 22:45:38 2017 OS/Arch: linux/amd64 Experimental: true

pennyandsean on Dec 11, 2017

Same problem here with docker running on CentOS

ChadCorsentino on Jul 11, 2017

Same issue with:

Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:04:27 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:04:27 2017
 OS/Arch:      linux/amd64
 Experimental: false

Tried with --with-registry-auth and without - same problem. But docker service create ... --with-registry-auth works perfectly with same repository. Official Docker Hub account used with official Hub Registry.

soar on May 29, 2017

+1 I’m able to reproduce it too!

saamalik on Mar 3, 2017

That looks to be the Red Hat fork of Docker, which has many changes in this area (and I’ve seen bugs reported that were caused by those changes); might be best to open a ticket in the Red Hat or CentOS issue tracker

thaJeztah on Feb 3, 2020

worked with 18.03.0-ce

sallespro on Apr 24, 2018

I have the same problem with Docker for Mac version 17.06.0-rc4-ce-mac15 (18357). –with-registry-auth allows the images to be pulled.

lucj on Jun 21, 2017

I have reproduced this on a deployed swarm cluster on azure as well.

Docker version 17.03.0-ce, build 60ccb22

using --with-registry-auth did not work.

jsturtevant on Mar 25, 2017

Hi, after I upgrade to Version 17.03.0-ce-mac2 (15654), I still see the same issues I original reported. And even the workaround not work anymore.

I have done a diagnostics, 17455168-32A4-4D0F-A89D-08DD2199CF95

zfeng-cliqr on Mar 16, 2017