moby: docker 20.10.4 on linux and windows breaks docker-compose run -T

Description

Upgrading from 20.10.3 to 20.10.4 on Windows 2019 (and also in Ubuntu 20.04) breaks docker-compose run -T. It never returns any output nor ends.

Please note that this works fine with 20.10.3.

Steps to reproduce the issue:

  1. docker-compose run -T hello

Describe the results you received:

None.

Describe the results you expected:

Hello World!

Additional information you deem important (e.g. issue happens only occasionally):

I’m using the docker binaries from https://github.com/rgl/docker-ce-windows-binaries-vagrant/releases

I’m using docker-compose 1.26.2.

I’m using this docker-compose file:

version: '3.8'
services:
  hello:
    image: mcr.microsoft.com/windows:1809
    command: cmd.exe /c "echo Hello World!"

Output of docker version:

Client:
 Version:           20.10.4
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        d3cb89ee53
 Built:             Fri Feb 26 20:00:01 2021
 OS/Arch:           windows/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.4
  API version:      1.41 (minimum version 1.24)
  Go version:       go1.13.15
  Git commit:       363e9a88a1
  Built:            Fri Feb 26 19:57:55 2021
  OS/Arch:          windows/amd64
  Experimental:     false

Output of docker info:

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 22
  Running: 0
  Paused: 0
  Stopped: 22
 Images: 100
 Server Version: 20.10.4
 Storage Driver: windowsfilter
  Windows:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: process
 Kernel Version: 10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434)
 Operating System: Windows Server 2019 Standard Evaluation Version 1809 (OS Build 17763.1757)
 OSType: windows
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.999GiB
 Name: windows
 ID: JTOV:XZLF:QOVU:7I3C:PUWJ:DGCD:TRBB:QRBV:AIT3:XIJH:YB4I:HD3S
 Docker Root Dir: C:\ProgramData\docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
  os=windows
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface

Additional environment details (AWS, VirtualBox, physical, etc.):

Docker is running in Windows 2019 in a qemu-kvm VM (as launched by https://github.com/rgl/gitlab-ci-vagrant).

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 34
  • Comments: 18 (7 by maintainers)

Commits related to this issue

Most upvoted comments

We’re also experiencing this issue. In our case we’re using docker-compose so it’s hanging on docker start:

compose.cli.main.call_docker: /usr/bin/docker start --attach --interactive 5952f64080b56c6262c56bb8965f30321ba00f66660ac6f7016b8050e794d7b5

Looking at the container, it’s just stuck in the “created” status indefinitely.

Rolling back to 20.10.3 resolves the issue.

$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 66
 Server Version: 20.10.4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-1024-aws
 Operating System: Ubuntu 20.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.18GiB
 Name: docker-host-1
 ID: URIR:W4P4:Q57M:TQ2P:W7DF:LA4E:2YJD:QNCV:K7DD:CYDQ:W44R:GCC2
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

$ docker version
Client: Docker Engine - Community
 Version:           20.10.4
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        d3cb89e
 Built:             Thu Feb 25 07:05:27 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.4
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       363e9a8
  Built:            Thu Feb 25 07:03:23 2021
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
+7
zacblazic on Feb 27, 2021

Using COMPOSE_INTERACTIVE_NO_CLI=1 worked for us.

+6
superdecimal on Mar 2, 2021

We’ve had major issues on CentOS 7, resulting in server outages. Here’s how we downgraded

sudo yum downgrade docker-ce-20.10.3-3.el7.x86_64
​sudo yum downgrade docker-ce-cli-20.10.3-3.el7.x86_64
+4
mendhak on Mar 1, 2021

@mendhak Thanks! Thas was useful. So difference looks to to be that for the reproducer, docker create does not have a --tty

Docker CLI 19.03:

$ docker create --name test alpine sh -c 'echo hello'
$ docker start --attach -i test
hello
$ docker rm test

Docker CLI 20.10

$ docker create --name test alpine sh -c 'echo hello'
$ docker start --attach -i test
# hangs

But on Docker 20.10 with a TTY attached (--tty on create);

$ docker create --name test --tty alpine sh -c 'echo hello'
$ docker start --attach -i test
hello
$ docker rm test

I narrowed it down to https://github.com/docker/cli/commit/3c87f01b1801789cb5d4aa470bd1d50f93869b59.

Reverting https://github.com/docker/cli/commit/3c87f01b1801789cb5d4aa470bd1d50f93869b59 (https://github.com/docker/cli/pull/2960, backport of https://github.com/docker/cli/pull/2929) fixes the problem.

I was actually looking at those changes earlier today (for other reasons); inclined to revert the PR (for now), while we dig further into what’s causing the issue.

/cc @cpuguy83

+3
thaJeztah on Mar 1, 2021

Downgrade only the CLI solves this issue, so at least this seems to be caused by the CLI component only.

Workaround for Ubuntu and derivatives:

$ sudo apt install 'docker-ce-cli=5:20.10.3~3-0~ubuntu-focal'
+2
lephuongbg on Mar 1, 2021

It works after upgrade to kernel 3.10.0-1160.24.1.el7.x86_64. Many thanks !

@thaJeztah @DrMeers

+1
detailyang on Apr 21, 2021

I confirm that docker 20.10.5 now works on linux and windows. Thank You!

+1
rgl on Mar 3, 2021

Thanks @markvr. Yes, it’s a bit odd, and we need to do more digging to see what causes the exact issue. I opened https://github.com/docker/cli/pull/2987 to revert that patch for now

+1
thaJeztah on Mar 1, 2021
Read more comments on GitHub
← moby: Docker 1.9.1 hanging at build step "Setting up ca-certificates-java"
moby: Docker 20.10.6: all containers stopped and cannot start if ipv6 is disabled on host →