moby: docker exec stop gives rpc error: code = 14 desc = grpc: the connection is unavailable, v17.04.0-ce-rc1
Description
docker exec and docker stop gives rpc error: code = 14 desc = grpc: the connection is unavailable
docker-containerd daemon somehow got killed after sometime of setup bringup.
dockerd and docker-containerd-shim process are running.
This issue is observed with docker 1.12.x, 1.13.x, 17.03.0 and the latest version 17.04.0-ce-rc1.
This issue is similar to this one https://github.com/docker/docker/issues/31074, which was closed, but the problem still exists.
The above issue was closed saying this was fix in this PR https://github.com/docker/docker/pull/31662
But the problem still exists with this PR.
Describe the results you received:
After few hours of setup bringup observed following things:
docker ps works.
dockerd daemon is up and running.
docker-containerd-shim of running containers exists.
docker-containerd daemon is not running.
sh-4.2# docker stop ac2a021285f8
Error response from daemon: Cannot stop container ac2a021285f8: Cannot kill container ac2a021285f893262cf308b6eb6d4533c7e3ff06779b8c529ebd914623a06141: rpc error: code = 14 desc = grpc: the connection is unavailable
sh-4.2# docker exec -it ac2a021285f8 sh
rpc error: code = 14 desc = grpc: the connection is unavailable
sh-4.2#
Additional information you deem important (e.g. issue happens only occasionally):
Running docker 17.04 on centos7.3 with kernel 4.8.3-1.el7.elrepo.x86_64
Used overlay2 driver with /mnt/docker as mount point foe docker data.
/mnt/docker is separate ebs volume with xfs filesystem with ftype=1
docker deamon start command :
/bin/dockerd -H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375 -g /mnt/docker --bip=172.17.0.1/24 --dns=172.17.0.1 --storage-driver=overlay2 --log-level=info --cluster-store=etcd://10.0.10.10:2379 --insecure-registry docker-registry.service.local:5040 --log-driver=syslog
Output of docker version:
Client:
Version: 17.04.0-ce-rc1
API version: 1.28
Go version: go1.7.5
Git commit: d2532c6
Built: Thu Mar 16 06:57:09 2017
OS/Arch: linux/amd64
Server:
Version: 17.04.0-ce-rc1
API version: 1.28 (minimum version 1.12)
Go version: go1.7.5
Git commit: d2532c6
Built: Thu Mar 16 06:57:09 2017
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
Containers: 16
Running: 16
Paused: 0
Stopped: 0
Images: 16
Server Version: 17.04.0-ce-rc1
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: syslog
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary:
containerd version: N/A (expected: 422e31ce907fd9c3833a38d7b8fdd023e5a76e73)
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.8.3-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 14.69GiB
Name: mcoverlay2-worker-1
ID: S4ID:SSDI:C6RJ:NZUO:JTBE:4X4O:4364:OTJD:INWW:OCE2:4DBF:H3WM
Docker Root Dir: /mnt/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Cluster Store: etcd://10.0.10.10:2379
Insecure Registries:
docker-registry.service.local:5040
127.0.0.0/8
Live Restore Enabled: false
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 39 (27 by maintainers)
@mlaventure I figured out the command is little different
docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc --debugThis is the output :