moby: journald restart crashes Docker
It appears that a journald restart on Redhat 7.2 crashes Docker. I was getting my logging settings in place when I noticed that after restarting journald my Docker daemon would crash. I backed things out to where even without any containers if I restarted journald Docker crashes and then restarts.
It is very easy to reproduce. Just restart journald.
$ systemctl restart systemd-journald.service
Then check Docker processes. After just a few times you get this.
$ docker ps
An error occurred trying to connect: Get http:///var/run/docker.sock/v1.21/containers/json: EOF
$ docker ps
Error response from daemon: 404 page not found
$ docker ps
CONTAINER ID IMAGE ...
You can see it recovers. However, if I have containers running it will stop all of them and even require me to manually remove contatiners that show up as Dead. I also have to remove the containers sometimes as they appear to be in a state where they will not start without errors.
docker version
:
Client: Version: 1.9.1 API version: 1.21 Go version: go1.4.2 Git commit: a34a1d5 Built: Fri Nov 20 13:25:01 UTC 2015 OS/Arch: linux/amd64
Server: Version: 1.9.1 API version: 1.21 Go version: go1.4.2 Git commit: a34a1d5 Built: Fri Nov 20 13:25:01 UTC 2015 OS/Arch: linux/amd64
docker info
:
Containers: 13 Images: 62 Server Version: 1.9.1 Storage Driver: devicemapper Pool Name: docker-253:0-67131623-pool Pool Blocksize: 65.54 kB Base Device Size: 107.4 GB Backing Filesystem: Data file: /dev/loop0 Metadata file: /dev/loop1 Data Space Used: 4.038 GB Data Space Total: 107.4 GB Data Space Available: 31.62 GB Metadata Space Used: 5.886 MB Metadata Space Total: 2.147 GB Metadata Space Available: 2.142 GB Udev Sync Supported: true Deferred Removal Enabled: false Deferred Deletion Enabled: false Deferred Deleted Device Count: 0 Data loop file: /var/lib/docker/devicemapper/devicemapper/data Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata Library Version: 1.02.107-RHEL7 (2015-10-14) Execution Driver: native-0.2 Logging Driver: json-file Kernel Version: 3.10.0-327.el7.x86_64 Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo) CPUs: 2 Total Memory: 7.64 GiB Name: vmtstcon01.roomandboard.com ID: 7J2R:VPDV:RZRX:NM3X:6JG3:5HGD:ZUIW:FGMW:RCJO:NULN:FS7Z:UD42 WARNING: bridge-nf-call-ip6tables is disabled
uname -a
:
Linux vmtstcon01.roomandboard.com 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
Environment details (AWS, VirtualBox, physical, etc.):
vmware
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Comments: 17 (15 by maintainers)
The basic problem is when you run a unit file systemd hooks up stdout/stderr to the journal, if the journal goes away. These sockets will get closed, and when you write to them you will get a sigpipe. systemd says programs should ignore sigpipe on stdout and stderr, but golang seems to refuse to do this more then 10 times.
This bugzilla explains the behaviour from a systemd/journald point of view.
https://bugs.freedesktop.org/show_bug.cgi?id=84923