moby: journald restart crashes Docker

It appears that a journald restart on Redhat 7.2 crashes Docker. I was getting my logging settings in place when I noticed that after restarting journald my Docker daemon would crash. I backed things out to where even without any containers if I restarted journald Docker crashes and then restarts.

It is very easy to reproduce. Just restart journald.

$ systemctl restart  systemd-journald.service

Then check Docker processes. After just a few times you get this.

$ docker ps
An error occurred trying to connect: Get http:///var/run/docker.sock/v1.21/containers/json: EOF
$ docker ps
Error response from daemon: 404 page not found
$ docker ps
CONTAINER ID        IMAGE ...

You can see it recovers. However, if I have containers running it will stop all of them and even require me to manually remove contatiners that show up as Dead. I also have to remove the containers sometimes as they appear to be in a state where they will not start without errors.

docker version:

Client: Version: 1.9.1 API version: 1.21 Go version: go1.4.2 Git commit: a34a1d5 Built: Fri Nov 20 13:25:01 UTC 2015 OS/Arch: linux/amd64

Server: Version: 1.9.1 API version: 1.21 Go version: go1.4.2 Git commit: a34a1d5 Built: Fri Nov 20 13:25:01 UTC 2015 OS/Arch: linux/amd64

docker info:

Containers: 13 Images: 62 Server Version: 1.9.1 Storage Driver: devicemapper Pool Name: docker-253:0-67131623-pool Pool Blocksize: 65.54 kB Base Device Size: 107.4 GB Backing Filesystem: Data file: /dev/loop0 Metadata file: /dev/loop1 Data Space Used: 4.038 GB Data Space Total: 107.4 GB Data Space Available: 31.62 GB Metadata Space Used: 5.886 MB Metadata Space Total: 2.147 GB Metadata Space Available: 2.142 GB Udev Sync Supported: true Deferred Removal Enabled: false Deferred Deletion Enabled: false Deferred Deleted Device Count: 0 Data loop file: /var/lib/docker/devicemapper/devicemapper/data Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata Library Version: 1.02.107-RHEL7 (2015-10-14) Execution Driver: native-0.2 Logging Driver: json-file Kernel Version: 3.10.0-327.el7.x86_64 Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo) CPUs: 2 Total Memory: 7.64 GiB Name: vmtstcon01.roomandboard.com ID: 7J2R:VPDV:RZRX:NM3X:6JG3:5HGD:ZUIW:FGMW:RCJO:NULN:FS7Z:UD42 WARNING: bridge-nf-call-ip6tables is disabled

uname -a:

Linux vmtstcon01.roomandboard.com 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.):

vmware

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 17 (15 by maintainers)

Commits related to this issue

Most upvoted comments

The basic problem is when you run a unit file systemd hooks up stdout/stderr to the journal, if the journal goes away. These sockets will get closed, and when you write to them you will get a sigpipe. systemd says programs should ignore sigpipe on stdout and stderr, but golang seems to refuse to do this more then 10 times.

This bugzilla explains the behaviour from a systemd/journald point of view.

https://bugs.freedesktop.org/show_bug.cgi?id=84923