moby: docker logs -f stops when json-file rotates log file

Description

When Docker json-file log driver rotates the log file, applications streaming the log such as docker logs -f or logspout stop receiving new log events.

Steps to reproduce the issue:

  1. Start container that produces logs: docker run --rm -d alpine sh -c 'while true; do echo $(date); done'
  2. Start tailing logs: docker logs -f <container-id>
  3. Observe log file size at /var/lib/docker/containers/<container-id>/<container-id>-json.log

Describe the results you received:

When the log file rotates, docker logs -f output stops.

Describe the results you expected:

I expect docker logs -f to continue outputting logs until terminated.

Additional information you deem important (e.g. issue happens only occasionally):

Seems to be a duplicate of https://github.com/moby/moby/issues/23913 which was supposed to have been fixed through https://github.com/moby/moby/pull/24514 in 1.13.

Output of docker version:

$ docker version
Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.6
 Git commit:   9ee9f40
 Built:        Thu Apr 26 04:27:49 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.6
  Git commit:   9ee9f40
  Built:        Thu Apr 26 04:27:49 2018
  OS/Arch:      linux/amd64
  Experimental: false

Output of docker info:

$ docker info
Containers: 24
 Running: 5
 Paused: 0
 Stopped: 19
Images: 47
Server Version: 18.03.1-ce
Storage Driver: overlay
 Backing Filesystem: extfs
 Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: 3641oiewax7rd3re6mbt0uhto
 Is Manager: true
 ClusterID: 3dg2kejti78ucxky28skve8cv
 Managers: 3
 Nodes: 6
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 172.18.5.178
 Manager Addresses:
  172.18.5.136:2377
  172.18.5.178:2377
  172.18.5.248:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: v0.13.2 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  Profile: default
 selinux
Kernel Version: 4.14.59-coreos-r2
Operating System: Container Linux by CoreOS 1800.6.0 (Rhyolite)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.853GiB
Name: REDACTED
ID: EK5F:II7L:AE6M:JCUO:C4CT:TM5F:XL6Z:XL6P:FJ2M:2EBC:LC45:VJTJ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 103
 Goroutines: 273
 System Time: 2018-08-15T12:35:17.758625842Z
 EventsListeners: 3
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Registry Mirrors:
 https://REDACTED/
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

CoreOS stable channel on EC2.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 3
  • Comments: 16 (4 by maintainers)

Commits related to this issue

Most upvoted comments

Hi workaround for this scenario is to include “–log-opt max-file=3” flag as well (3 is just a random number you have any number depending upon your need) so to make it work,

do following 1- docker run --log-opt max-size=10m --log-opt max-file=3 --rm -d alpine sh -c ‘while true; do echo $(date); done’ 2- Start tailing logs: docker logs -f <container-id> 3- Observe log file size at /var/lib/docker/containers/<container-id>/<container-id>-json.log*

you will see 3 log files rotating at /var/lib/docker/containers/<container-id>/<container-id>-json.log /var/lib/docker/containers/<container-id>/<container-id>-json.log.1 /var/lib/docker/containers/<container-id>/<container-id>-json.log.2

and docker logs -f <container-id> is also pointing to the <container-id>-json.log,

Thanks Asim

That might work because that’s more or less what it does for max-file >= 2, I’ll give it a shot and open a PR if that’s OK