moby: docker logs -f stops when json-file rotates log file
Description
When Docker json-file log driver rotates the log file, applications streaming the log such as docker logs -f
or logspout stop receiving new log events.
Steps to reproduce the issue:
- Start container that produces logs:
docker run --rm -d alpine sh -c 'while true; do echo $(date); done'
- Start tailing logs:
docker logs -f <container-id>
- Observe log file size at
/var/lib/docker/containers/<container-id>/<container-id>-json.log
Describe the results you received:
When the log file rotates, docker logs -f
output stops.
Describe the results you expected:
I expect docker logs -f
to continue outputting logs until terminated.
Additional information you deem important (e.g. issue happens only occasionally):
Seems to be a duplicate of https://github.com/moby/moby/issues/23913 which was supposed to have been fixed through https://github.com/moby/moby/pull/24514 in 1.13.
Output of docker version
:
$ docker version
Client:
Version: 18.03.1-ce
API version: 1.37
Go version: go1.9.6
Git commit: 9ee9f40
Built: Thu Apr 26 04:27:49 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm
Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.6
Git commit: 9ee9f40
Built: Thu Apr 26 04:27:49 2018
OS/Arch: linux/amd64
Experimental: false
Output of docker info
:
$ docker info
Containers: 24
Running: 5
Paused: 0
Stopped: 19
Images: 47
Server Version: 18.03.1-ce
Storage Driver: overlay
Backing Filesystem: extfs
Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: 3641oiewax7rd3re6mbt0uhto
Is Manager: true
ClusterID: 3dg2kejti78ucxky28skve8cv
Managers: 3
Nodes: 6
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 172.18.5.178
Manager Addresses:
172.18.5.136:2377
172.18.5.178:2377
172.18.5.248:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: v0.13.2 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
seccomp
Profile: default
selinux
Kernel Version: 4.14.59-coreos-r2
Operating System: Container Linux by CoreOS 1800.6.0 (Rhyolite)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.853GiB
Name: REDACTED
ID: EK5F:II7L:AE6M:JCUO:C4CT:TM5F:XL6Z:XL6P:FJ2M:2EBC:LC45:VJTJ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 103
Goroutines: 273
System Time: 2018-08-15T12:35:17.758625842Z
EventsListeners: 3
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Registry Mirrors:
https://REDACTED/
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
CoreOS stable channel on EC2.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 3
- Comments: 16 (4 by maintainers)
Commits related to this issue
- Fix issue with log rotation and max-file=1 Closes #37646 Signed-off-by: Alexandre Chaintreuil <alexandre.chaintreuil@datadoghq.com> — committed to achntrl/moby by achntrl 6 years ago
Hi workaround for this scenario is to include “–log-opt max-file=3” flag as well (3 is just a random number you have any number depending upon your need) so to make it work,
do following 1- docker run --log-opt max-size=10m --log-opt max-file=3 --rm -d alpine sh -c ‘while true; do echo $(date); done’ 2- Start tailing logs: docker logs -f <container-id> 3- Observe log file size at /var/lib/docker/containers/<container-id>/<container-id>-json.log*
you will see 3 log files rotating at /var/lib/docker/containers/<container-id>/<container-id>-json.log /var/lib/docker/containers/<container-id>/<container-id>-json.log.1 /var/lib/docker/containers/<container-id>/<container-id>-json.log.2
and docker logs -f <container-id> is also pointing to the <container-id>-json.log,
Thanks Asim
That might work because that’s more or less what it does for max-file >= 2, I’ll give it a shot and open a PR if that’s OK