compose: "docker-compose logs -f" stops outputting logs of service after some time
Currently I’m running a Docker Compose project with 20 services. One of these services is “gitlab/gitlab-ce:latest”, which is quite chatty in it’s logging output. [about 80 lines/min]
I always start my services with “docker-compose up -d” followed by a “docker-compose logs -f” The Problem is, that after a (until now) unspecified time the logging of the “gitlab” container/service is not outputted with “docker-compose logs -f”. But investigation of the actual logfile of the container/service shows that new log output is generated, but somehow ignored by “logs -f”. A termination of the command (via CTRL+C) and a new execution of the command “docker-compose logs -f” shows and follows again the output of the “gitlab” container/service for an unspecified time.
I realize that this is a specific scenario that I’m presenting. I will therefore try to find a minimal reproducible example of this behavior.
Additional Info:
docker-compose version 1.21.2, build a133471
docker-py version: 3.3.0
CPython version: 3.6.5
OpenSSL version: OpenSSL 1.0.1t 3 May 2016
Client:
Version: 18.03.0-ce
API version: 1.37
Go version: go1.9.4
Git commit: 0520e24
Built: Wed Mar 21 23:10:06 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm
Server:
Engine:
Version: 18.03.0-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.4
Git commit: 0520e24
Built: Wed Mar 21 23:08:35 2018
OS/Arch: linux/amd64
Experimental: false
Debian GNU/Linux 9 (stretch)
4.9.0-6-amd64 #1 SMP Debian 4.9.88-1 (2018-04-29) x86_64 GNU/Linux
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 19
- Comments: 23
I can still reproduce this lol
Edit: Update version
This is still a problem. Currently happening very frequently when using
nodemon
. Seems to happen when the log stays idle for some time.On Linux Ubuntu 20 With Docker version 20.10.12, build e91ed57 docker-compose version 1.29.2, build 5becea4c
I was facing the same issue with the node.js (nodemon) development environment, with “docker-compose logs -f server” logs were not appearing in the terminal after some time.
I set COMPOSE_HTTP_TIMEOUT=86400 in .env (1 day - which is fine for me in development mode) and it seems to be working fine now.
Running
COMPOSE_HTTP_TIMEOUT=86400 docker-compose logs -ft service1 service2
works here as well. No more having to restart docker-compose logs every few minutes.Just to clarify, I still have this issue and I too am currently on
docker-compose version 1.23.2, build 1110ad01
docker version 18.09.0
So, are no logs generated at all, is
docker log
not showing up anything or isdocker-compose logs -f
seizing to display any output after a while? The latter is the cause of this issue. if you have a different phenomenon I would suggest opening a separate issue.Further information regarding this issue:
It has now happened to many different services, all of which generate more output than their counterparts which continue to display their output via the
logs -f
command.It is especially prone to happen when one does a
docker-compose logs -f
after the containers have lived for a while (which means that all their past logs all displayed at once).This suggests that this issue is probably due to a race condition, which I haven’t figured out yet.
Just want to inform that I found the cause for my problem.
I had configured the docker logging options to
max-size: 5m
andmax-file: 1
, and every time the logfile reached the max size and was rotated, the trailing ofdocker-compose logs -f
would just stop/hang.Note that I do not have a timeout issue.
Turns out this is not a
docker-compose
specific issue, but adocker
issue in general, as it also happens withdocker logs -f
.Here is a related issue in the docker repo
My fix that works until now is to just increase the
max-file
setting to at least 2.I will leave this issue still open until I can confirm this issue/fix for sure, and link an issue to the docker repository.
Experienced on 1.24 and also experienced after downgrading to 1.23.1, as suggested above. Specifically I’m trying to use
docker-compose up
to show logs from a stack on an Ubuntu 16.04 dev workstation with versionsI traced the problem to that fact that if no logs come in within
COMPOSE_HTTP_TIMEOUT
seconds (defaults to 60) then all logs stop, forever. This seems to be alluded to in https://github.com/docker/compose/issues/3633#issuecomment-365101768 (“Unresponsive container operations … being addressed at the engine level … https://github.com/moby/moby/issues/35933”, but the linked issue is frankly TLDR).This you could handle also with
docker-compose
by doing followingdocker-compose logs -f service1 service2 service3 ...
Happening to me all the time - sometimes even very shortly after I start containers, where one of them stops showing in the log drain. Updated docker yesterday to Docker Desktop 4.4.2 (73305) (Docker for Mac)
Thanks @Greek64 for reporting back. Since https://github.com/moby/moby/issues/39235 seems to be in hand, and is any case a better place to track this issue I’m going to close this issue.
To anyone else who is seeing the same symptom, please check if you are affected by the engine issue @Greek64 has identified. If your issue wasn’t caused by that same underlying issue I would suggest opening fresh issues so we can track different problems which happen to cause similar symptoms separately rather than in one big issue which will soon get unwieldy.
Possible related issue: #4263