compose: "docker-compose logs -f" stops outputting logs of service after some time

Currently I’m running a Docker Compose project with 20 services. One of these services is “gitlab/gitlab-ce:latest”, which is quite chatty in it’s logging output. [about 80 lines/min]

I always start my services with “docker-compose up -d” followed by a “docker-compose logs -f” The Problem is, that after a (until now) unspecified time the logging of the “gitlab” container/service is not outputted with “docker-compose logs -f”. But investigation of the actual logfile of the container/service shows that new log output is generated, but somehow ignored by “logs -f”. A termination of the command (via CTRL+C) and a new execution of the command “docker-compose logs -f” shows and follows again the output of the “gitlab” container/service for an unspecified time.

I realize that this is a specific scenario that I’m presenting. I will therefore try to find a minimal reproducible example of this behavior.

Additional Info:

docker-compose version 1.21.2, build a133471
docker-py version: 3.3.0
CPython version: 3.6.5
OpenSSL version: OpenSSL 1.0.1t  3 May 2016

Client:
 Version:       18.03.0-ce
 API version:   1.37
 Go version:    go1.9.4
 Git commit:    0520e24
 Built: Wed Mar 21 23:10:06 2018
 OS/Arch:       linux/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:      18.03.0-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   0520e24
  Built:        Wed Mar 21 23:08:35 2018
  OS/Arch:      linux/amd64
  Experimental: false

Debian GNU/Linux 9 (stretch)
4.9.0-6-amd64 #1 SMP Debian 4.9.88-1 (2018-04-29) x86_64 GNU/Linux

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 19
Comments: 23

Most upvoted comments

I can still reproduce this lol

Docker version 20.10.7, build f0df35096d
docker-compose version 1.29.2, build unknown

Edit: Update version

+17

nathanfranke on Jun 28, 2021

This is still a problem. Currently happening very frequently when using nodemon. Seems to happen when the log stays idle for some time.

divmgl on May 10, 2019

On Linux Ubuntu 20 With Docker version 20.10.12, build e91ed57 docker-compose version 1.29.2, build 5becea4c

I was facing the same issue with the node.js (nodemon) development environment, with “docker-compose logs -f server” logs were not appearing in the terminal after some time.

I set COMPOSE_HTTP_TIMEOUT=86400 in .env (1 day - which is fine for me in development mode) and it seems to be working fine now.

vinaysudani on Jan 1, 2022

I set COMPOSE_HTTP_TIMEOUT=86400 in .env (1 day - which is fine for me in development mode) and it seems to be working fine now.

Running COMPOSE_HTTP_TIMEOUT=86400 docker-compose logs -ft service1 service2 works here as well. No more having to restart docker-compose logs every few minutes.

taisph on Jan 8, 2022

Just to clarify, I still have this issue and I too am currently on docker-compose version 1.23.2, build 1110ad01
docker version 18.09.0

Server: All servers don’t work, including a node server, golang server, adminer, postgres, none of them output anything.

So, are no logs generated at all, is docker log not showing up anything or is docker-compose logs -f seizing to display any output after a while? The latter is the cause of this issue. if you have a different phenomenon I would suggest opening a separate issue.

Further information regarding this issue:

It has now happened to many different services, all of which generate more output than their counterparts which continue to display their output via the logs -f command.
It is especially prone to happen when one does a docker-compose logs -f after the containers have lived for a while (which means that all their past logs all displayed at once).
This suggests that this issue is probably due to a race condition, which I haven’t figured out yet.

Greek64 on Dec 29, 2018

Just want to inform that I found the cause for my problem.
I had configured the docker logging options to max-size: 5m and max-file: 1, and every time the logfile reached the max size and was rotated, the trailing of docker-compose logs -f would just stop/hang.
Note that I do not have a timeout issue.

Turns out this is not a docker-compose specific issue, but a docker issue in general, as it also happens with docker logs -f.
Here is a related issue in the docker repo

My fix that works until now is to just increase the max-file setting to at least 2.

I will leave this issue still open until I can confirm this issue/fix for sure, and link an issue to the docker repository.

Greek64 on May 18, 2019

Experienced on 1.24 and also experienced after downgrading to 1.23.1, as suggested above. Specifically I’m trying to use docker-compose up to show logs from a stack on an Ubuntu 16.04 dev workstation with versions

docker==3.7.2
docker-compose==1.23.1
docker-ce:
  Installed: 5:18.09.3~3-0~ubuntu-xenial

I traced the problem to that fact that if no logs come in within COMPOSE_HTTP_TIMEOUT seconds (defaults to 60) then all logs stop, forever. This seems to be alluded to in https://github.com/docker/compose/issues/3633#issuecomment-365101768 (“Unresponsive container operations … being addressed at the engine level … https://github.com/moby/moby/issues/35933”, but the linked issue is frankly TLDR).

feltech on May 3, 2019

docker-compose logs has some uninteresting services logged […] Follows only containers you want to […]

This you could handle also with docker-compose by doing following docker-compose logs -f service1 service2 service3 ...

Greek64 on Jan 21, 2019

Happening to me all the time - sometimes even very shortly after I start containers, where one of them stops showing in the log drain. Updated docker yesterday to Docker Desktop 4.4.2 (73305) (Docker for Mac)

pjar on Jan 24, 2022

Thanks @Greek64 for reporting back. Since https://github.com/moby/moby/issues/39235 seems to be in hand, and is any case a better place to track this issue I’m going to close this issue.

To anyone else who is seeing the same symptom, please check if you are affected by the engine issue @Greek64 has identified. If your issue wasn’t caused by that same underlying issue I would suggest opening fresh issues so we can track different problems which happen to cause similar symptoms separately rather than in one big issue which will soon get unwieldy.

ijc on May 20, 2019

Possible related issue: #4263

Greek64 on May 20, 2018