logspout: Logspout stops logging after truncating a logfile from a container

For those who are dealing with this issue: We are trying this tool now: https://sematext.com/blog/2016/04/04/docker-cloud-monitoring-logging/ And it seems to be working.

Edit: This also works great on Docker Cloud

@jarrold @SqiSch @furkanayhan @sybrandy @ebr @say5 can you guys try gliderlabs/logspout:master with image ID 1097f19fae45 and report back?

I believe the problem is solved in logspout:master but you have to set an envvar like INACTIVITY_TIMEOUT = 5m for it to work.

I was having this problem before and it cleared up after I switched to master and set that variable. I’m using Docker 1.11.1-cs1

FWIW, we switched to journald log driver in docker (from json-file), and this is no longer an issue, because there’s no file to be truncated. If you are stuck using json-file driver, this is definitely an issue. I actually had to create a “helper” service that would restart all logspout containers hourly as a workaround (and of course there was still a potential of missing up to 1 hour of logs, in case the truncation happened right after a restart)

I still have this issue. I have installed logspout may 29th so I’m suppose to have the fix. After the logrotate occurs, I must restart logspout so I can have my log in logdna.com gliderlabs/logspout latest 87895b8ba614 7 days ago 22.61 MB

What infos you want me to provide to help you solve this?

@michaelshobbs I think that fixed it. My logs have been flowing for the last few hours. Thanks for this. I didn’t use your image ID but I did build a fresh custom image from master.

Docker 1.12.6, --log-opt max-file=2, master, INACTIVITY_TIMEOUT, still seeing freezes. In logspout logs I see: stopped with error: inactivity time exceeded timeout for the same container over and over again.

I believe the problem is solved in logspout:master but you have to set an envvar like INACTIVITY_TIMEOUT = 5m for it to work.

If I understand INACTIVITY_TIMEOUT = 5m correctly, this just means that logspout will miss 5 minutes of logs after a rotation, and then start logging again. Setting it to a much smaller value isn’t a great solution either since that will mean frequent restarts when logging is legitimately silent.

I’m currently working on expanding the test suite so we can more reliably accept PRs. #276 is a start

I was chatting with @progrium the other day, and the main issue why these bugs remain open is that there’s a lack of capacity to maintain this project. It seems as though they’d love to accept help from those who are willing and have some available cycles.

Anyone interested?

@patagona-afriemann Ugh, yeah, I forgot about that. I’m stuck in the CentOS/Systemd world, so that never even crossed my mind. My bad.

@olivergrahl It looks like gliderlabs pulled a workaround for this issue into the master branch back in November:

https://github.com/gliderlabs/logspout/commit/977986a9e30db2fb37ba6401f1c1ab4082de40d5

The patch was not pulled into latest. I just changed my docker service to pull from master so I guess I’ll see if it works.

I’m on Docker Cloud as well, experiencing the problem of logspout stopping to log from some containers after hours, from other containers after days, frequently. After trying many other possible logspout fixes (#235, #210) I found this issue and can confirm that Docker Cloud’s log rotation was the culprit. By default, Docker Cloud rotates logs at 10M file size, and Logspout will stop working immediately after a log gets rotated. I fixed this by monitoring the log directory and restarting Logspout when logs get rotated. With my current solution, logspout-restart, I may miss some logs for 10-20 seconds, but besides that, logging has finally been fixed for me.

@sybrandy my problem is that I am on Docker Cloud. They don’t support docker log drivers.