fluent-bit: [BUG] in_tail plugin isn't continue watch log file after logrotate was ran on k8s logs file.

I have run fluent-bit for k8s, but after run logrotate, in_tail is not watch log file, which has been rotated. My fluentbit config:

[SERVICE]
  Flush        5
  Daemon       Off
  Log_Level    debug
  Parsers_File parsers.conf

[INPUT]
  Name          tail
  Tag           kube.*
  Path          /log/*_default_*.log
  DB            /log/kube.db
  Mem_Buf_Limit 5MB
[OUTPUT]
  Name stdout
  Match *

My logrotate config

/var/lib/docker/containers/*/*.log {
    daily
    rotate 1
    copytruncate
    compress
    delaycompress
    missingok
    notifempty
}

I thinks something was wrong after logs file has changed outside container, how I reproduce: I run a fluent-bit containers in docker, mount volume [current_folder]:/log

Append log inside containers:

# Run this command inside fluent-bit container
$ echo '{"log":"hello world","stream":"stdout","time":"2017-08-14T07:16:44.331313977Z"}' >> message_default_inside.log
# This work!

append log outside of container

# Run this command outside of fluent-bit container
$ echo '{"log":"hello world","stream":"stdout","time":"2017-08-14T07:16:44.331313977Z"}' >> message_default_outside.log
# This not work!

Trying start, restart sometime.

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 27 (13 by maintainers)

Commits related to this issue

in_tail: when file is truncated, reset state (#375) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> — committed to fluent/fluent-bit by edsiper 6 years ago
in_tail: when file is truncated, reset state (#375) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> — committed to fluent/fluent-bit by edsiper 6 years ago
in_tail: when file is truncated, reset state (#375) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> — committed to micklonge/fluent-bit by edsiper 6 years ago
out stackdriver: add severity_key and update local_resource_id format (#375) * out_stackdriver: fix incorrect format of local_resource_id, add ending dot Signed-off-by: Jeff Luo <jeffluoo@google.c... — committed to rawahars/fluent-bit by JeffLuoo 4 years ago

Most upvoted comments

What about the copied file, would it be consume from start? If I had a log file named a.log which was half processed and was copied to a.1.log, the truncated a.log would be processed correctly, but what would happen to a.1.log? Does its content would be re-consumed or just ignored? Note that it’s possible that content in a.1.log is half processed which means the unprocessed parts should continue to be processed and the processed parts shouldn’t be re-consumed.

WqyJh on Nov 24, 2022

I challenge the similar behaviour. We discovered it’s related to logrotate “copytruncate” option.

copytruncate

          Truncate the original log file in place after creating a copy, instead of moving the old log file and optionally creating a new one.  It can be used when some program cannot be told to
          close  its logfile and thus might continue writing (appending) to the previous log file forever.  Note that there is a very small time slice between copying the file and truncating it,
          so some logging data might be lost.  When this option is used, the create option will have no effect, as the old log file stays in place.

Fluent bit should recognize number of lines in file, and if that is < then the previous value, it should re-read the file from scratch + reset it’s position (whatever to get un-blocked).

epcim on Feb 5, 2018

@rmoriz @epcim

I pushed some improvements on GIT master to handle file truncation. Would you please re-build and test ?

edsiper on Feb 16, 2018

note that when a third-party tool rotate a file Fluent Bit catch this event (which is a file rename), and what it does is to keep monitoring the rotated file for the next 5 seconds (Rotate_Wait option), after that is not longer monitored.

Now when a file is rotated, likely the original application that create the logs will re-create the file (same name), but in order to let Fluent Bit catch that file creation it needs to re-scan the path, this operation is handled by the Refresh_Interval option, by default it re-scan every 60 seconds, I suggest to keep this value low as 5 seconds.

If the issue mentioned do not address the problem explained above, please provide detailed steps to try to reproduce the problem.

edsiper on Sep 26, 2017