fluent-bit: Tail plug-in fails to keep track of state of file when parser is used

I have noticed tail plugin does not obey the state of the file when a parser is defined. Plugin will always start at the head of the file unless parser is blanked out. This issue can also be replicated using command line:

fluent-bit --parser=parsers.conf -i tail -p path=/var/log/syslog -p db=/var/log/xyz.sqlite -p parser=syslog-custom -o stdout

This command will work as expected if parser parameter is taken out. This is my sosreport:


[Fluent Bit]
    Edition             Community Edition
    Version             0.12.12
    Built Flags          JSMN_PARENT_LINKS JSMN_STRICT FLB_HAVE_TLS FLB_HAVE_SQLDB FLB_HAVE_BUFFERING FLB_HAVE_FLUSH_LIBCO FLB_HAVE_FORK FLB_HAVE_PROXY_GO FLB_HAVE_REGEX FLB_HAVE_C_TLS FLB_HAVE_SETJMP FLB_HAVE_ACCEPT4 FLB_HAVE_INOTIFY

[Operating System]
    Name                Linux
    Release             4.1.15
    Version             #3 SMP PREEMPT Fri Jul 7 11:18:13 PDT 2017

[Hardware]
    Architecture        armv7l
    Processors          1

[Built Plugins]
    Inputs              cpu mem kmsg tail proc disk netif dummy head health serial stdin tcp mqtt lib forward random syslog
    Filters             grep stdout kubernetes parser record_modifier
    Outputs             counter es exit file forward http influxdb kafka-rest nats null plot stdout retry td lib flowcounter

[SERVER] Runtime configuration
    Flush               5
    Daemon              Off
    Log_Level           Debug

[INPUT] Instance
    Name                tail.0 (tail, id=0)
    Flags               DYN_TAG
    Threaded            No
    Tag                 tail.0
    Path                /var/log/syslog
    db                  /var/log/landing_files.sqlite
    Parser              syslog-custom


[OUTPUT] Instance
    Name                stdout.0 (stdout, mask_id=1)
    Match               *
    TLS Active          No
    Retry Limit         1

I am hoping I have done something wrong on my end. Thanks!

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 24 (11 by maintainers)

Commits related to this issue

Most upvoted comments

This might be a system/conf specific problem. I used the same set up on my ubuntu VM and it was working just fine in either mode. I will close this issue for now. Thank you.

@2x-bullet-time thanks for raising this issue.

I did some local tests and the problem that I found in in_tail plugin, is that if a parser fails against a log line (bad regex), that content is missed, the right behavior is to keep that information in “raw text” mode. I placed a fix for that behavior in GIT master and 0.12 branches.