fluent-bit: Tail plug-in fails to keep track of state of file when parser is used
I have noticed tail plugin does not obey the state of the file when a parser is defined. Plugin will always start at the head of the file unless parser is blanked out. This issue can also be replicated using command line:
fluent-bit --parser=parsers.conf -i tail -p path=/var/log/syslog -p db=/var/log/xyz.sqlite -p parser=syslog-custom -o stdout
This command will work as expected if parser parameter is taken out. This is my sosreport:
[Fluent Bit]
Edition Community Edition
Version 0.12.12
Built Flags JSMN_PARENT_LINKS JSMN_STRICT FLB_HAVE_TLS FLB_HAVE_SQLDB FLB_HAVE_BUFFERING FLB_HAVE_FLUSH_LIBCO FLB_HAVE_FORK FLB_HAVE_PROXY_GO FLB_HAVE_REGEX FLB_HAVE_C_TLS FLB_HAVE_SETJMP FLB_HAVE_ACCEPT4 FLB_HAVE_INOTIFY
[Operating System]
Name Linux
Release 4.1.15
Version #3 SMP PREEMPT Fri Jul 7 11:18:13 PDT 2017
[Hardware]
Architecture armv7l
Processors 1
[Built Plugins]
Inputs cpu mem kmsg tail proc disk netif dummy head health serial stdin tcp mqtt lib forward random syslog
Filters grep stdout kubernetes parser record_modifier
Outputs counter es exit file forward http influxdb kafka-rest nats null plot stdout retry td lib flowcounter
[SERVER] Runtime configuration
Flush 5
Daemon Off
Log_Level Debug
[INPUT] Instance
Name tail.0 (tail, id=0)
Flags DYN_TAG
Threaded No
Tag tail.0
Path /var/log/syslog
db /var/log/landing_files.sqlite
Parser syslog-custom
[OUTPUT] Instance
Name stdout.0 (stdout, mask_id=1)
Match *
TLS Active No
Retry Limit 1
I am hoping I have done something wrong on my end. Thanks!
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 24 (11 by maintainers)
Commits related to this issue
- in_tail: if parser fails, pack raw text (#499) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> — committed to fluent/fluent-bit by edsiper 6 years ago
- in_tail: if parser fails, pack raw text (#499) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> — committed to fluent/fluent-bit by edsiper 6 years ago
- in_tail: if parser fails, pack raw text (#499) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> — committed to micklonge/fluent-bit by edsiper 6 years ago
- outputs: fix slack doc grammar (#499) Signed-off-by: Sanya Kochhar <kochhars@microsoft.com> — committed to rawahars/fluent-bit by SanyaKochhar 3 years ago
This might be a system/conf specific problem. I used the same set up on my ubuntu VM and it was working just fine in either mode. I will close this issue for now. Thank you.
@2x-bullet-time thanks for raising this issue.
I did some local tests and the problem that I found in in_tail plugin, is that if a parser fails against a log line (bad regex), that content is missed, the right behavior is to keep that information in “raw text” mode. I placed a fix for that behavior in GIT master and 0.12 branches.