fluent-bit: Introducing rewrite_tag breaks multiline in 1.8.9

Bug Report

Describe the bug I have a pipeline: tail -> kubernetes -> multiline -> (omitted for brevity) -> es. I want to introduce rewrite_tag so that I have: tail -> kubernetes -> rewrite_tag -> multiline -> (omitted for brevity) -> es. However, this change breaks multiline parsing. The records are not concatenated AND Kubernetes tags are only left on lines that correspond to start_state.

To Reproduce

Download and extract gist: https://gist.github.com/starteleport/ed66f30b7161f185658e711a7a95a2a2
Compare outputs of fluent-bit -c fluent-bit-repro-norewrite.conf and fluent-bit -c fluent-bit-repro-rewrite.conf.

Expected behavior Both configs produce the same set of multiline records with Kubernetes tags being correctly set.

Screenshots Not applicable

Your Environment

Version used: reproduced on 1.8.9
Configuration: please see gist above
Environment name and version (e.g. Kubernetes? What version?): local
Server type and version:
Operating System and version: macOS Monterey
Filters and plugins: rewrite_tag, multiline, multiline_parser

Additional context This works on 1.8.6.

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 1
Comments: 25 (10 by maintainers)

Commits related to this issue

in_emitter: support async mode(#4325) in_emitter writes records to msgpack buffer directly from v1.8.9. It is to avoid exausting memory on high load. However it causes an issue when using mutiline pa... — committed to nokute78/fluent-bit by nokute78 3 years ago

Most upvoted comments

@starteleport Thank you. v1.8.8 buffers records at in_emitter and v1.8.9 doesn’t. It changes a number of records which are received filter_multiline at one time. I think it is root cause.

@ozzy-ext Does it work correctly on v1.8.8 ?

nokute78 on Nov 30, 2021

@ozzy-ext Does it work correctly on v1.8.8 ?

Yes, it does! But v1.8.9 has issue. It suits me! Good!

ozzy-ext on Dec 16, 2021

Yes, you are right, it’s only 1.8.9. I’ve tried many times with different versions and got it mixed up. It’s not even working with rewrite_tag on 1.8.7 and 1.8.8: I’ve got no records output in case rewrite_tag is present.

starteleport on Nov 23, 2021

Thank you.

I tested below config to simplify the issue. I think https://github.com/fluent/fluent-bit/pull/4128 causes this issue and it is merged from v1.8.9.

[SERVICE]
    Daemon Off
    Flush 1
    Log_Level debug
    Parsers_File custom_parsers-repro.conf
    HTTP_Server Off
    HTTP_Listen 0.0.0.0
    HTTP_Port 2020
    Health_Check Off

[INPUT]
    Name tail
    Path ./repro-external-api-745f757c59-ndgg8_staging_repro-external-api-4303caf1c98782551d123e21f7ea93b56f87b3c59c27e1d325cb126abf6eda6d.log
    multiline.parser docker, cri
    Tag kube.*
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On
    Refresh_Interval 5
    Read_from_Head true

[FILTER]
    Name record_modifier
    Match kube.*
    Record kubernetes staging

[FILTER]
    Name          rewrite_tag
    Match         kube.*
    Rule          kubernetes ^(staging|production)$ apps false
    Emitter_Name  apps_tag_rewriter

[FILTER]
    name                  multiline
    match                 apps
    multiline.key_content log
    multiline.parser      multiline_dotnet_console

[OUTPUT]
    Name stdout
    Match *

nokute78 on Nov 21, 2021