fluent-bit: Memory leak when defined several filters

Bug Report

fluentbit gets killed by OOMKilled when several filters with dynamic tags are defined in the configuration file.

Related issue #693 has been closed, but the issue occurs with Fluentbit 1.3.2 only when a bunch of filters are enabled.

Steps to reproduce the problem:

Expected behavior Fluentbit filters contains tags to match the input log file and does filtering based on the defined regex.

Issue does not occur when only one Filter is defined:

    [FILTER]
        Name    grep
        Match   *
        Regex   level ERROR

Screenshots

Fluentbit container is configured with 500MiB and 1Gi (two namespaces, both are failing with OOMKilled). The metrics:

  • the container is configured with 500MiB Fluentbit_500Mb
  • the container is configured with 1Gi Fluentbit_1Gi

Environment

  • Version used: 1.3.2
  • Configuration file:
apiVersion: v1
data:
  config.flb: |-
    [SERVICE]
        Flush       1
        Daemon      off
        Log_Level   debug
        Parsers_File    /etc/fluent-bit/parsers.conf

    @INCLUDE inputs.conf

    [FILTER]
        Name    grep
        Match   serverlog
        Regex   level ERROR|WARN

    [FILTER]
        Name    grep
        Match   mqlog
        Regex   level ERROR

    [FILTER]
        Name    grep
        Match   actlog
        Regex   level FAIL

    [FILTER]
        Name    grep
        Match   jboss-accesslog
        Regex   message .*HTTP\/.*?. ((4|5)0[0-9]) .*

    [OUTPUT]
        Name    as_prom_endpoint
        Match   *
        HTTP_Port   9145

  inputs.conf: |-
    [INPUT]
        Name    tail
        Tag     serverlog
        Path   /path_to_file/server*.log
        Key log
        Multiline On
        Parser_Firstline server_log_parser
        Refresh_Interval 2

    [INPUT]
        Name    tail
        Tag     mqlog
        Path    /path_to_file/messagingapi*.log
        Multiline On
        Parser_Firstline messaging_log_parser
        Key log
        Refresh_Interval 2

    [INPUT]
        Name    tail
        Tag     actlog
        Path   /path_to_file/act*.log
        Multiline On
        Parser_Firstline  act_log_parser
        Key log
        Refresh_Interval 2

    [INPUT]
        Name    tail
        Tag     jboss-accesslog
        Path    /path_to_file/access*.log
        Key log
        Parser jboss_access_log_parser
        Refresh_Interval 2
  parsers.conf: |-
    [PARSER]
        Name    server_log_parser
        Format  regex
        Regex   ^(?<timestamp>([0-9\/:,]+)+\s([0-9\/:,]+))\s(?<level>[^ ]*)\s\[(?<class>[a-zA-Z\.]+)]\s(?<message>[^ ][^\n]+(\n(((\t*)?at [^\n]+)|(Caused by: [^\n]+)|((\t*)?\.\.\. [^\n]+)))*)

    [PARSER]
        Name    messaging_log_parser
        Format  regex
        Regex   ^(?<timestamp>([0-9\/:;,.]+)) LogLevel=\"(?<level>[^ ;]*)\";\[class (?<class>[^ ]*)\](?<message>[^ ][^\n]+(\n(((\t*)?at [^\n]+)|(Caused by: [^\n]+)|((\t*)?\.\.\. [^\n]+)))*)

    [PARSER]
        Name    act_log_parser
        Format  regex
        Regex   ^(?<timestamp>([0-9\/:;.]+));(?<class>[^ ;]*);(?<level>[^ ;]*);(?<timespent>[\d]*)?;(?<message>[^ ][^\n]+(\n(((\t*)?at [^\n]+)|(Caused by: [^\n]+)|((\t*)?\.\.\. [^\n]+)))*)

    [PARSER]
        Name    jboss_access_log_parser
        Format  regex
        Regex   ^(?<timestamp>([0-9\/:,]+)+\s([0-9\/:,]+))\s(?<level>[^ ]*)\s+\[(?<class>[a-zA-Z\.]+)]\s(?<message>[^ ].*)
kind: ConfigMap

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 24 (14 by maintainers)

Most upvoted comments

thanks @TetianaKuziaieva and @fujimotos for identify the root cause of the problem! , I am taking a look at the code