fluent-bit: Memory leak when defined several filters
Bug Report
fluentbit
gets killed by OOMKilled when several filters with dynamic tags are defined in the configuration file.
Related issue #693 has been closed, but the issue occurs with Fluentbit 1.3.2 only when a bunch of filters are enabled.
Steps to reproduce the problem:
Expected behavior Fluentbit filters contains tags to match the input log file and does filtering based on the defined regex.
Issue does not occur when only one Filter is defined:
[FILTER]
Name grep
Match *
Regex level ERROR
Screenshots
Fluentbit container is configured with 500MiB and 1Gi (two namespaces, both are failing with OOMKilled). The metrics:
- the container is configured with 500MiB
- the container is configured with 1Gi
Environment
- Version used: 1.3.2
- Configuration file:
apiVersion: v1
data:
config.flb: |-
[SERVICE]
Flush 1
Daemon off
Log_Level debug
Parsers_File /etc/fluent-bit/parsers.conf
@INCLUDE inputs.conf
[FILTER]
Name grep
Match serverlog
Regex level ERROR|WARN
[FILTER]
Name grep
Match mqlog
Regex level ERROR
[FILTER]
Name grep
Match actlog
Regex level FAIL
[FILTER]
Name grep
Match jboss-accesslog
Regex message .*HTTP\/.*?. ((4|5)0[0-9]) .*
[OUTPUT]
Name as_prom_endpoint
Match *
HTTP_Port 9145
inputs.conf: |-
[INPUT]
Name tail
Tag serverlog
Path /path_to_file/server*.log
Key log
Multiline On
Parser_Firstline server_log_parser
Refresh_Interval 2
[INPUT]
Name tail
Tag mqlog
Path /path_to_file/messagingapi*.log
Multiline On
Parser_Firstline messaging_log_parser
Key log
Refresh_Interval 2
[INPUT]
Name tail
Tag actlog
Path /path_to_file/act*.log
Multiline On
Parser_Firstline act_log_parser
Key log
Refresh_Interval 2
[INPUT]
Name tail
Tag jboss-accesslog
Path /path_to_file/access*.log
Key log
Parser jboss_access_log_parser
Refresh_Interval 2
parsers.conf: |-
[PARSER]
Name server_log_parser
Format regex
Regex ^(?<timestamp>([0-9\/:,]+)+\s([0-9\/:,]+))\s(?<level>[^ ]*)\s\[(?<class>[a-zA-Z\.]+)]\s(?<message>[^ ][^\n]+(\n(((\t*)?at [^\n]+)|(Caused by: [^\n]+)|((\t*)?\.\.\. [^\n]+)))*)
[PARSER]
Name messaging_log_parser
Format regex
Regex ^(?<timestamp>([0-9\/:;,.]+)) LogLevel=\"(?<level>[^ ;]*)\";\[class (?<class>[^ ]*)\](?<message>[^ ][^\n]+(\n(((\t*)?at [^\n]+)|(Caused by: [^\n]+)|((\t*)?\.\.\. [^\n]+)))*)
[PARSER]
Name act_log_parser
Format regex
Regex ^(?<timestamp>([0-9\/:;.]+));(?<class>[^ ;]*);(?<level>[^ ;]*);(?<timespent>[\d]*)?;(?<message>[^ ][^\n]+(\n(((\t*)?at [^\n]+)|(Caused by: [^\n]+)|((\t*)?\.\.\. [^\n]+)))*)
[PARSER]
Name jboss_access_log_parser
Format regex
Regex ^(?<timestamp>([0-9\/:,]+)+\s([0-9\/:,]+))\s(?<level>[^ ]*)\s+\[(?<class>[a-zA-Z\.]+)]\s(?<message>[^ ].*)
kind: ConfigMap
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 24 (14 by maintainers)
thanks @TetianaKuziaieva and @fujimotos for identify the root cause of the problem! , I am taking a look at the code