fluent-bit: Errors and crashing after upgrading to 1.8.12 version
Bug Report
Describe the bug After upgrading to 1.18.12 the pod isn’t sending logs. Also there are huge amount of the errors in the log : [error] [multiline] expected MAP type in first line state buffer Eventually the pod consumes high amount of memory and crashing.
The config: `[SERVICE] Flush 1 Daemon Off Log_Level info Parsers_File parsers.conf Parsers_File custom_parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020
[INPUT] Name tail Path /var/log/containers/.log multiline.parser docker Tag kube. Mem_Buf_Limit 60MB Buffer_Max_Size 10MB Read_from_head true DB /var/run/flbkube.db Skip_Long_Lines On Skip_Empty_Lines On Refresh_Interval 5
[INPUT] Name systemd Tag host.* Systemd_Filter _SYSTEMD_UNIT=kubelet.service Read_From_Tail On
[FILTER] Name kubernetes Match kube.* Merge_Log Off Keep_Log On K8S-Logging.Parser On K8S-Logging.Exclude On Annotations Off Labels Off
[FILTER] name multiline match kube.* multiline.key_content log multiline.parser redacted_management_multi_line
[FILTER] name multiline match kube.* multiline.key_content log multiline.parser redacted_analyzer_multi_line
[FILTER] Name parser parser nginx parser redacted_management Match kube.* Key_Name log Reserve_Data On Preserve_Key On
[FILTER] Name parser parser redacted_management_nginx Match kube.* Key_Name log Reserve_Data On Preserve_Key On
[FILTER] Name parser parser redacted_analyzer Match kube.* Key_Name log Reserve_Data On Preserve_Key On [FILTER] Name modify Match * # Hard_rename log message Add ClusterName ${ELASTIC_USER}
[FILTER] Name modify Match host.* Add nodeName ${kubeNodeName}
[OUTPUT] Name es Match kube.* Host ${ELASTIC_HOST} Port ${ELASTIC_PORT} HTTP_User ${ELASTIC_USER} HTTP_Passwd ${ELASTIC_PASSWORD} tls On tls.verify ${ELASTIC_VERIFY_TLS} Logstash_Format On logstash_prefix_key $kubernetes[‘namespace_name’] Retry_Limit False Generate_ID On
[OUTPUT] Name es Match host.* Host ${ELASTIC_HOST} Port ${ELASTIC_PORT} HTTP_User ${ELASTIC_USER} HTTP_Passwd ${ELASTIC_PASSWORD} tls On tls.verify ${ELASTIC_VERIFY_TLS} Logstash_Format On Logstash_Prefix node Retry_Limit False Generate_ID On
[PARSER] Name docker_no_time Format json Time_Keep Off Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L
[PARSER] Name redacted_management Format regex # Regex (?<time>\d{4}-\d{1,2}-\d{1,2}T\d{1,2}:\d{1,2}:\d{1,2},\d{1,6})(?<log>.) Regex (?<time>\d{4}-\d{1,2}-\d{1,2}T\d{1,2}:\d{1,2}:\d{1,2},\d{1,6}) (?<level>[^\s].) [(?<thread>.)] [(?<user_id>.)] (?<class_name>[^\s]+) - (?<message>[^\s].*) Time_Key time Time_Format %Y-%m-%dT%H:%M:%S,%L Time_Keep Off
[PARSER] Name redacted_analyzer Format regex Regex (?<time>\d{4}-\d{1,2}-\d{1,2}T\d{1,2}:\d{1,2}:\d{1,2}.\d{1,15}+\d{1,2}:\d{1,2})\s+(?<level>[^\s]) (?<message>[^\s].) Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep Off
[PARSER]
Name redacted_management_nginx
Format regex
Regex (?<time>\d{4}-\d{1,2}-\d{1,2}T\d{1,2}:\d{1,2}:\d{1,2},\d{1,6}) (?<level>[^\s].) [(?<thread>.)] [(?<user_id>.)] (?<class_name>[^\s]+) - (?\d{1,4}) (?<method>\S+) (?<path>\S+) (?<requested_protocol>\S+) (?<size>[^ ]) (?<referer>\S+) (?<agent>[^"]) (?<http_scheme>[^"])
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S,%L
Time_Keep Off
[MULTILINE_PARSER] name redacted_management_multi_line type regex flush_timeout 10000 # parser docker # key_content log
# Regex rules for multiline parsing
# ---------------------------------
# configuration hints:
# - first state always has the name: start_state
# - every field in the rule must be inside double quotes
# rules | state name | regex pattern | next state name
# --------|----------------|-----------------------------------------------------------------------
rule "start_state" "/(\d{4}-\d{1,2}-\d{1,2}T\d{1,2}:\d{1,2}:\d{1,2},\d{1,6})(.*)/" "cont"
rule "cont" "/^\D+/" "cont"
[MULTILINE_PARSER] name redacted_analyzer_multi_line type regex flush_timeout 10000 # parser docker # key_content log
# Regex rules for multiline parsing
# ---------------------------------
# configuration hints:
# - first state always has the name: start_state
# - every field in the rule must be inside double quotes
# rules | state name | regex pattern | next state name
# --------|----------------|-----------------------------------------------------------------------
rule "start_state" "/(\d{4}-\d{1,2}-\d{1,2}T\d{1,2}:\d{1,2}:\d{1,2}.\d{1,15}\+\d{1,2}:\d{1,2})(.*)/" "cont"
rule "cont" "/^\D+/" "cont"
`
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (9 by maintainers)
As I am currently working on the multiline filter… I will try figure this one out some time in the next 2 weeks… unfortunately this error is coming from some code deep in the multiline core library…