fluent-bit: could not enqueue records into the ring buffer

Bug Report

Describe the bug Error Log:

[2023/03/26 07:32:13] [error] [input:tail:tail.0] could not enqueue records into the ring buffer
[tail.0] failed buffer write, retries=0
[tail.0] failed buffer write, retries=1
[tail.0] failed buffer write, retries=2
[tail.0] failed buffer write, retries=3
[tail.0] failed buffer write, retries=4
[tail.0] failed buffer write, retries=5
[tail.0] failed buffer write, retries=0
[tail.0] failed buffer write, retries=1
[tail.0] failed buffer write, retries=2
[tail.0] failed buffer write, retries=3
[tail.0] failed buffer write, retries=4
[tail.0] failed buffer write, retries=5
[tail.0] failed buffer write, retries=6
[tail.0] failed buffer write, retries=7
[tail.0] failed buffer write, retries=8
[tail.0] failed buffer write, retries=0
[tail.0] failed buffer write, retries=0
[tail.0] failed buffer write, retries=0
[tail.0] failed buffer write, retries=0
[tail.0] failed buffer write, retries=1
[tail.0] failed buffer write, retries=2
[tail.0] failed buffer write, retries=0

fluent-bit config

apiVersion: v1
data:
  custom_parsers.conf: |
    [PARSER]
        Name docker_no_time
        Format json
        Time_Keep Off
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
    [PARSER]
        Name custom-nginx
        Format regex
        Regex ^(?<domain>[^ ]*) (?<remote_ip>[^\, ]*)(?:,? ?[^\[]*) \[(?<time>[^\]]*)\]\[(?<msec>[^ ]*)\] "(?<method>[^ ]*) (?<url>[^\?]*)(?:\??)(?<url_params>.*) (HTTP\/)?(?<http_version>[^ ]*)" (?<status>[^ ]*) (?<body_bytes>[^ ]*) "(?<referer>[^\"]*)" "(?<http_user_agent>[^\"]*)" "(?<body>[^ ]*)" (?<request_time>[^ ]*) (?<response_time>[^ ]*) (?<app_id>[^ ]*) (?<digest>[^ ]*) "(?<firmware_version>[^ ]*)" "(?<device_model>[^ ]*)" "(?<app_version>[^ ]*)" "(?<token>[^\?]*)" "(?<resp_body>[^"]*)" "(?<request_id>[^ "]*)"
        Time_Format %d/%b/%Y:%H:%M:%S %z
        Time_Key time
    [PARSER]
        Name   logfmt
        Format logfmt
  fluent-bit.conf: |
    [SERVICE]
        Daemon Off

        Flush 5
        Log_Level warn
        Parsers_File parsers.conf
        Parsers_File custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        storage.path              /var/log/flb-storage/
        storage.sync              normal
        storage.checksum          off
        storage.backlog.mem_limit 300M
        storage.max_chunks_up     256
        storage.metrics           on

    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
        DB /var/log/flb_kube.db
        Mem_Buf_Limit  256MB
        Rotate_Wait 30
        Buffer_Chunk_Size 5MB
        Buffer_Max_Size 512MB
        threaded on
    [INPUT]
        Name              tail
        Tag               kube_audit.*
        Path              /var/log/kube-apiserver-audit.log
        Parser            docker_no_time
        DB                /var/log/flb_kube_audit.db
        Mem_Buf_Limit  32MB
        Rotate_Wait    30
        Buffer_Chunk_Size 5MB
        Buffer_Max_Size 32MB
        threaded on

    [FILTER]
        Name kubernetes
        Match kube.*
        Kube_URL       https://kubernetes.default.svc:443
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude Off
    [FILTER]
        Name           grep
        Match          *
        Exclude        http_user_agent ELB-HealthChecker
    [FILTER]
        Name             geoip2
        Match            *
        Database         /fluent-bit/geo/GeoLite2-City.mmdb
        Lookup_key       remote_ip
        Record  country_name  remote_ip %{country.names.en}
        Record  country_code2 remote_ip %{country.iso_code}
    [FILTER]
        Name           nest
        Match          *
        Operation      nest
        Wildcard       country*
        Nest_under     geoip
    [FILTER]
        Name           lua
        Match          *
        script         /fluent-bit/scripts/qnap_filter.lua
        call           ip_filter

    [OUTPUT]
        Name loki
        Match *
        Host loki-loki-distributed-distributor.monitoring.svc
        Port 3100
        labels job=fluent-bit
        label_map_path  /fluent-bit/etc/labelmap.json
        remove_keys kubernetes.container_hash, kubernetes.docker_id, kubernetes.annotations, stream, _p
        auto_kubernetes_labels off
        line_format json
        Retry_Limit False
  labelmap.json: |
    {
      "kubernetes": {
        "container_name": "container",
        "host": "node",
        "labels": {
          "app": "app",
          "k8s_app": "app",
          "release": "release"
        },
        "namespace_name": "namespace",
        "pod_name": "instance"
      },
      "stream": "stream"
    }

edit deployment:

**To Reproduce**
When I add "fluent-bit/parser" in deployment

template: metadata: annotations: fluentbit.io/parser: custom-nginx


Fluentibit can't send log to loki. Log miss. 

**Your Environment**
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used: kubernetes 1.25.6. fluent-bit 2.0.8

Could everyone have any suggestion?

About this issue

Original URL
State: closed
Created a year ago
Comments: 17 (1 by maintainers)

Most upvoted comments

Yes, that’s an issue we’re actively working on and expect to fix within the week.

leonardo-albertovich on May 10, 2023