fluent-bit: chunks are getting stuck

Bug Report

Describe the bug When I retry forever in the output plugins and the output destinations are unreliable, then chunks sometimes get stuck.

I see messages like

[engine] failed to flush chunk '1-1612396545.855856569.flb', retry in 1485 seconds: task_id=143, input=forward.0 > output=tcp.0

but sometimes this is the last thing I see of the chunk. If I send the CONT signal to fluentbit I see that fluentbit still has them.

To Reproduce I got currently a Java dummy application (if required, I can try to trim it down show I can share it easily) which starts 25 threads. Once those threads are all started, each thread sends 2570 log messages, no ack, each around 4500 bytes, to fluent-bit (via https://github.com/komamitsu/fluency). Between each message, each thread sleeps 10ms. The application is a webserver which logs on startup a couple of lines. The actual log messages I trigger via curl. This application runs on my laptop.

My fluent-bit config

[SERVICE]
    HTTP_Server On
    HTTP_Listen 0.0.0.0
    HTTP_PORT 2020
    Flush 2
    log_level TRACE
    storage.path /fluent-bit/log
    storage.metrics on

[INPUT]
    Name forward
    Listen 0.0.0.0
    Port 24225
    Buffer_Chunk_Size 64k
    Buffer_Max_Size 2M


[OUTPUT]
    Name   syslog
    Match  *

    host   TARGET_IP
    port   514
    mode   tcp

    syslog_format rfc5424
    syslog_message_key log
    Syslog_maxsize 65534
    syslog_appname_key logdest

    # retry forever
    retry_Limit False
    net.connect_timeout 30
    net.keepalive on
    net.keepalive_idle_timeout 10
    storage.total_limit_size 1G

My dockerfile

FROM fluent/fluent-bit:1.6.10
ADD fluent-bit.conf /fluent-bit/etc/fluent-bit.conf

Started via

docker build --no-cache -f Dockerfile -t ruwen-fluent-bit-bug .
docker run -p 24225:24225 -it ruwen-fluent-bit-bug:latest

on my laptop.

I connect to TARGET_IP via a VPN. TARGET_IP is a t2.micro with rsyslog.

Now the following is happening:

• I connect to the VPN
• rsyslog is already running on TARGET_IP
• I start fluentbit on my laptop
• I start the webserver on my laptop
• I disconnect the vpn
• I trigger the endpoint to send all the log messages
• Once all messages are sent (the last few might still be buffered and send a second later), I reconnect to the VPN
• Now fluentbit is successfully sending some of the messages. The little rsyslog box is sometimes a bit overwhelmed so I need more retries.
• The VPN sometimes dies during this process, so I have to reconnect.
• I am keeping an eye on the logs. Since the logs tell me when a retry is expected, I kinda know when to wait and when not. And I always wait some additional time.

This way I can reproduce the issue reliable. This produces the following log: fluentbit_syslog.log

I tried to reproduce it by running syslog locally and use iptables to sometimes drop packages and sometimes not. Without success. But I managed to reproduce the same problem with the tcp output plugin. I tried that to make sure that the problem is unrelated to the syslog output plugin.

The config is used for that:

[SERVICE]
    HTTP_Server On
    HTTP_Listen 0.0.0.0
    HTTP_PORT 2020
    Flush 2
    log_level TRACE
    storage.path /fluent-bit/log
    storage.metrics on

[INPUT]
    Name forward
    Listen 0.0.0.0
    Port 24225
    Buffer_Chunk_Size 64k
    Buffer_Max_Size 2M


[OUTPUT]
    Name   tcp
    Match  *

    host   TARGET_IP
    port   514
    format   json

    # retry forever
    retry_Limit False
    net.connect_timeout 30
    net.keepalive on
    net.keepalive_idle_timeout 10
    storage.total_limit_size 1G

I followed the same procedures as described. On my aws box I initially used nc, which stopped after each connection. Then I used it with the -k option, but that didn’t help. I ended up installed ncat via

ncat -l -k  514 > /dev/null

The whole restarting nc causes even more havoc. With the syslog test, I normally get 1 chunk stuck. Now I got even more. The resulting logfile. fluentbit_tcp.log

To see which chunks are missing, I wrote a little python script which parses those logs. It might be helpful.

#!/usr/bin/env python3

outstanding_chunks_to_error_msgs = dict()

with open('/tmp/fluentbit.log', 'r') as f:
    for line in f:
        line = line.strip()
        line = line.replace("\x1b[1m[\x1b[0m", "").replace("\x1b[1m]\x1b[0m", "")
        if 'failed to flush chunk' in line:
            chunk = line.split("'")[1]
            errors = outstanding_chunks_to_error_msgs.get(chunk)

            parts = line.split(" ")
            error_msg = f"{parts[0]} {parts[1]}: {parts[12]} {parts[13][:-1]}"
            if errors:
                errors.append(error_msg)
            else:
                outstanding_chunks_to_error_msgs[chunk]=[error_msg]
        elif 'flush chunk' in line and 'succeeded at retry' in line:
            chunk = line.split("'")[1]
            del outstanding_chunks_to_error_msgs[chunk]

for chunk, errors in outstanding_chunks_to_error_msgs.items():
    print(f"{chunk}: {errors}")

print(f"{len(outstanding_chunks_to_error_msgs)} missing")

Expected behavior Chunks being retried until they succeed.

Your Environment

• Version used: Docker fluent/fluent-bit:1.6.10. I also tried Docker fluent/fluent-bit:1.7.0-rc3, without success. Since some log messages are now missing, tracing the issue is harder, see https://github.com/fluent/fluent-bit/issues/3001
• Configuration: see reproduce section
• Environment name and version: local laptop
• Server type and version: 16 GB RAM, 8 Cores (Intel® Core™ i7-8650U CPU @ 1.90GHz), SSD
• Operating System and version:

uname -a
Linux ruwenwork 4.19.0-13-amd64 #1 SMP Debian 4.19.160-2 (2020-11-28) x86_64 GNU/Linux

• Filters and plugins: see reproduce section

Additional context How has this issue affected you? What are you trying to accomplish?

I am currently building a logging pipeline to use fluent-bit. My goal is to be able to ingest 256 MB/minute (that’s peak, I normally don’t need that) and to be able buffer data for a while when upstream is having issues. I am currently not imposing a memory limit, I didn’t get that far in my tests.

Please let me know if you need any further details. I am more than happy to build any other branch and try my luck with it.

Cheers Ruwen