fluent-bit: Duplicate Entries when 32+ inputs are configured

Bug Report

BACKGROUND

We’re running amazon/aws-for-fluent-bit:2.10.0 docker container
Input comes from the “tail” plugin
Output goes into AWS CloudWatch using the “cloudwatch_logs” plugin

PROBLEM Since upgrading to aws-for-fluent-bit:2.10.0 (from 2.1.0), we’re getting duplicate entries in the cloudwatch streams, where some entries get duplicated and end up in a different stream within the same group.

TROUBLESHOOTING I believe the problem lies with the tail INPUT plugin, with the issue is appearing in versions 2.8.0 and above of amazon/aws-for-fluent-bit. (I’ve tested against every version.)

The boundary between 2.7.0 and below (working) and 2.8.0 and above (broken) is described here, mentioning the Fluent Bit component versions.

Testing has shown:

The problem remains when switching from cloudwatch_logs OUTPUT plugin to file OUTPUT plugin (except the entries are duplicated to the same log and not to a different stream)
Using the file output plugin and going back to aws-for-fluent-bit:2.7.0 and below solves the issue
Simplifying our configuration, I can provide a very simple config the reproduces the problem (below)

STEPS TO REPRODUCE

Create docker-compose.yml, parsers.conf and fluent-bit.conf using the content below.
mkdir -p DL-5175/output DL-5175/test
Delete previous state (input logs, output logs, databases) and re-create the container docker stop aws_fluentbit; sudo rm dbs/main*; docker rm aws_fluentbit; sudo find DL-5175/ -type f |sudo xargs -I{} rm {}; docker ps; docker-compose up -d; docker ps; sleep 1
Send a test message to all input logs, with the log name in the message (so we know the source) grep -B3 Parser fluent-bit.conf |awk '/Path/ {print $2}' |sed 's%/logs%/opt/aws-fluent-bit/DL-5175%' |sort |while read log; do echo "${now} - DL-5175: TEST MESSAGE ADDED TO ${log} LOG" >> $log; done
Wait a minute for some output logs to appear sleep 60
Count how many output logs there are (expect 36 for 2.7.0 and below where it works fine) cat DL-5175/output/* |awk -F: '{print $1}' |sort |uniq -c |wc -l
Count how many messages in each output log (expect 1-per-log for 2.7.0 and below where it works fine) cat DL-5175/output/* |awk -F: '{print $1}' |sort |uniq -c
See the contents output log, so to closely examine any duplication cat DL-5175/output/*

docker-compse.yml

services:
  aws_fluentbit:
    image: amazon/aws-for-fluent-bit:2.10.0
    hostname: lenny
    container_name: aws_fluentbit
    restart: always
    environment: []
    volumes:
      # main config
      - /opt/aws-fluent-bit/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
      # parsers
      - /opt/aws-fluent-bit/parsers.conf:/fluent-bit/parsers/parsers.conf
      # dbs folder
      - /opt/aws-fluent-bit/dbs:/dbs
      # folders mounted from profiles
      - /opt/aws-fluent-bit/DL-5175:/logs

parsers.conf

    Name        simple_parser
    Format      regex
    Regex       ^

fluent-bit.conf

    Parsers_File        /fluent-bit/parsers/parsers.conf

[INPUT]
    Name                tail
    Path                /logs/test/file_0001.log
    Parser              simple_parser
    DB                  /dbs/main
    Tag                 tag_0001

[OUTPUT]
    Name                file
    Match               tag_0001
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0002.log
    Parser              simple_parser
    DB                  /dbs/main
    Tag                 tag_0002

[OUTPUT]
    Name                file
    Match               tag_0002
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0003.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0003

[OUTPUT]
    Name                file
    Match               tag_0003
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0004.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0004

[OUTPUT]
    Name                file
    Match               tag_0004
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0005.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0005

[OUTPUT]
    Name                file
    Match               tag_0005
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0006.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0006

[OUTPUT]
    Name                file
    Match               tag_0006
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0007.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0007

[OUTPUT]
    Name                file
    Match               tag_0007
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0008.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0008

[OUTPUT]
    Name                file
    Match               tag_0008
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0009.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0009

[OUTPUT]
    Name                file
    Match               tag_0009
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0010.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0010

[OUTPUT]
    Name                file
    Match               tag_0010
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0011.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0011

[OUTPUT]
    Name                file
    Match               tag_0011
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0012.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0012

[OUTPUT]
    Name                file
    Match               tag_0012
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0013.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0013

[OUTPUT]
    Name                file
    Match               tag_0013
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0014.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0014

[OUTPUT]
    Name                file
    Match               tag_0014
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0015.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0015

[OUTPUT]
    Name                file
    Match               tag_0015
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0016.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0016

[OUTPUT]
    Name                file
    Match               tag_0016
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0017.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0017

[OUTPUT]
    Name                file
    Match               tag_0017
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0018.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0018

[OUTPUT]
    Name                file
    Match               tag_0018
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0019.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0019

[OUTPUT]
    Name                file
    Match               tag_0019
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0020.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0020

[OUTPUT]
    Name                file
    Match               tag_0020
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0021.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0021

[OUTPUT]
    Name                file
    Match               tag_0021
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0022.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0022

[OUTPUT]
    Name                file
    Match               tag_0022
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0023.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0023

[OUTPUT]
    Name                file
    Match               tag_0023
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0024.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0024

[OUTPUT]
    Name                file
    Match               tag_0024
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0025.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0025

[OUTPUT]
    Name                file
    Match               tag_0025
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0026.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0026

[OUTPUT]
    Name                file
    Match               tag_0026
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0027.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0027

[OUTPUT]
    Name                file
    Match               tag_0027
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0028.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0028

[OUTPUT]
    Name                file
    Match               tag_0028
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0029.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0029

[OUTPUT]
    Name                file
    Match               tag_0029
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0030.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0030

[OUTPUT]
    Name                file
    Match               tag_0030
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0031.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0031

[OUTPUT]
    Name                file
    Match               tag_0031
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0032.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0032

[OUTPUT]
    Name                file
    Match               tag_0032
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0033.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0033

[OUTPUT]
    Name                file
    Match               tag_0033
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0034.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0034

[OUTPUT]
    Name                file
    Match               tag_0034
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0035.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0035

[OUTPUT]
    Name                file
    Match               tag_0035
    Path                /logs/output

[INPUT]
    Name                tail
    Path                /logs/test/file_0036.log
    Parser              simple_parser
    DB                  /dbs/main   
    Tag                 tag_0036

[OUTPUT]
    Name                file
    Match               tag_0036
    Path                /logs/output

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 16 (8 by maintainers)

Most upvoted comments

@edsiper Confirmation, that’s working now in UAT!

etm42 on Feb 8, 2021

Following the replication steps and extensively testing (using only file OUTPUT), I have made the following observations:

The number of INPUT streams has a direct relation to the number of duplicate entries. As long as fewer than 32 are used, I was not able to replicate any duplicated entries. When the number of streams is more than 32 (N-32 > 0), then we observe N-32*2 duplicates, namely the first and last N-32 tags. We also see a missing stream with odd numbers above 32. The following were tested:

N=34
tag_0001 - tag_0034 :
Duplicate entries found in:
2 tag_0001
2 tag_0034
Obervation:
first and last log receive duplicates

N=34
Removed tag_0026 and tag_0016
Duplicate entries found in:
2 tag_0001
2 tag_0034
Obervation:
first and last log receive duplicates (again)

N=33
Removed `tag_0002`,`tag_0009`,`tag_0034`
No duplicates observed, BUT:
tag_0033 is missing from output entirely

N=32
Removed `tag_0002`,`tag_0009`,`tag_0034`,`tag_0026`
No duplicates observed AND no missing logs

N=38
Added `tag_0037` and `tag_0038`
Duplicates:
      2 tag_0001
      2 tag_0002
      2 tag_0003
      2 tag_0004
      2 tag_0034
      2 tag_0035
      2 tag_0037
      2 tag_0038

N=37
Added `tag_0037`
Duplicates:
      2 tag_0001
      2 tag_0002
      2 tag_0003
      2 tag_0034
      2 tag_0035
      2 tag_0037
AND tag_0034 is missing.

N=24
Tags tag_0001 through tag_0024
No duplicates and no missing logs

stawny on Jan 27, 2021