fluent-bit: Fluent-bit 2.0.8 throws tons of TLS errors

Bug Report

Describe the bug I’m running a fluent-bit v2.0.8 WIndows 64 bit on a 2012 server that forwards it’s logs to a dockerized 2.0.8 instance on RHEL7. Same problem occured with a 2.0.8 fluent-bit on Windows 11 ARM to fluent-bit in docker container running on Macos. So network related issues are ruled out.

cpu-load on the server side is 100% when the issue occurs

To Reproduce Config of docker container (ingests forwarded logs and pushes to kafka):

[SERVICE]
    log_level    error

[INPUT]
    Name              forward
    Listen            0.0.0.0
    Port              24001
    Buffer_Chunk_Size 1M
    Buffer_Max_Size   6M
    tls on
    tls.verify off
    tls.crt_file /fluent-bit/etc/self_signed.crt
    tls.key_file /fluent-bit/etc/self_signed.key

[OUTPUT]
    Name        kafka
    Match       winlog
    Brokers     kafka-2:9092
    Topics      winlog
    Timestamp_Format iso8601

Config on Windows side (collecting logs):

[INPUT]
    Name         winlog
    Channels     Security,System,Application
    Interval_Sec 1
    Tag          winlog

[FILTER]
    Name record_modifier
    Match       winlog
    Record      log_type winlog
[FILTER]
    Name modify
    Match winlog
    Condition Key_value_equals EventID 5152
    Condition Key_value_equals EventID 5157

[OUTPUT]
    Name          forward
    Match         *
    Host          forwarder
    Port          24001
    tls on
    tls.verify off

Server logs:

Fluent Bit v2.0.8
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/01/14 22:12:33] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=104] Connection reset by peer
[2023/01/14 22:12:33] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:33] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=104] Connection reset by peer
[2023/01/14 22:12:33] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:33] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:33] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:33] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:33] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:33] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:33] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:33] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:33] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:33] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:33] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:33] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:33] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:33] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:33] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:33] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:33] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:33] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:33] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:33] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:33] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:33] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:33] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=104] Connection reset by peer
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib
[2023/01/14 22:12:34] [error] [/src/fluent-bit/src/tls/openssl.c:433 errno=0] Success
[2023/01/14 22:12:34] [error] [tls] syscall error: error:00000005:lib(0):func(0):DH lib

Behaviour on the Windows side:

Steps to reproduce the problem: see above

Expected behavior Logs not being flooded

Screenshots

Your Environment 2.0.8 Win64 and WinARM builds 2.0.8 docker containers on RHEL7 / Macos 13.1

Additional context

About this issue

Original URL
State: closed
Created a year ago
Reactions: 1
Comments: 15 (4 by maintainers)

Most upvoted comments

Hello I was able to reproduce this. We’ll get back to you once we have news.

lecaros on Apr 4, 2023

I’ve been watching this issue closely and tested it out this morning. So far, working great!

adamagnew on Apr 17, 2023

@iplparm @raulgupto @r-lindner or @ict-one-nl Could anyone try this out and let me know if it fixes the issue?

Syn3rman on Apr 17, 2023

Same issue reproduced in the same way impacts 2.0.10. It has heavy impact on CPU. It automatically recovers after around 3 minutes.

iplparm on Apr 12, 2023