fluentd: SSL_accept SYSCALL returned=5 errno=0

Describe the bug

I am seeing this warning continuously and causing logs lost… [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"

To Reproduce Run td-agent with the following config Expected behavior The warning should not occur

Your Environment

Fluentd or td-agent version: fluentd --version or td-agent --version td-agent 1.10.0
Operating system: cat /etc/os-release RED HAT 7.7
Kernel version: uname -r 3.10.0-1062.12.1.el7.x86_64
Ruby 2.0.0p648 (2015-12-16) [x86_64-linux]
Openssl 1.0.2k-fips 26 Jan 2017

Your Configuration

<source>
  @type forward
  port 52466
  <security>
      self_hostname XXX
      shared_key yyyyyyyy
  </security>
  <transport tls>
      cert_path /etc/td-agent/certs/test.crt
      private_key_path /etc/td-agent/certs/test.key
  </transport>
</source>

Your Error Log

2020-04-14 17:21:44 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:44 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:45 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:45 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:45 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:45 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:45 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:45 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:45 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:46 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:46 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:46 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"
2020-04-14 17:21:46 +0000 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=unknown state"

Additional context

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 9
Comments: 28 (2 by maintainers)

Most upvoted comments

@venky999 , do you use fluentd behind load balancer? we’re experiencing same errors on td-agents behind AWS ELB.

konstantin-kornienko on Aug 26, 2020

I get warning from <source> section, even if there is no <match **> section,

<source>
    @type forward
    <transport tls>
        cert_path /etc/ssl/xx/xx.crt
        private_key_path /etc/ssl/xx/xx.key
        private_key_passphrase xx
    </transport>
</source>

I downgraded to td-agent 1.9.2 version and the ssl warning isn’t visible anymore, probably because it is being logged as a trace in this version.

kritisingh on Apr 27, 2020

I found the same problem using an AWS network load balancer (NLB) with an EKS cluster, but NLB has no health check options and solution proposed by @bloodguard can’t be used. Has somebody else faced the same issue?

farnsworth on Oct 26, 2021

Im writting this here to say i had the same issue, but i was able to resolve it.

A bit of context. The FluentD is on an EKS cluster. That EKS has a load balancer spawned with AWS Load Balancer Controller. I had the same error: "unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error=“SSL_accept SYSCALL returned=5 errno=0 state=before SSL initialization”. I recently updated to the latest version of fluentdD (v1.12.3) and was able to see the IP of the host of the connection. The IP was from the Load Balancer. I changed the health check port/path of the Load Balancer to the path of the monitor_agent of the fluentd, and the error just stopped appearing.

bloodguard on Jun 1, 2021

We have the same issue here:

fluentd 1.11.2
ruby 2.7.0

<source>
  @type forward
  @id source_forward
  @label processing
  <transport tls>
    cert_path *****
    private_key_path ******
    private_key_passphrase ''
  </transport>
  <security>
    shared_key *****
    self_hostname *****
  </security>
  bind 0.0.0.0
  port 12225
</source>

2021-03-24 18:35:10 +0000 [warn]: #0 [source_forward] unexpected error before accepting TLS connection by OpenSSL error_class=OpenSSL::SSL::SSLError error="SSL_accept SYSCALL returned=5 errno=0 state=before SSL initialization"

Not sure if we lose logs because of that, we don’t see any issue on nodes that send logs to it.

Even weirder, we don’t have the issue on other fluentd relays with exactly the same config (same distro, same config file, same versions, etc)

We don’t use any LB between forwards output and input, all connections are established directly on the same private network.

cedriclaudrel on Mar 24, 2021

TL;DR - Ensure that you can configure your load balancer (if you are using one) to route health checks/traffic over the SSL protocol.

I had this issue also.

My setup was:

GCP VM sending SSL encrypted traffic with a self-signed certificate -> GCP Internal Load balancer -> GCP VMs running Fluentd hosting a self-signed cert/key.

@bloodguard proposed solution helped, upgrading Fluentd (we used 1.13.1) allowed us to see the IP address of the GCP Internal LB.

The original health check configured showed the OpenSSL error in Fluent logs:

Protocol: TCP
Port: 9000

Once we changed the protocol to SSL, but kept the port to our designated Fluentd port, the OpenSSL errors went away:

Protocol: SSL
Port: 9000

Hope this helps anyone with this issue.

sky-amoncadot on Jul 15, 2021

Yes. Above change shows this warning logs. But we can’t judge this issue is fluentd bug or certificate/openssl setting mismatch. We need more information and reproducible step, e.g. how to create certificates, old fluentd works or not and more.

repeatedly on Apr 24, 2020

@kritisingh no…seeing same issue

venky999 on Apr 21, 2020