aws-for-fluent-bit: [all versions] exec input SIGSEGV/crash due unitialized memory [fix in 2.31.12]

Describe the question/issue

Hi folks,

We’ve been basing our EKS logging infrastructure on AWS-for-fluent-bit. However, recently, we’ve been noticing some Fluent Bit pods crash with a SIGSEGV on startup and go into a CrashLoopBackoff loop on deployment. Redoing the deployment leads to the same problem on the very same physical hosts while other pods elsewhere on other hosts in the same cluster run fine. The EKS workers are configured identically, so it’s a bit of a head scratcher as to why this is happening on a handful of random nodes in a cluster and that too persistently while the majority of the pods are running fine.

If we let the pod retries run long enough on a host, eventually, it will succeed but that can take anywhere from an hour to a day, unacceptable for a production environment.

Configuration

Please find the config map for aws-for-fluent-bit attached. This contains the fluent bit config file as well. Note that the daemonset is running in its own namespace (“logging”). The namespace is attached to a dedicated service account, “fb-service-account”

aws-for-fluent-bit-conf.txt

Fluent Bit Log Output

DebugLog.txt

Fluent Bit Version Info

AWS For Fluent Bit Image: 2.31.11 though we’ve also seen the same behavior for 2.31.10 and 2.31.6 For debugging , we used debug-2.31.11.

Cluster Details

EKS cluster: K8s version 1.25 (v1.25.9-eks-0a21954) though we’ve also noticed this for earlier versions Instance types: mostly r6gd.16xl and the occasional r5d.16xlarge We base our internal AMIs on the following images:

• arm64 - ami-0aa7aa4c87fe47ff6
• amd64 - ami-071432800334eb200

Application Details

This occurred on an essentially idle cluster with no applications running. Fluent Bit crashes immediately on startup, so load wasn’t a factor.

Steps to reproduce issue

We reviewed the suggestions provided in:

• https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#segfaults-and-crashes-sigsegv
• https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/tutorials/remote-core-dump/README.md#step-3-using-gdb-with-a-live-fluent-bit

We generated the debug log as follows:

• on a machine where FB was repeatedly crashing , we copied over the fluent-bit config files to /home/myname/etc
• docker run -it --entrypoint=/bin/bash --ulimit core=-1 -v /var/log:/var/log -v /home/sacharya:/cores -v $(pwd):/fluent-bit/etc public.ecr.aws/aws-observability/aws-for-fluent-bit:debug-2.31.11
• Note that while the image name is the same,
• once in pod shell, we had to install some libraries to get rid of gdb warnings:

yum -y install yum-utils
debuginfo-install bzip2-libs-1.0.6-13.amzn2.0.3.aarch64 cyrus-sasl-lib-2.1.26-24.amzn2.aarch64 elfutils-libelf-0.176-2.amzn2.aarch64 elfutils-libs-0.176-2.amzn2.aarch64 keyutils-libs-1.5.8-3.amzn2.0.2.aarch64 krb5-libs-1.15.1-55.amzn2.2.5.aarch64 libcap-2.54-1.amzn2.0.1.aarch64 libcom_err-1.42.9-19.amzn2.0.1.aarch64 libcrypt-2.26-63.amzn2.aarch64 libgcc-7.3.1-15.amzn2.aarch64 libgcrypt-1.5.3-14.amzn2.0.3.aarch64 libgpg-error-1.12-3.amzn2.0.3.aarch64 libselinux-2.5-12.amzn2.0.2.aarch64 libyaml-0.1.4-11.amzn2.0.2.aarch64 lz4-1.7.5-2.amzn2.0.1.aarch64 pcre-8.32-17.amzn2.0.2.aarch64 systemd-libs-219-78.amzn2.0.22.aarch64 xz-libs-5.2.2-1.amzn2.0.3.aarch64 zlib-1.2.7-19.amzn2.0.2.aarch64

yum remove openssl-debuginfo-1:1.0.2k-24.amzn2.0.6.aarch64
debuginfo-install openssl11-libs-1.1.1g-12.amzn2.0.13.aarch64

• and then, finally:

export FLB_LOG_LEVEL=debug
gdb /fluent-bit/bin/fluent-bit
r -e /fluent-bit/firehose.so -e /fluent-bit/cloudwatch.so -e /fluent-bit/kinesis.so -c /fluent-bit/etc/fluent-bit.conf

Note that we were able to manually run FB successfully via docker on the other hosts in the cluster where FB pods had been successfully launched by our usual (terraform/helm based) deployment process.

Related Issues