fluent-bit: High CPU usage on Raspberry Pi 3

Bug Report

Describe the bug

I’m seeing very high CPU (almost always at 100%) on a Raspberry Pi 3.

To Reproduce

  • Rubular link if applicable:
  • Example log message if applicable:
{"log":"YOUR LOG MESSAGE HERE","stream":"stdout","time":"2018-06-11T14:37:30.681701731Z"}
  • Steps to reproduce the problem:

Expected behavior

I’m expecting CPU usage to be under 35% as it’s needed for other, CPU intensive, important programs.

Screenshots

Screenshot 2019-05-22 15 58 17

Your Environment

  • Version used: Fluent Bit v1.1.1
  • Configuration:
# fluent-bit.conf
# https://docs.fluentbit.io/manual/service
[SERVICE]
    Flush           ${FLB_FLUSH_INT}
    Daemon          Off
    Log_Level       info
    Log_File        ${FLB_LOG_FILE}
    Parsers_File    parsers.conf

    # https://docs.fluentbit.io/manual/configuration/buffering
    storage.path              ${FLB_CONF_DIR}/buffer/
    storage.sync              normal
    storage.checksum          Off
    storage.backlog.mem_limit 5M

    # https://docs.fluentbit.io/manual/configuration/monitoring
    HTTP_Server     On
    HTTP_Listen     ${FLB_HTTP_HOST}
    HTTP_Port       ${FLB_HTTP_PORT}

@INCLUDE inputs.conf
@INCLUDE filters.conf
@INCLUDE outputs.conf

# parsers.conf
# Parse logs from files generated by easyloggingapp
[PARSER]
    Name         el_json
    Format       json
    Time_Key     datetime
    Time_Format  %Y-%m-%dT%H:%M:%S.%L
    Time_Keep    off


# inputs.conf
# Fluent Bit Monitoring
# https://docs.fluentbit.io/manual/configuration/monitoring
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           fluent_bit
    Tag             fluent-bit
    Path            ${LOGS_DIR}/fbit-metrics.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/fbit.db

# Pastri
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           pastri
    Tag             pastri
    Path            ${LOGS_DIR}/pastri.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/pastri.db

# EDFA
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           edfa
    Tag             edfa
    Path            ${LOGS_DIR}/edfa.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/edfa.db

# SFP
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           sfp
    Tag             sfp
    Path            ${LOGS_DIR}/sfp.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/sfp.db

# Power Scan
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           power_scan
    Tag             power-scan
    Path            ${LOGS_DIR}/power-scan.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/power_scan.db

# Encoder (AEAT6600)
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           encoder
    Tag             encoder
    Path            ${LOGS_DIR}/aeat6600.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/encoder.db

# ACM0
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           beacon
    Tag             beacon
    Path            ${LOGS_DIR}/acm0.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/beacon.db

# Heartbeat
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           heartbeat
    Tag             heartbeat
    Path            ${LOGS_DIR}/heartbeat.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/heartbeat.db

# Reboots
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           reboots
    Tag             reboots
    Path            ${LOGS_DIR}/reboots.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/reboots.db

# Repo Status
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           repo_status
    Tag             repo_status
    Path            ${LOGS_DIR}/repo-status.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/repo_status.db

# WireGuard
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           wireguard
    Tag             wireguard
    Path            ${LOGS_DIR}/wireguard-keepalive.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/wireguard.db

# iPerf
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           iperf
    Tag             iperf
    Path            ${LOGS_DIR}/iperf-client.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/iperf.db

# CPU
# https://docs.fluentbit.io/manual/input/cpu
# =============================================================================
[INPUT]
    Name            cpu
    Alias           cpu_usage
    Tag             cpu
    Interval_Sec    ${FLB_PULL_INT}
    Interval_NSec   0

# Disk
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           disk_usage
    Tag             disk
    Path            ${LOGS_DIR}/disk-usage.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/du.db

# Memory
# https://docs.fluentbit.io/manual/input/mem
# =============================================================================
[INPUT]
    Name            mem
    Alias           memory_usage
    Tag             memory
    Interval_Sec    ${FLB_PULL_INT}
    Interval_NSec   0

# Network (WLAN, Ethernet, Loopback, VPN)
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           net
    Tag             net
    Path            ${LOGS_DIR}/network.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/net.db

# Raspi GPU
# https://docs.fluentbit.io/manual/input/tail
# =============================================================================
[INPUT]
    Name            tail
    Alias           gpu_info
    Tag             gpu
    Path            ${LOGS_DIR}/gpu.log
    Parser          el_json
    Path_Key        off
    DB              ${FLB_CONF_DIR}/gpu.db

# TODO: Figure out what other processes we need to keep an eye on
# Process
# NOTE: Group proc messages into 1 tag
# https://docs.fluentbit.io/manual/input/proc
# =============================================================================
[INPUT]
    Name            proc
    Alias           pastri_ps
    Tag             ps
    Proc_Name       pastri
    Interval_Sec    ${FLB_PULL_INT}
    Interval_NSec   0
    Fd              true
    Mem             true

# filters.conf
# Rename mem stats keys
[FILTER]
    Name    modify
    Alias   rename_mem
    Match   memory
    Rename  Mem.total mem_total
    Rename  Mem.used mem_used
    Rename  Mem.free mem_free
    Rename  Swap.total swap_total
    Rename  Swap.used swap_used
    Rename  Swap.free swap_free

# Add {hostname} to every record
# https://docs.fluentbit.io/manual/filter/record_modifier
[FILTER]
    Name    record_modifier
    Alias   add_hostname
    Match   *
    Record  hostname ${HOSTNAME}

# Rename process stats keys
[FILTER]
    Name    modify
    Alias   rename_ps
    Match   ps
    Rename  mem.VmPeak vm_peak
    Rename  mem.VmSize vm_size
    Rename  mem.VmLck vm_lck
    Rename  mem.VmHWM vm_hwm
    Rename  mem.VmRSS vm_rss
    Rename  mem.VmData vm_data
    Rename  mem.VmStk vm_stk
    Rename  mem.VmExe vm_exe
    Rename  mem.VmLib vm_lib
    Rename  mem.VmPTE vm_pte
    Rename  mem.VmSwap vm_swap

# outputs.conf
# Forward logs to Fluentd
# https://docs.fluentbit.io/manual/output/forward#secure_forward_setup
# =============================================================================
[OUTPUT]
    Name            forward
    Alias           forward
    Match           *
    Host            ${FLD_HOST}
    Port            ${FLD_PORT}
    Shared_Key      ${FLD_SHARED_KEY}
    Self_Hostname   ${HOSTNAME}.local
    tls             on
    tls.verify      on
    tls.ca_file     ${FLB_CERTS_DIR}/ca.cert.pem
    tls.crt_file    ${FLB_CERTS_DIR}/fbit.cert.pem
    tls.key_file    ${FLB_CERTS_DIR}/fbit.key.pem
    tls.key_passwd  ${FLB_CERT_PASS}
  • Environment name and version (e.g. Kubernetes? What version?):
  • Server type and version:
  • Operating System and version:
Linux raspi 4.14.44-v7+ #1117 SMP Thu May 31 16:57:56 BST 2018 armv7l GNU/Linux

PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"
NAME="Raspbian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
  • Filters and plugins: See config for all plugins and filters.

Additional context

I’m building FB myself using the following bash script:

#!/bin/bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
BIN_DIR=${DIR}/installer/bin

# Fluent Bit
FLB_VERSION=1.1.1
FLB_DOWNLOAD_DIR=/tmp/fluent-bit
FLB_ARCHIVE=fluent-bit-${FLB_VERSION}.tar.gz
FLB_ROOT_DIR=${FLB_DOWNLOAD_DIR}/fluent-bit-${FLB_VERSION}
FLB_BUILD_DIR=${FLB_ROOT_DIR}/build

# Get Fluent Bit
mkdir -p ${FLB_ROOT_DIR}
wget -P ${FLB_DOWNLOAD_DIR} https://fluentbit.io/releases/1.1/${FLB_ARCHIVE}
tar xzf ${FLB_DOWNLOAD_DIR}/${FLB_ARCHIVE} -C ${FLB_DOWNLOAD_DIR}
rm -rf ${FLB_DOWNLOAD_DIR}/${FLB_ARCHIVE}

# Build Fluent Bit
# https://docs.fluentbit.io/manual/installation/build_install
# https://docs.fluentbit.io/manual/installation/build_static_configuration
pushd .
cd ${FLB_BUILD_DIR}
# Note that the FLB_OUT_TD cannot be turned off as it will cause an error at compile time
cmake -DFLB_EXAMPLES=No \
  -DFLB_JEMALLOC=Yes \
  -DFLB_HTTP_SERVER=Yes \
  -DFLB_LUAJIT=No \
  -DFLB_METRICS=Yes \
  -DFLB_PROXY_GO=No \
  -DFLB_STREAM_PROCESSOR=No \
  -DFLB_FILTER_KUBERNETES=Off \
  -DFLB_FILTER_LUA=Off \
  -DFLB_IN_DUMMY=Off \
  -DFLB_IN_FORWARD=Off \
  -DFLB_IN_HEAD=Off \
  -DFLB_IN_KMSG=Off \
  -DFLB_IN_MQTT=Off \
  -DFLB_IN_RANDOM=Off \
  -DFLB_IN_SERIAL=Off \
  -DFLB_IN_SYSLOG=Off \
  -DFLB_IN_STDIN=Off \
  -DFLB_OUT_AZURE=Off \
  -DFLB_OUT_BIGQUERY=Off \
  -DFLB_OUT_COUNTER=Off \
  -DFLB_OUT_EXIT=Off \
  -DFLB_OUT_FLOWCOUNTER=Off \
  -DFLB_OUT_GELF=Off \
  -DFLB_OUT_HTTP=Off \
  -DFLB_OUT_INFLUXDB=Off \
  -DFLB_OUT_KAFKA=Off \
  -DFLB_OUT_KAFKA_REST=Off \
  -DFLB_OUT_NATS=Off \
  -DFLB_OUT_NULL=Off \
  -DFLB_OUT_PLOT=Off \
  -DFLB_OUT_SPLUNK=Off \
  -DFLB_OUT_STACKDRIVER=Off \
  ../

# NOTE: Using all cores (4) might crash the Raspi
make -j 3 # $(getconf _NPROCESSORS_ONLN)

# Copy binaries
cp -a ./bin/. ${BIN_DIR}/

# Verify binary
${BIN_DIR}/fluent-bit --version

popd

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 15 (7 by maintainers)

Most upvoted comments

FYI: I think the upcoming Keepalive support for network connections will help to reduce CPU usage generated by TLS handshakes: #1704

+2
edsiper on Nov 2, 2019
Read more comments on GitHub
← fluent-bit: Grep filter doesn't work for systemd logs
fluent-bit: Hot reload finished but had "requested parser xxx not found" and other failures →