fluent-bit: fluent-bit crashes when using azure blob output

Bug Report

Describe the bug When using azure_blob output, fluent-bit crashes repeatedly on multiple nodes and does not recover. Connections with azure storage is successful because there are logs in the storage.

To Reproduce Install fluent-bit using helm chart 0.7.12

Config from values.yaml

image:
  repository: fluent/fluent-bit
  tag: 1.6.8
  pullPolicy: IfNotPresent
env: ...
resources:
  requests:
    cpu: 30m
    memory: 20Mi
  limits:
    cpu: 100m
    memory: 100Mi
config:
  service: |
    [SERVICE]
      Flush         5
      Log_Level     info
      Daemon        off
      Parsers_File  parsers.conf
      Parsers_File  custom_parsers.conf
      HTTP_Server   On
      HTTP_Listen   0.0.0.0
      HTTP_PORT     2020
  inputs: | 
    [INPUT]
      Name              tail
      Tag               kube.*
      Path              /var/log/containers/*.log
      Exclude_Path      *kube-*,*azure-*,*blackbox-exporter-*,*blobfuse-flexvol-*,*coredns-*,*istio-*,*node-exporter-*,*keyvault-flexvolume-*,*kiali-*,*fluent-bit*
      Parser            docker2
      DB                /var/log/flb_kube.db
      Mem_Buf_Limit     10MB
      Skip_Long_Lines   On
      Refresh_Interval  10
    [INPUT]
      Name              tail
      Tag               kube-ingress-controller
      Path              /var/log/containers/*ingress-nginx-controller*.log
      Parser            docker2
      DB                /var/log/flb_kube_ingress_controller.db
      Mem_Buf_Limit     10MB
      Skip_Long_Lines   On
      Refresh_Interval  10
  filters: |
    [FILTER]
      Name                kubernetes
      Match               kube.*
      Kube_URL            https://kubernetes.default.svc.cluster.local:443
      Merge_Log           On
      Keep_Log            Off
      K8S-Logging.Parser  On
      K8S-Logging.Exclude On

    [FILTER]
      Name record_modifier
      Match *
      Record resource_group ${RESOURCE_GROUP}
  outputs: | 
    [OUTPUT]
      Name             es
      Match            kube.*
      Host             elasticsearch-master.logging.svc.cluster.local
      Port             9200
      HTTP_User        ${ELASTIC_USERNAME}
      HTTP_Passwd      ${ELASTIC_PASSWORD}
      Buffer_Size      32KB
      Logstash_Format  On
      Logstash_Prefix  fluent-bit
      Type             _doc
      Replace_Dots     On
      Time_Key         @ts
      Time_Key_Format  %Y-%m-%dT%H:%M:%S
      tls              On
      tls.verify       Off
    [OUTPUT]
      name                  azure_blob
      match                 kube-ingress-controller
      account_name          ${LOG_STORAGE_ACCOUNT}
      shared_key            ${LOG_STORAGE_ACCOUNT_KEY}
      container_name        access-logs
      blob_type             blockblob
      auto_create_container on
      tls                   on
  customParsers: |
    [PARSER]
      Name        docker2
      Format      json
      Time_Key    time
      Time_Format %Y-%m-%dT%H:%M:%S.%L
      Time_Keep   On
      # Command      |  Decoder | Field | Optional Action
      # =============|==================|=================
      Decode_Field_As   escaped    log   

Logs from pods that crash:

[2020/12/08 13:33:21] [ warn] [engine] failed to flush chunk '1-1607434393.967383841.flb', retry in 11 seconds: task_id=9, input=tail.0 > output=es.0
[2020/12/08 13:33:21] [ warn] [engine] failed to flush chunk '1-1607434394.70654065.flb', retry in 7 seconds: task_id=10, input=tail.0 > output=es.0
[2020/12/08 13:33:21] [ info] [output:azure_blob:azure_blob.1] content appended to blob successfully
[2020/12/08 13:33:21] [ info] [output:azure_blob:azure_blob.1] blob id ZmxiLTE2MDc0MzQ0MDEuMTcxMy5pZA== committed successfully
[2020/12/08 13:33:22] [ warn] [engine] failed to flush chunk '1-1607434400.967468964.flb', retry in 6 seconds: task_id=1, input=tail.0 > output=es.0
[2020/12/08 13:33:22] [ warn] [engine] failed to flush chunk '1-1607434401.169775787.flb', retry in 11 seconds: task_id=8, input=tail.0 > output=es.0
[2020/12/08 13:33:22] [ info] [output:azure_blob:azure_blob.1] content appended to blob successfully
[2020/12/08 13:33:22] [ info] [output:azure_blob:azure_blob.1] blob id ZmxiLTE2MDc0MzQ0MDIuMTcwMy5pZA== committed successfully
[2020/12/08 13:33:27] [ warn] [engine] failed to flush chunk '1-1607434402.741153588.flb', retry in 8 seconds: task_id=4, input=tail.0 > output=es.0
[2020/12/08 13:33:27] [ warn] [engine] failed to flush chunk '1-1607434407.15776933.flb', retry in 8 seconds: task_id=17, input=tail.0 > output=es.0
[2020/12/08 13:33:27] [ info] [output:azure_blob:azure_blob.1] content appended to blob successfully
[2020/12/08 13:33:27] [ info] [output:azure_blob:azure_blob.1] blob id ZmxiLTE2MDc0MzQ0MDcuMTc4MC5pZA== committed successfully
[2020/12/08 13:33:28] [ warn] [engine] chunk '1-1607434400.967468964.flb' cannot be retried: task_id=1, input=tail.0 > output=es.0
[2020/12/08 13:33:28] [ warn] [engine] chunk '1-1607434394.70654065.flb' cannot be retried: task_id=10, input=tail.0 > output=es.0
[2020/12/08 13:33:30] [engine] caught signal (SIGSEGV)
#0  0x55f72d985c03      in  ???() at lib/monkey/deps/flb_libco/libco.c:0

Expected behavior Fluent-bit not to crash.

Screenshots

Your Environment

  • Version used: 1.6.8

  • Configuration:

  • Environment name and version: Kubernetes 1.8.8

  • Filters and plugins: Azure blob output plugin

Additional context I am trying to send ingress controller logs from k8s to azure storage blob.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 3
  • Comments: 26 (7 by maintainers)

Most upvoted comments

hi, any updates on this? it is still a relevant issue from my point of view

@edsiper would you like to connect / screen share to walk through the repro steps?