fluent-bit: fluent-bit crashes when using azure blob output
Bug Report
Describe the bug When using azure_blob output, fluent-bit crashes repeatedly on multiple nodes and does not recover. Connections with azure storage is successful because there are logs in the storage.
To Reproduce Install fluent-bit using helm chart 0.7.12
Config from values.yaml
image:
repository: fluent/fluent-bit
tag: 1.6.8
pullPolicy: IfNotPresent
env: ...
resources:
requests:
cpu: 30m
memory: 20Mi
limits:
cpu: 100m
memory: 100Mi
config:
service: |
[SERVICE]
Flush 5
Log_Level info
Daemon off
Parsers_File parsers.conf
Parsers_File custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_PORT 2020
inputs: |
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
Exclude_Path *kube-*,*azure-*,*blackbox-exporter-*,*blobfuse-flexvol-*,*coredns-*,*istio-*,*node-exporter-*,*keyvault-flexvolume-*,*kiali-*,*fluent-bit*
Parser docker2
DB /var/log/flb_kube.db
Mem_Buf_Limit 10MB
Skip_Long_Lines On
Refresh_Interval 10
[INPUT]
Name tail
Tag kube-ingress-controller
Path /var/log/containers/*ingress-nginx-controller*.log
Parser docker2
DB /var/log/flb_kube_ingress_controller.db
Mem_Buf_Limit 10MB
Skip_Long_Lines On
Refresh_Interval 10
filters: |
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc.cluster.local:443
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
[FILTER]
Name record_modifier
Match *
Record resource_group ${RESOURCE_GROUP}
outputs: |
[OUTPUT]
Name es
Match kube.*
Host elasticsearch-master.logging.svc.cluster.local
Port 9200
HTTP_User ${ELASTIC_USERNAME}
HTTP_Passwd ${ELASTIC_PASSWORD}
Buffer_Size 32KB
Logstash_Format On
Logstash_Prefix fluent-bit
Type _doc
Replace_Dots On
Time_Key @ts
Time_Key_Format %Y-%m-%dT%H:%M:%S
tls On
tls.verify Off
[OUTPUT]
name azure_blob
match kube-ingress-controller
account_name ${LOG_STORAGE_ACCOUNT}
shared_key ${LOG_STORAGE_ACCOUNT_KEY}
container_name access-logs
blob_type blockblob
auto_create_container on
tls on
customParsers: |
[PARSER]
Name docker2
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
# Command | Decoder | Field | Optional Action
# =============|==================|=================
Decode_Field_As escaped log
Logs from pods that crash:
[2020/12/08 13:33:21] [ warn] [engine] failed to flush chunk '1-1607434393.967383841.flb', retry in 11 seconds: task_id=9, input=tail.0 > output=es.0
[2020/12/08 13:33:21] [ warn] [engine] failed to flush chunk '1-1607434394.70654065.flb', retry in 7 seconds: task_id=10, input=tail.0 > output=es.0
[2020/12/08 13:33:21] [ info] [output:azure_blob:azure_blob.1] content appended to blob successfully
[2020/12/08 13:33:21] [ info] [output:azure_blob:azure_blob.1] blob id ZmxiLTE2MDc0MzQ0MDEuMTcxMy5pZA== committed successfully
[2020/12/08 13:33:22] [ warn] [engine] failed to flush chunk '1-1607434400.967468964.flb', retry in 6 seconds: task_id=1, input=tail.0 > output=es.0
[2020/12/08 13:33:22] [ warn] [engine] failed to flush chunk '1-1607434401.169775787.flb', retry in 11 seconds: task_id=8, input=tail.0 > output=es.0
[2020/12/08 13:33:22] [ info] [output:azure_blob:azure_blob.1] content appended to blob successfully
[2020/12/08 13:33:22] [ info] [output:azure_blob:azure_blob.1] blob id ZmxiLTE2MDc0MzQ0MDIuMTcwMy5pZA== committed successfully
[2020/12/08 13:33:27] [ warn] [engine] failed to flush chunk '1-1607434402.741153588.flb', retry in 8 seconds: task_id=4, input=tail.0 > output=es.0
[2020/12/08 13:33:27] [ warn] [engine] failed to flush chunk '1-1607434407.15776933.flb', retry in 8 seconds: task_id=17, input=tail.0 > output=es.0
[2020/12/08 13:33:27] [ info] [output:azure_blob:azure_blob.1] content appended to blob successfully
[2020/12/08 13:33:27] [ info] [output:azure_blob:azure_blob.1] blob id ZmxiLTE2MDc0MzQ0MDcuMTc4MC5pZA== committed successfully
[2020/12/08 13:33:28] [ warn] [engine] chunk '1-1607434400.967468964.flb' cannot be retried: task_id=1, input=tail.0 > output=es.0
[2020/12/08 13:33:28] [ warn] [engine] chunk '1-1607434394.70654065.flb' cannot be retried: task_id=10, input=tail.0 > output=es.0
[2020/12/08 13:33:30] [engine] caught signal (SIGSEGV)
#0 0x55f72d985c03 in ???() at lib/monkey/deps/flb_libco/libco.c:0
Expected behavior Fluent-bit not to crash.
Screenshots
Your Environment
-
Version used: 1.6.8
-
Configuration:
-
Environment name and version: Kubernetes 1.8.8
-
Filters and plugins: Azure blob output plugin
Additional context I am trying to send ingress controller logs from k8s to azure storage blob.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 3
- Comments: 26 (7 by maintainers)
hi, any updates on this? it is still a relevant issue from my point of view
@edsiper would you like to connect / screen share to walk through the repro steps?