fluent-bit: [v1.6] storage.total_limit_size is throwing error
Bug Report
Describe the bug
no enough space in filesystem to buffer chunk 1-1602687091.955423455.flb in plugin es.0 coming when I enable storage.total_limit_size 5M in es output
To Reproduce
- Example log message if applicable:
│ [2020/10/14 15:18:04] [debug] [input:tail:tail.0] inode=1291902 events: IN_MODIFY │
│ [2020/10/14 15:18:04] [debug] [storage] tail.0:1-1602688684.470023866.flb mapped OK │
│ [2020/10/14 15:18:04] [debug] [input chunk] chunk 1-1602688684.470023866.flb required 840 bytes and 8206935 bytes left in plugin es.0 │
│ [2020/10/14 15:18:04] [error] [input chunk] no enough space in filesystem to buffer chunk 1-1602688684.470023866.flb in plugin es.0 │
│ [2020/10/14 15:18:04] [debug] [storage] [cio file] synced at: tail.0/1-1602688684.470023866.flb │
│ [2020/10/14 15:18:04] [error] [input chunk] no available chunk │
│ [2020/10/14 15:18:04] [debug] [storage] tail.0:1-1602688684.470514408.flb mapped OK │
│ [2020/10/14 15:18:04] [debug] [input chunk] chunk 1-1602688684.470514408.flb required 963 bytes and 8206935 bytes left in plugin es.0 │
│ [2020/10/14 15:18:04] [error] [input chunk] no enough space in filesystem to buffer chunk 1-1602688684.470514408.flb in plugin es.0 │
│ [2020/10/14 15:18:04] [debug] [storage] [cio file] synced at: tail.0/1-1602688684.470514408.flb │
│ [2020/10/14 15:18:04] [error] [input chunk] no available chunk │
│ [2020/10/14 15:18:04] [debug] [input:tail:tail.0] inode=1291902 events: IN_MODIFY │
│ [2020/10/14 15:18:04] [debug] [storage] tail.0:1-1602688684.471037398.flb mapped OK │
│ [2020/10/14 15:18:04] [debug] [input chunk] chunk 1-1602688684.471037398.flb required 943 bytes and 8206935 bytes left in plugin es.0 │
│ [2020/10/14 15:18:04] [error] [input chunk] no enough space in filesystem to buffer chunk 1-1602688684.471037398.flb in plugin es.0 │
│ [2020/10/14 15:18:04] [debug] [storage] [cio file] synced at: tail.0/1-1602688684.471037398.flb
- Steps to reproduce the problem:
Expected behavior
Storage LRU should kick in once the limit exeeds
Your Environment
-
Version used: 1.6
-
Configuration: kubernetes manifest fluentbit-complete.yaml
-
Environment name and version (e.g. Kubernetes? What version?):
Client Version: v1.19.2
Server Version: v1.18.6+k3s1
Additional context My disk got completely utilized by fluentbit when my es was down. So figured out I’ve to upgrade to 1.6 to have LRU like fluentd.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 1
- Comments: 90 (48 by maintainers)
Commits related to this issue
- input: chunk: fix output instance 'new bytes' calculation (#2688) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> — committed to fluent/fluent-bit by yqgdb000 4 years ago
- input: chunk: fix output instance 'new bytes' calculation (#2688) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> — committed to fluent/fluent-bit by yqgdb000 4 years ago
@ujjwalgarg1995 It is fixed but it still doesn’t work well with the
rewrite_tagplugin. Please see https://github.com/fluent/fluent-bit/issues/3468. If you pipeline doesn’t use therewrite_tagplugin then it is fine.it seem like regression, i noticed in 1.8.9 too … see my slack post: https://fluent-all.slack.com/archives/C0CTQGHKJ/p1635964738252500 … most of the responses are in https://fluent-all.slack.com/archives/C0CTQGHKJ/p1635968668257800
thanks everyone. The fix c27896fd is already in place. It will be part of v1.6.2 release today.
@rjshrjndrn @Cyanopus Confirmed that I was able to reproduce the issue in a cluster environment. Debugging now.
Thanks!
Thank you, I will look into it.