fluent-bit: Errors when forwarding to ElasticSearch
Bug Report
Describe the bug Getting loads of warning followed immediately by an error in the logs. example below.
I’m running in a Kubernetes cluster in AWS. The cluster is running on top of CoreOS. ElasticSearch cluster is set up with the elasticsearch-operator helm chart To Reproduce
- Example log message if applicable:
{"log":"[2018/08/31 16:15:33] [ warn] [out_es] Elasticsearch error\n","stream":"stderr","time":"2018-08-31T16:15:33.595296858Z"}
{"log":"{\"took\":2205,\"errors\":true,\"items\":[{\"index\":{\"_index\":\"fbit-2018.08\",\"_type\":\"flb_type\",\"_id\":\"WMvEkGUBflYo7f9bpukw\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":2351097,\"_primary_term\":1,\"status\":201}},{\"index\":{\"_index\":\"fbit-2018.08\",\"_type\":\"flb_type\",\"_id\":\"WcvEkGUBflYo7f9bpukw\",\"status\":429,\"error\":{\"type\":\"es_rejected_execution_exception\",\"reason\":\"rejected execution of org.elasticsearch.transport.TransportService$7@227349ba on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@4d3e4d91[Running, pool size = 4, active threads = 4, queued tasks = 202, completed tasks = 210040]]\"}}},{\"index\":{\"_index\":\"fbit-2018.08\",\"_type\":\"flb_type\",\"_id\":\"WsvEkGUBflYo7f9bpukw\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":2354152,\"_primary_term\":1,\"status\":201}},{\"index\":{\"_index\":\"fbit-2018.08\",\"_type\"\n","stream":"stderr","time":"2018-08-31T16:15:33.595299577Z"}
{"log":"[2018/08/31 16:15:37] [error] [out_es] could not pack/validate JSON response\n","stream":"stderr","time":"2018-08-31T16:15:37.652275127Z"}
{"log":"{\"took\":2331,\"errors\":true,\"items\":[{\"index\":{\"_index\":\"fbit-2018.08\",\"_type\":\"flb_type\",\"_id\":\"f8XEkGUBjFzZ4VPRtU-U\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":2356841,\"_primary_term\":1,\"status\":201}},{\"index\":{\"_index\":\"fbit-2018.08\",\"_type\":\"flb_type\",\"_id\":\"g8XEkGUBjFzZ4VPRtU-U\",\"_version\":1,\"result\":\"created\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":2359901,\"_primary_term\":1,\"status\":201}},{\"index\":{\"_index\":\"fbit-2018.08\",\"_type\":\"flb_type\",\"_id\":\"hcXEkGUBjFzZ4VPRtU-U\",\"status\":429,\"error\":{\"type\":\"es_rejected_execution_exception\",\"reason\":\"rejected execution of org.elasticsearch.transport.TransportService$7@68e6d98f on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@4d3e4d91[Running, pool size = 4, active threads = 4, queued tasks = 201, completed tasks = 210410]]\"}}},{\"index\":{\"_index\":\"fbi\n","stream":"stderr","time":"2018-08-31T16:15:37.652302381Z"}
- Steps to reproduce the problem: just deploying the fluentbit helm chart is enough to trigger it.
Expected behavior I’m not expecting to see those errors at all.
Your Environment
- Version used: 0.14.1 docker image.
- Configuration:
[SERVICE]
Flush 1
Daemon Off
Log_Level info
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
[INPUT]
Name tail
Path /var/log/containers/*.log
Parser docker
Tag kube.*
Refresh_Interval 5
Mem_Buf_Limit 5MB
Skip_Long_Lines On
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Merge_JSON_Log On
K8S-Logging.Parser On
K8S-Logging.Exclude On
[OUTPUT]
Name es
Match *
Host elasticsearch-logs-cluster
Port 9200
Logstash_Format On
Retry_Limit False
Type flb_type
Logstash_Prefix fbit
Logstash_DateFormat %Y.%m
tls on
tls.verify off
tls.debug 1
- Environment name and version (e.g. Kubernetes? What version?): Kubernetes 1.11.2
- Server type and version: AWS EC2 m5.xLarge
- Operating System and version: CoreOS
- Filters and plugins:
- Kubernetes
Additional context
Logs are clearly making it into elasticsearch, but there is far too much error logging from fluentbit that I fear I’ll miss something important.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 4
- Comments: 17 (2 by maintainers)
Commits related to this issue
- docker: update for 1.9 release (#731) Signed-off-by: Patrick Stephens <pat@calyptia.com> — committed to rawahars/fluent-bit by patrick-stephens 2 years ago
@edsiper I’ve got the same problem, also related to
es_rejected_execution_exceptionbut:could not pack/validate JSON response. That still seems like a bug to me? It’s at least misleadingSee how the lines are cut off and there are two per
_id. Not sure if this is just a minor issue with the stdout log (this is running on k8s).@jorgebirck setting
Merge_Log Offwill not work if you output logs in JSON and want those logs appear as map fields as part of the log structure. Should this issue be re-opened?@discordianfish Were you able to solve this log chomping off problem? Seems like we are also having the same issue and without the full logs is a nightmare to debug.
Having the same issue. Did anyone find a solution? Here is some snippet from my logs: