strimzi-kafka-operator: Kafka Exporter has **CrashLoopBackOff** and can't recover.
Please use this to only for bug reports. For questions or when you need help, you can use the GitHub Discussions, our #strimzi Slack channel or out user mailing list.
Describe the bug Kafka Exporter has CrashLoopBackOff and can’t recover.
To Reproduce Sometimes, when I create Kafka Cluster and it works. But after a while, the Kafka Exporter has CrashLoopBackOff and is always in this status.
Readiness probe failed: Get "http://10.130.0.49:9404/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Expected behavior A clear and concise description of what you expected to happen.
Environment (please complete the following information):
- Strimzi version: main
- Installation method: OperatorHub
- Kubernetes cluster: Openshift 4.9
- Infrastructure: BareMetal
YAML files and logs
[kafka_exporter] [INFO] 2022/01/04 07:38:01 Starting kafka_exporter (version=1.3.1.redhat-00001, branch=master, revision=eb1f5c4229ce4ca51d64d2034926ce64c60e05e9)
[kafka_exporter] [INFO] 2022/01/04 07:38:01 Build context (go=go1.13, user=worker@pnc-ba-pod-4c2d6e, date=20210708-16:03:34)
[kafka_exporter] [INFO] 2022/01/04 07:38:01 Done Init Clients
[kafka_exporter] [INFO] 2022/01/04 07:38:01 Listening on :9404
[kafka_exporter] [INFO] 2022/01/04 07:38:02 Refreshing client metadata
[kafka_exporter] [INFO] 2022/01/04 07:38:05 concurrent calls detected, waiting for first to finish
[kafka_exporter] [INFO] 2022/01/04 07:38:17 concurrent calls detected, waiting for first to finish
[kafka_exporter] [INFO] 2022/01/04 07:38:32 concurrent calls detected, waiting for first to finish
[kafka_exporter] [INFO] 2022/01/04 07:38:35 concurrent calls detected, waiting for first to finish
[kafka_exporter] [INFO] 2022/01/04 07:38:45 concurrent calls detected, waiting for first to finish
[kafka_exporter] [INFO] 2022/01/04 07:38:47 concurrent calls detected, waiting for first to finish
[kafka_exporter] [INFO] 2022/01/04 07:39:05 concurrent calls detected, waiting for first to finish
[kafka_exporter] [INFO] 2022/01/04 07:39:15 concurrent calls detected, waiting for first to finish
[kafka_exporter] [INFO] 2022/01/04 07:39:15 concurrent calls detected, waiting for first to finish
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 29 (13 by maintainers)
No resources does not mean unlimited. There could be a LimitRange to set some defaults. And even if it wouldn’t be, it means that it might get only little resources which are left because without setting the resources Kube will not know properly what it needs and where to schedule it.
Triaged on 2.8.2022: This has to be fixed in the Kafka Exporter. Once it has a new release, we can update Strimzi to use it. We should also add a warning to the docs about this problem (=> e.g. something like If you don’t use consumer groups, it will not work … just with more fancy wording 😮). It can be added for example somewhere here: https://strimzi.io/docs/operators/latest/deploying.html#con-metrics-kafka-exporter-lag-str
CC @PaulRMellor ^^^
We need to also re-open the discussion about the Kafka Exporter future since it is a long time since a new release or some fixed issues.
I don’t think it really uses the consumer groups. It just reads and decodes the content of it. So it does not trigger its creation.