strimzi-kafka-operator: [Bug]: Grafana dashboards through Helm chart missing datasource
Bug Description
Setting dashboards.enabled: true adds dashboards to Grafana that give the following error Failed to upgrade legacy queries Datasource ${DS_PROMETHEUS} was not found in Grafana 9.5.2.
I suspect this is because the dashboards only set DS_PROMETHEUS on import through the UI.
Steps to reproduce
- Set
dashboards.enabledtotrue - Wait until dashboards are loaded in Grafana
- Go to one of the added dashboards
- Observe the error
Expected behavior
Same behavior as on manual import; dashboard should use a datasource, preferably the default datasource.
Strimzi version
0.35.0
Kubernetes version
1.26
Installation method
Helm chart
Infrastructure
Azure Kubernetes
Configuration files and logs
No response
Additional context
This could probably be solved by removing the __inputs part of the dashboard and adding the following to templating.list:
{
"current": {
"selected": false,
},
"description": null,
"error": null,
"hide": 0,
"includeAll": false,
"label": null,
"multi": false,
"name": "DS_PROMETHEUS",
"options": [],
"query": "prometheus",
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"type": "datasource"
}
Which is what we currently use for some of the Strimzi dashboards that we manually deploy
Example of a broken dashboard:
About this issue
- Original URL
- State: closed
- Created a year ago
- Reactions: 1
- Comments: 16 (9 by maintainers)
Commits related to this issue
- Grafana dashboards: use empty default datasource instead of var cf. https://github.com/strimzi/strimzi-kafka-operator/issues/8545 This appears to be sufficient; the default data source seems to be d... — committed to tjanson/strimzi-kafka-operator by deleted user a year ago
- Grafana dashboards: use empty default datasource instead of var cf. https://github.com/strimzi/strimzi-kafka-operator/issues/8545 This appears to be sufficient; the default data source seems to be d... — committed to tjanson/strimzi-kafka-operator by deleted user a year ago
- Grafana dashboards: use empty default datasource instead of var cf. https://github.com/strimzi/strimzi-kafka-operator/issues/8545 This appears to be sufficient; the default data source seems to be d... — committed to tjanson/strimzi-kafka-operator by deleted user a year ago
- Grafana dashboards: use empty default datasource instead of var cf. https://github.com/strimzi/strimzi-kafka-operator/issues/8545 This appears to be sufficient; the default data source seems to be d... — committed to tjanson/strimzi-kafka-operator by deleted user a year ago
kube-prometheus-stack which is imo the defacto standard for deploying prometheus-grafana in kubernetes provides default dashboards with their chart/operator. It seems to me that whatever solution they use for the datasource should be the correct default for strimzi as well.
I’m not sure but I think this might be the configuration used by kube-prometheus-stack?
https://github.com/prometheus-community/helm-charts/blob/eba5b198f597a39f2d40d3edd209dfa09429623e/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/alertmanager-overview.yaml#LL476C1-L492C19