strimzi-kafka-operator: [Bug]: Grafana dashboards through Helm chart missing datasource

Bug Description

Setting dashboards.enabled: true adds dashboards to Grafana that give the following error Failed to upgrade legacy queries Datasource ${DS_PROMETHEUS} was not found in Grafana 9.5.2. I suspect this is because the dashboards only set DS_PROMETHEUS on import through the UI.

Steps to reproduce

  1. Set dashboards.enabled to true
  2. Wait until dashboards are loaded in Grafana
  3. Go to one of the added dashboards
  4. Observe the error

Expected behavior

Same behavior as on manual import; dashboard should use a datasource, preferably the default datasource.

Strimzi version

0.35.0

Kubernetes version

1.26

Installation method

Helm chart

Infrastructure

Azure Kubernetes

Configuration files and logs

No response

Additional context

This could probably be solved by removing the __inputs part of the dashboard and adding the following to templating.list:

      {
        "current": {
          "selected": false,
        },
        "description": null,
        "error": null,
        "hide": 0,
        "includeAll": false,
        "label": null,
        "multi": false,
        "name": "DS_PROMETHEUS",
        "options": [],
        "query": "prometheus",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "type": "datasource"
      }

Which is what we currently use for some of the Strimzi dashboards that we manually deploy

Example of a broken dashboard: image

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 1
  • Comments: 16 (9 by maintainers)

Commits related to this issue

Most upvoted comments

Triaged on 1.6.2023: This might be a bug and should be tracked. But it is not obvious what is the right fix which would work for everyone and not fix it for some and break for others. Should be looked into in more detail.

kube-prometheus-stack which is imo the defacto standard for deploying prometheus-grafana in kubernetes provides default dashboards with their chart/operator. It seems to me that whatever solution they use for the datasource should be the correct default for strimzi as well.

I’m not sure but I think this might be the configuration used by kube-prometheus-stack?

https://github.com/prometheus-community/helm-charts/blob/eba5b198f597a39f2d40d3edd209dfa09429623e/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/alertmanager-overview.yaml#LL476C1-L492C19