prometheus-operator: Grafana dashboards not getting persisted.

What did you do? Installed prometheus operator and kube prometheus using helm. Configured storagespec for grafana. (To persist grafana data)

What did you expect to see? Expect the dashboards created on grafana UI to persist even after pod restart.

What did you see instead? Under which circumstances? Grafana dashboards are not persisted

Environment

  • Kubernetes version information: Client Version: version.Info{Major:“1”, Minor:“9”, GitVersion:“v1.9.6”, GitCommit:“9f8ebd171479bec0ada837d7ee641dec2f8c6dd1”, GitTreeState:“clean”, BuildDate:“2018-03-21T15:21:50Z”, GoVersion:“go1.9.3”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“9”, GitVersion:“v1.9.6”, GitCommit:“9f8ebd171479bec0ada837d7ee641dec2f8c6dd1”, GitTreeState:“clean”, BuildDate:“2018-03-21T15:13:31Z”, GoVersion:“go1.9.3”, Compiler:“gc”, Platform:“linux/amd64”}

  • Kubernetes cluster kind: Kops

  • Manifests:

grafana:
  storageSpec:
    class: px-repl1
    accessMode: ReadWriteOnce
    resources:
      requests:
        storage: 2Gi
    selector: {}

*Other Details

Issue is not with persisting the data. When I exec to the pod and check the path to which the volume is mounted, grafana.db (file), sessions (folder), plugins (folder) are found. this data gets persisted. But when i see in the UI newly created dashboards are not available even though the data is there, after restart of the pod.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 6
  • Comments: 42 (10 by maintainers)

Most upvoted comments

This repository is very opinionated about how Grafana is managed, if you don’t want to declaratively configure Grafana that’s fine, but it’s a non-goal for this project. I hope you understand and I’d highly recommend for you to try the declarative approach, as a stateful Grafana is an incredible pain to manage even though it’s not necessary and rollback are near impossible. We’ve been running Grafana in an entirely stateless way for years, and have had no problems with the approach.

I hope you understand. I’m closing this here as it’s out of scope for this project.

+23
brancz on Aug 31, 2018

I’ve got the PersistentVolume created and the users and other items are persisted but not the dashboards or data sources. This is really a pain, I’d love to be able to not worry about my dashboards disappearing if the pod is restarted.

+20
JessieAMorris on Aug 30, 2018

These are the steps we make to persist dashboards to outlive pod deletion:

  1. Create your custom dashboard through the UI.

  2. Click on the “Save” button which will pop up a “Cannot save provisioned dashboard” window. There you can click on the “Save JSON to file” button.

  3. Save the json file to your computer.

  4. Using kubectl command-line tool which is configured to communicate with your k8s cluster, create a new ConfigMap using the following command: kubectl create configmap my-custom-dashboard --from-file=path-to-file.json

  5. Add a label to the newly created ConfigMap: kubectl label configmaps my-custom-dashboard grafana_dashboard=1

+14
itayvallach on Jul 16, 2019

This is a huge pain and makes grafana unusable 😦 Really need a fix for this.

+14
jcalahan on Aug 30, 2018

Dashboards are written as jsonnet code and checked into version control. When updated in version control, a redeploy of Grafana is triggered and Grafana is fully provisioned (both dashboards and datasources) from files using the Grafana provisioning features.

Users are managed by an external identity provider using the bitly/oauth2_proxy, and the following Grafana configurations:

[auth]
disable_login_form = true
disable_signout_menu = true
[auth.basic]
enabled = false
[auth.proxy]
auto_sign_up = true
enabled = true
header_name = X-Forwarded-User
[paths]
data = /var/lib/grafana
logs = /var/lib/grafana/logs
plugins = /var/lib/grafana/plugins
provisioning = /etc/grafana/provisioning
[server]
http_addr = 127.0.0.1
http_port = 3001

That as jsonnet code that is (you can read here for more detail on how to customize your Grafana install via jsonnet):

{
  _config+:: {
    grafana+:: {
      config: {
        sections: {
          paths: {
            data: '/var/lib/grafana',
            logs: '/var/lib/grafana/logs',
            plugins: '/var/lib/grafana/plugins',
            provisioning: '/etc/grafana/provisioning',
          },
          server: {
            http_addr: '127.0.0.1',
            http_port: '3001',
          },
          auth: {
            disable_login_form: true,
            disable_signout_menu: true,
          },
          'auth.basic': {
            enabled: false,
          },
          'auth.proxy': {
            enabled: true,
            header_name: 'X-Forwarded-User',
            auto_sign_up: true,
          },
        },
      },
    },
  },
}

And we have Grafana bind to the loopback interface, and because of the above settings any user successfully authenticated with Grafana is automatically created in Grafana. In the future we would like Grafana to also use the users login token for requests against Prometheus as then we can make use of authorization proxies for Prometheus so we can limit what a user can see through Grafana without any customization/configuration necessary.

We manage all alerts as Prometheus alerts that are also checked in to version control and deployed and reloaded by Prometheus whenever checked in.

+9
brancz on Aug 31, 2018

We’re also running into some weirdness. But what is strange is that some assets are persisted. The following objects are persisted between pod restarts (things I have found so far):

  • Users
  • Playlists (but totally corrupted and can’t be removed)
  • Plugins

Unfortunately Dashboards, Data Sources and other objects are totally lost after a pod deletion.

+9
cpressland on Aug 15, 2018

we went the PVC way for users persistence

+5
den-is on Oct 24, 2019

(Did not read everything) You can use ConfigMaps to persist the dashboards instead and prometheus reloads them into you graphana server. Thats what i do

+5
dioguerra on Mar 19, 2019

@den-is try this:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-storage
  namespace: monitoring
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: csi-rbd
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  labels:
    app: grafana
  name: grafana
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      initContainers:
      - command:
        - sh
        - -c
        - "whoami; chmod 777 /var/lib/grafana"
        image: harbor.geniusafc.com/docker.io/busybox
        name: volume-permissions
        volumeMounts:
        - mountPath: /var/lib/grafana
          name: grafana-storage
          readOnly: false
      containers:
      - image: harbor.geniusafc.com/docker.io/grafana/grafana:6.0.0-beta1
        name: grafana
        securityContext:
          runAsNonRoot: true
          runAsUser: 65534
        ports:
        - containerPort: 3000
          name: http
        readinessProbe:
          httpGet:
            path: /api/health
            port: http
        resources:
          limits:
            cpu: 200m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - mountPath: /var/lib/grafana
          name: grafana-storage
          readOnly: false
        - mountPath: /etc/grafana/provisioning/datasources
          name: grafana-datasources
          readOnly: false
        - mountPath: /etc/grafana/provisioning/dashboards
          name: grafana-dashboards
          readOnly: false
        - mountPath: /grafana-dashboard-definitions/0/etcd
          name: grafana-dashboard-etcd
          readOnly: false
        - mountPath: /grafana-dashboard-definitions/0/k8s-cluster-rsrc-use
          name: grafana-dashboard-k8s-cluster-rsrc-use
          readOnly: false
        - mountPath: /grafana-dashboard-definitions/0/k8s-node-rsrc-use
          name: grafana-dashboard-k8s-node-rsrc-use
          readOnly: false
        - mountPath: /grafana-dashboard-definitions/0/k8s-resources-cluster
          name: grafana-dashboard-k8s-resources-cluster
          readOnly: false
        - mountPath: /grafana-dashboard-definitions/0/k8s-resources-namespace
          name: grafana-dashboard-k8s-resources-namespace
          readOnly: false
        - mountPath: /grafana-dashboard-definitions/0/k8s-resources-pod
          name: grafana-dashboard-k8s-resources-pod
          readOnly: false
        - mountPath: /grafana-dashboard-definitions/0/nodes
          name: grafana-dashboard-nodes
          readOnly: false
        - mountPath: /grafana-dashboard-definitions/0/persistentvolumesusage
          name: grafana-dashboard-persistentvolumesusage
          readOnly: false
        - mountPath: /grafana-dashboard-definitions/0/pods
          name: grafana-dashboard-pods
          readOnly: false
        - mountPath: /grafana-dashboard-definitions/0/statefulset
          name: grafana-dashboard-statefulset
          readOnly: false
      nodeSelector:
        beta.kubernetes.io/os: linux

      serviceAccountName: grafana
      volumes:
      - name: grafana-storage
        persistentVolumeClaim:
          claimName: grafana-storage
          readOnly: false
      - name: grafana-datasources
        secret:
          secretName: grafana-datasources
      - configMap:
          name: grafana-dashboards
        name: grafana-dashboards
      - configMap:
          name: grafana-dashboard-etcd
        name: grafana-dashboard-etcd
      - configMap:
          name: grafana-dashboard-k8s-cluster-rsrc-use
        name: grafana-dashboard-k8s-cluster-rsrc-use
      - configMap:
          name: grafana-dashboard-k8s-node-rsrc-use
        name: grafana-dashboard-k8s-node-rsrc-use
      - configMap:
          name: grafana-dashboard-k8s-resources-cluster
        name: grafana-dashboard-k8s-resources-cluster
      - configMap:
          name: grafana-dashboard-k8s-resources-namespace
        name: grafana-dashboard-k8s-resources-namespace
      - configMap:
          name: grafana-dashboard-k8s-resources-pod
        name: grafana-dashboard-k8s-resources-pod
      - configMap:
          name: grafana-dashboard-nodes
        name: grafana-dashboard-nodes
      - configMap:
          name: grafana-dashboard-persistentvolumesusage
        name: grafana-dashboard-persistentvolumesusage
      - configMap:
          name: grafana-dashboard-pods
        name: grafana-dashboard-pods
      - configMap:
          name: grafana-dashboard-statefulset
        name: grafana-dashboard-statefulset
+4
zgfh on Jun 13, 2019

For me it is working with an existing PVC/PV, I am creating it before the first install of Grafana.

values.yaml

persistence:
  enabled: true
  storageClassName: standard
  finalizers:
    - kubernetes.io/pvc-protection
  accessModes:
     - ReadWriteOnce
  size: 15Gi
  # annotations: {}
  # subPath: ""
  existingClaim: grafana-pvc

pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: grafana
  namespace: monitoring
  labels:
    app: grafana
spec:
  storageClassName: "standard"
  capacity:
    storage: 15Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  gcePersistentDisk:
    pdName: grafana
    fsType: ext4

pvc.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: grafana-pvc
  namespace: monitoring
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 15Gi
  storageClassName: standard
  selector:
    matchLabels:
      app: "grafana"
+1
Tim-Schwalbe on May 26, 2022
Read more comments on GitHub
← prometheus-operator: grafana-dashboards ConfigMap is invalid (metadata.annotations too long)
prometheus-operator: Grafana error using persisten storage →