rancher: New Monitoring Prometheus PVC doesn't work with OpenEBS

What kind of request is this (question/bug/enhancement/feature request): Bug

Steps to reproduce (least amount of steps as possible): Rancher 2.2.1 on premise HA setup with OpenEBS as Persistent Storage provider When enabling monitoring, it is never becoming available.

Kubernetes version ist 1.12.5 and Docker 17.03 on Ubuntu 16.04. VM Sizing is 8GB Ram, 4 cores, 120GB disk

Result: The error of the pod prometheus-cluster-monitoring-0 is pod has unbound immediate PersistentVolumeClaims (repeated 3 times)

Digging deeper the PVC is exceeding the PVC name restriction of 63 chars. It is called prometheus-cluster-monitoring-db-prometheus-cluster-monitoring-0 65 chars

failed to provision volume with StorageClass "openebs-jiva-default": Internal Server Error: failed to create volume 'pvc-a333df8a-56e0-11e9-ac14-02000a463c08': response: Service "pvc-a333df8a-56e0-11e9-ac14-02000a463c08-ctrl-svc" is invalid: metadata.labels: Invalid value: "prometheus-project-monitoring-db-prometheus-project-monitoring-0": must be no more than 63 characters

This is something that seems valid as seen on https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set

–

Other details that may be helpful: I have tried an installtion in the cloud of DigitalOcean with their persistent storage provider, and it works perfectly. It ignores the limit of 63 chars.

Here, the Kubernetes is 1.13.5 with Docker 18.09, 4GB RAM, 2 CPU cores

Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): v2.2.1
Installation option (single install/HA): HA

Cluster information

Cluster type (Hosted/Infrastructure Provider/Custom/Imported): Custom, on premise
Machine type (cloud/VM/metal) and specifications (CPU/memory): VM, KVM, 4CPU, 8GB RAM
Kubernetes version (use kubectl version):

1.12.5

Docker version (use docker version):

17.03.3

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 23 (5 by maintainers)

Commits related to this issue

Add option to use release name of prometheus pvc **Problem:** for some storage providers like OpenEBS and ceph, the default pvc name is exceeding the PVC name restriction of 63 chars. https://github.... — committed to guangbochen/system-charts by guangbochen 5 years ago
Upgrade to 0.0.3 version - Embed operator as sub charts + Support to configure operator like other charts + Adjust operator default limit - Add permission to kube-state exporter - Replace loc... — committed to rancher/system-charts by deleted user 5 years ago
Upgrade to 0.0.3 version - Embed operator as sub charts + Support to configure operator like other charts + Adjust operator default limit - Add permission to kube-state exporter - Replace loc... — committed to cbron/system-charts by deleted user 5 years ago
Upgrade to 0.0.3 version - Embed operator as sub charts + Support to configure operator like other charts + Adjust operator default limit - Add permission to kube-state exporter - Replace loc... — committed to prachidamle/system-charts by deleted user 5 years ago
Upgrade to 0.0.3 version - Embed operator as sub charts + Support to configure operator like other charts + Adjust operator default limit - Add permission to kube-state exporter - Replace loc... — committed to brendarearden/system-charts by deleted user 5 years ago
Upgrade to 0.0.3 version - Embed operator as sub charts + Support to configure operator like other charts + Adjust operator default limit - Add permission to kube-state exporter - Replace loc... — committed to paynejacob/system-charts by deleted user 5 years ago

Most upvoted comments

For others that may have this problem, at least on Rancher v2.2.7, after enabling Persistent Storage for Prometheus, click on Show advanced options at the bottom of the Cluster/Project Monitoring Configuration page , and set prometheus.persistence.name to any name and prometheus.persistent.useReleaseName to true.

Goobaii on Aug 23, 2019

Nothing from above worked for me, with Rancher 2.5.9

For anyone else, when installing Rancher monitoring edit yaml and add name manually. Be sure not to make it too long otherwise the full name will still be too much.

...
        storageSpec:
          volumeClaimTemplate:
            metadata:
              name: prometheus
            spec:
...

PVC name = prometheus-prometheus-rancher-monitoring-prometheus-0

Not on Rancher, but I am floored that manually setting the PV is not supported. https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/storage.md

jonathon2nd on Sep 27, 2021