prometheus-operator: Prometheus operator does not cleanup PVCs created by Prometheus resources

What did you do? Install prometheus-operator and kube-prometheus using the provided helm charts, configuring a dynamically-provisioned PVC for Prometheus with the following config in kube-prometheus/default-values.yaml (following the Prometheus Operator storage user guide):

prometheus:
  storageSpec:
    volumeClaimTemplate:
      spec:
        storageClassName: gp2
        accessModes:
          - "ReadWriteOnce"
        resources:
          requests:
            storage: 50Gi

The PVC, PV, and underlying AWS EBS volume were dynamically provisioned and successfully mounted inside the Prometheus pod. However, the PVC resource is not cleaned up/deleted as I would expect.

What did you expect to see? Upon a helm delete kube-prometheus, I expected prometheus-operator to delete the dynamic PVC and PV (and the underlying EBS), as the gp2 StorageClass has a ReclaimPolicy of delete.

What did you see instead? Under which circumstances? The PVC resource was not deleted by prometheus-operator, so the underlying PV and EBS were not cleaned up.

Environment

• Helm chart versions: prometheus-operator version: 0.0.29 kube-prometheus version: 0.0.105

• Prometheus Operator version: v0.20.0

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:08:19Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind: Cluster created with kubeadm on AWS EC2 instances.

• Manifests:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    prometheus-operator-input-checksum: 795d8a8b34b9e053a5635059490824ca
  creationTimestamp: 2018-09-14T14:04:17Z
  generation: 1
  labels:
    app: prometheus
    chart: prometheus-0.0.51
    heritage: Tiller
    prometheus: kube-prometheus
    release: kube-prometheus
  name: prometheus-kube-prometheus
  namespace: kube-system
  ownerReferences:
  - apiVersion: monitoring.coreos.com/v1
    blockOwnerDeletion: true
    controller: true
    kind: Prometheus
    name: kube-prometheus
    uid: 11d5ed89-b827-11e8-871b-02e9efe6218a
  resourceVersion: "2067"
  selfLink: /apis/apps/v1/namespaces/kube-system/statefulsets/prometheus-kube-prometheus
  uid: 11ebb9cd-b827-11e8-871b-02e9efe6218a
spec:
  podManagementPolicy: OrderedReady
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: prometheus
      prometheus: kube-prometheus
  serviceName: prometheus-operated
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: prometheus
        prometheus: kube-prometheus
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app: prometheus
                  prometheus: kube-prometheus
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - args:
        - --config.file=/etc/prometheus/config_out/prometheus.env.yaml
        - --storage.tsdb.path=/prometheus
        - --storage.tsdb.retention=24h
        - --web.enable-lifecycle
        - --storage.tsdb.no-lockfile
        - --web.external-url=http://XXXXXXXXX/prometheus
        - --web.route-prefix=/prometheus
        image: quay.io/prometheus/prometheus:v2.2.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 6
          httpGet:
            path: /prometheus/-/healthy
            port: web
            scheme: HTTP
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 3
        name: prometheus
        ports:
        - containerPort: 9090
          name: web
          protocol: TCP
        readinessProbe:
          failureThreshold: 120
          httpGet:
            path: /prometheus/-/ready
            port: web
            scheme: HTTP
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 3
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/prometheus/config_out
          name: config-out
          readOnly: true
        - mountPath: /etc/prometheus/rules
          name: rules
        - mountPath: /prometheus
          name: prometheus-kube-prometheus-db
          subPath: prometheus-db
        - mountPath: /etc/prometheus/secrets/prometheus-etcd-client-certs
          name: secret-prometheus-etcd-client-certs
          readOnly: true
      - args:
        - --reload-url=http://localhost:9090/prometheus/-/reload
        - --config-file=/etc/prometheus/config/prometheus.yaml
        - --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
        command:
        - /bin/prometheus-config-reloader
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        image: quay.io/coreos/prometheus-config-reloader:v0.20.0
        imagePullPolicy: IfNotPresent
        name: prometheus-config-reloader
        resources:
          limits:
            cpu: 10m
            memory: 50Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/prometheus/config
          name: config
        - mountPath: /etc/prometheus/rules
          name: rules
        - mountPath: /etc/prometheus/config_out
          name: config-out
      - args:
        - --webhook-url=http://localhost:9090/prometheus/-/reload
        - --volume-dir=/etc/prometheus/rules
        image: quay.io/coreos/configmap-reload:v0.0.1
        imagePullPolicy: IfNotPresent
        name: alerting-rule-files-configmap-reloader
        resources:
          limits:
            cpu: 5m
            memory: 10Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/prometheus/rules
          name: rules
          readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 2000
        runAsNonRoot: true
        runAsUser: 1000
      serviceAccount: kube-prometheus
      serviceAccountName: kube-prometheus
      terminationGracePeriodSeconds: 600
      volumes:
      - name: config
        secret:
          defaultMode: 420
          secretName: prometheus-kube-prometheus
      - emptyDir: {}
        name: config-out
      - configMap:
          defaultMode: 420
          name: prometheus-kube-prometheus-rulefiles
        name: rules
      - name: secret-prometheus-etcd-client-certs
        secret:
          defaultMode: 420
          secretName: prometheus-etcd-client-certs
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - metadata:
      creationTimestamp: null
      name: prometheus-kube-prometheus-db
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 50Gi
      storageClassName: gp2
    status:
      phase: Pending
status:
  collisionCount: 0
  currentReplicas: 1
  currentRevision: prometheus-kube-prometheus-67fdb4647
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updateRevision: prometheus-kube-prometheus-67fdb4647

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
  creationTimestamp: 2018-09-14T14:04:17Z
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: prometheus
    prometheus: kube-prometheus
  name: prometheus-kube-prometheus-db-prometheus-kube-prometheus-0
  namespace: kube-system
  resourceVersion: "1878"
  selfLink: /api/v1/namespaces/kube-system/persistentvolumeclaims/prometheus-kube-prometheus-db-prometheus-kube-prometheus-0
  uid: 11efe8d6-b827-11e8-871b-02e9efe6218a
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: gp2
  volumeName: pvc-11efe8d6-b827-11e8-871b-02e9efe6218a
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Gi
  phase: Bound

---
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    kubernetes.io/createdby: aws-ebs-dynamic-provisioner
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
  creationTimestamp: 2018-09-14T14:04:28Z
  finalizers:
  - kubernetes.io/pv-protection
  labels:
    failure-domain.beta.kubernetes.io/region: eu-central-1
    failure-domain.beta.kubernetes.io/zone: eu-central-1b
  name: pvc-11efe8d6-b827-11e8-871b-02e9efe6218a
  resourceVersion: "1877"
  selfLink: /api/v1/persistentvolumes/pvc-11efe8d6-b827-11e8-871b-02e9efe6218a
  uid: 1823f2a2-b827-11e8-871b-02e9efe6218a
spec:
  accessModes:
  - ReadWriteOnce
  awsElasticBlockStore:
    fsType: ext4
    volumeID: aws://eu-central-1b/vol-XXXXXXXXXXX
  capacity:
    storage: 50Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: prometheus-kube-prometheus-db-prometheus-kube-prometheus-0
    namespace: kube-system
    resourceVersion: "1866"
    uid: 11efe8d6-b827-11e8-871b-02e9efe6218a
  persistentVolumeReclaimPolicy: Delete
  storageClassName: gp2
status:
  phase: Bound

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: 2018-09-14T14:01:07Z
  name: gp2
  resourceVersion: "707"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/gp2
  uid: a03f0343-b826-11e8-871b-02e9efe6218a
parameters:
  encrypted: "true"
  kmsKeyId: XXXXXXXXXX
  type: gp2
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete
volumeBindingMode: Immediate

• Prometheus Operator Logs: No relevant logs from prometheus-operator after helm delete kube-prometheus.

---

Is it is intended that the PVC is not deleted by Prometheus Operator? And if so, would it be possible to add some field or configuration option to cleanup unused PVCs that were created by Prometheus resources definitions?