prometheus-operator: Upgrading to version 0.26.0 breaks PV access

Suspect that this is possibly the result of https://github.com/coreos/prometheus-operator/pull/2109

What did you do? Upgraded from prometheus-operator v0.25 to v0.26

What did you expect to see? Prometheus pods should come up with Prometheus v0.25

What did you see instead? Under which circumstances? Pods crashlooping with

level=error ts=2018-11-30T13:20:02.660393999Z caller=main.go:625 err="opening storage failed: open /prometheus/wal/00000949: permission denied"

This may be only apparent when using persistent volumes

Environment Azure

  • Prometheus Operator version:

v0.26

  • Kubernetes version information:
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
  • Kubernetes cluster kind:

kubeadm

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 7
  • Comments: 21 (10 by maintainers)

Most upvoted comments

^^ Having the same problem

It worked for me by adding the following under prometheusSpec

prometheusSpec: retention: 90d # Data retention value securityContext: runAsNonRoot: false runAsUser: 0 fsGroup: 0

Also having this problem when using Trident as the volume provisioner. Reverting to 0.25.0 resolved the issue.