prometheus-operator: Opening storage failed, permission denied

Similar to #541, I am having the same issue where when I include my usual volumeClaimTemplate block to my Prometheus configuration, my pods get stuck in a crash loop with the error:

level=info ts=2017-12-15T22:51:22.073242152Z caller=main.go:314 msg="Starting TSDB"
level=error ts=2017-12-15T22:51:22.074477852Z caller=main.go:323 msg="Opening storage failed" err="open DB in /var/prometheus/data: open /var/prometheus/data/181997666: permission denied"

This only started happening once I upgraded from Prometheus 1.8 to 2.0.

Things I have tried: Deleting my persistent volumes and re-adding them, and adding the following to my prometheus-k8s.yaml file:

spec:
  securityContext:
    runAsUser: 1000
    fsGroup: 2000
    runAsNonRoot: true

I am running the newest version of prometheus-operator (0.15.0). I also can’t exec -ti to ssh to the pod so I can’t look at or change permissions on /var/prometheus/data. Any ideas?

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 16 (9 by maintainers)

Most upvoted comments

make a config folder in the location of your docker-compose.yml. insert the files prometheus.yml and alert_rules.yml in the config folder. This code worked for me on v2.10 prometheus image

  prometheus:
    image: prom/prometheus
    container_name: prometheus
    volumes:
      - './config:/prometheus'
    restart: unless-stopped
    ports:
      - 9090:9090

+10

alokhom on May 24, 2019

make a config folder in the location of your docker-compose.yml. insert the files prometheus.yml and alert_rules.yml in the config folder. This code worked for me on v2.10 prometheus image

  prometheus:
    image: prom/prometheus
    container_name: prometheus
    volumes:
      - './config:/prometheus'
    restart: unless-stopped
    ports:
      - 9090:9090

alokhom on May 24, 2019

If you have a dedicated Prometheus user give that user permission to access the /var/lib/prometheus folder You can refer this article if this helps: https://linuxhint.com/install_prometheus_ubuntu/

yashrahurikar23 on Jun 10, 2021

@brancz NFSv4 is POSIX compliant I believe? I think you are thinking of the old NFSv3? Certainly AWS EFS is claiming POSIX compliance for its NFSv4 clusters.

In the prometheus/prometheus#3534 example they solved the NFS problem by updating to a newer tsdb build rather than switching from NFS: “For what it’s worth, we built Prometheus against the latest prometheus/tsdb and that solved this particular issue with NFS”

And in the prometheus/prometheus#3506 example they also cite a tsdb patch as the solution.

I guess if ext4 is not ‘POSIX compliant’, then it is all relative: People talk a lot about “POSIX compliance,” but in reality most file system implementations do not strictly adhere to the spec, including local Linux file systems like ext4 and XFS. For example, for performance reasons, the atomicity requirements for reads are relaxed: processing reading from a file that is also being written may see torn results.

whereisaaron on Feb 9, 2018