rook: mon pods crashlooping due to permission issue,

Opening this issue as per https://github.com/rook/rook/discussions/15549#discussioncomment-12581800

mon pods are crashlooping with below error. though we set ROOK_HOSTPATH_REQUIRES_PRIVILEGED to true, also SElinux is enabled.

debug 2025-03-20T07:40:36.408+0000 7fdd7f5f2b00  4 rocksdb:     kZlibCompression supported: 1
debug 2025-03-20T07:40:36.408+0000 7fdd7f5f2b00  4 rocksdb:     kLZ4HCCompression supported: 1
debug 2025-03-20T07:40:36.408+0000 7fdd7f5f2b00  4 rocksdb:     kSnappyCompression supported: 1
debug 2025-03-20T07:40:36.408+0000 7fdd7f5f2b00  4 rocksdb: Fast CRC32 supported: Supported on x86
debug 2025-03-20T07:40:36.408+0000 7fdd7f5f2b00  4 rocksdb: DMutex implementation: pthread_mutex_t
debug 2025-03-20T07:40:36.408+0000 7fdd7f5f2b00  3 rocksdb: [db/db_impl/db_impl_open.cc:2086] DB::Open() failed: Invalid argument: /var/lib/ceph/mon/ceph-a/store.db/CURRENT: does not exist (create_if_missing is false)
debug 2025-03-20T07:40:36.408+0000 7fdd7f5f2b00  4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work
debug 2025-03-20T07:40:36.408+0000 7fdd7f5f2b00  4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete
debug 2025-03-20T07:40:36.408+0000 7fdd7f5f2b00 -1 rocksdb: Invalid argument: /var/lib/ceph/mon/ceph-a/store.db/CURRENT: does not exist (create_if_missing is false)
debug 2025-03-20T07:40:36.408+0000 7fdd7f5f2b00 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-a': (22) Invalid argument

setting the below environment variable in ceph-operator is not helping. does this only work on OpenShift ?

Image

Please note, when we manually edit the rook-ceph-mon deployment and set runAsUser: 0 under seucrityContext. then the pods came up fine. but these changes are not persistent. whenever operator kick starts or restarts mon pod we face the same issue again.

The Mon deployment has the below spec even after setting the ROOK_HOST_PATH_REQUIRES_PRIVILEGED: true

securityContext: 
     capabilities: 
         drop: 
           - NET_RAW
     privileged: true 

when we edit the mon deployment and set explicitly run as user 0, then pod comes up fine. even tried changing the directory ownership as 167 ( chown -R 167:167) also didn’t helped.

please adjust the code to run mon as 0 user.

Ceph Version: v1.16.2 k8s version: 1.29.3 ( vanilla k8s) OS: Redhat Linux 8.9 SElinux enabled

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 25 (11 by maintainers)

Most upvoted comments

exec /usr/local/bin/rook: exec format error I think thats common when mixing archs, speaking from experience 😄