rook: mon pods crashlooping due to permission issue,
Opening this issue as per https://github.com/rook/rook/discussions/15549#discussioncomment-12581800
mon pods are crashlooping with below error. though we set ROOK_HOSTPATH_REQUIRES_PRIVILEGED to true, also SElinux is enabled.
debug 2025-03-20T07:40:36.408+0000 7fdd7f5f2b00 4 rocksdb: kZlibCompression supported: 1
debug 2025-03-20T07:40:36.408+0000 7fdd7f5f2b00 4 rocksdb: kLZ4HCCompression supported: 1
debug 2025-03-20T07:40:36.408+0000 7fdd7f5f2b00 4 rocksdb: kSnappyCompression supported: 1
debug 2025-03-20T07:40:36.408+0000 7fdd7f5f2b00 4 rocksdb: Fast CRC32 supported: Supported on x86
debug 2025-03-20T07:40:36.408+0000 7fdd7f5f2b00 4 rocksdb: DMutex implementation: pthread_mutex_t
debug 2025-03-20T07:40:36.408+0000 7fdd7f5f2b00 3 rocksdb: [db/db_impl/db_impl_open.cc:2086] DB::Open() failed: Invalid argument: /var/lib/ceph/mon/ceph-a/store.db/CURRENT: does not exist (create_if_missing is false)
debug 2025-03-20T07:40:36.408+0000 7fdd7f5f2b00 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work
debug 2025-03-20T07:40:36.408+0000 7fdd7f5f2b00 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete
debug 2025-03-20T07:40:36.408+0000 7fdd7f5f2b00 -1 rocksdb: Invalid argument: /var/lib/ceph/mon/ceph-a/store.db/CURRENT: does not exist (create_if_missing is false)
debug 2025-03-20T07:40:36.408+0000 7fdd7f5f2b00 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-a': (22) Invalid argument
setting the below environment variable in ceph-operator is not helping. does this only work on OpenShift ?
Please note, when we manually edit the rook-ceph-mon deployment and set runAsUser: 0 under seucrityContext. then the pods came up fine. but these changes are not persistent. whenever operator kick starts or restarts mon pod we face the same issue again.
The Mon deployment has the below spec even after setting the ROOK_HOST_PATH_REQUIRES_PRIVILEGED: true
securityContext:
capabilities:
drop:
- NET_RAW
privileged: true
when we edit the mon deployment and set explicitly run as user 0, then pod comes up fine. even tried changing the directory ownership as 167 ( chown -R 167:167) also didn’t helped.
please adjust the code to run mon as 0 user.
Ceph Version: v1.16.2 k8s version: 1.29.3 ( vanilla k8s) OS: Redhat Linux 8.9 SElinux enabled
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 25 (11 by maintainers)
exec /usr/local/bin/rook: exec format errorI think thats common when mixing archs, speaking from experience 😄