rook: rook v1.15.5 and v1.15.0 adds wrong keyring to osds repeatedly
- Bug Report
Deviation from expected behavior:
rook adds a keyring into a newly created osd.1. osd.1 is a reused number - more on this soon.
Expected behavior:
rook to add correct keyring.
How to reproduce it (minimal and precise):
What I did:
- find the osd / host
- update the keyring on the host path to the key returned by
ceph auth ls
What happened:
- a few minutes later, the hostpath file was updated again with the incorrect key.
Additional information:
- osd.1 is an old osd number, of an osd that was deleted
- I did manually delete the keyring multiple times, but something (likely the rook operator) creates a new keyring
- That keyring is always different when checking ceph auth ls - so the behaviour is correct
- However the keyring that is stored inside osd.1 is always the SAME, incorrect one
Further Additional information:
- Many osds where newly added at the same time on different hosts
- There used to be an osd.1 deployment on server118, while the current deployment targets server116
- This might be a race condition when multiple prepare jobs create osds at the same time
Cluster Status to submit:
Environment:
- OS (e.g. from /etc/os-release): alpine-linux
- Kernel (e.g.
uname -a):6.6.61-0-lts - Cloud provider or hardware configuration: hardware
- Rook version (use
rook versioninside of a Rook Pod):v1.15.0 and v1.15.5 both confirmed - Storage backend version (e.g. for ceph do
ceph -v): ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) - Kubernetes version (use
kubectl version): 1.31.1 - Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): kubeadm
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 34 (9 by maintainers)
Not yet, it is still under investigation. To help the ceph team understand the priority/severity, please do comment on the ceph tracker: https://tracker.ceph.com/issues/69173 (looks like https://tracker.ceph.com/issues/69697 is a dup of that one)
@sp98 The OSD is being created, it’ a new one:
osd output
activate output (showing the correct key, redacted):