rook: `useAllDevices: true` targets the wrong device after upgrading to `v1.13.2`
Is this a bug report or feature request?
- Bug Report
Deviation from expected behavior:
I upgraded to v1.13.2 and suddenly one of my 3 nodes OSD (osd-0) is not starting up with Init:Error status. I did k -n rook-ceph logs rook-ceph-osd-0-xxxx -c activate and got this:
+ OSD_ID=0
+ CEPH_FSID=a7a469e6-2749-4ad7-9d7b-b5a790b93f96
+ OSD_UUID=1716a1ea-68db-4748-8393-549fcdf4fc47
+ OSD_STORE_FLAG=--bluestore
+ OSD_DATA_DIR=/var/lib/ceph/osd/ceph-0
+ CV_MODE=raw
+ DEVICE=/dev/sdb
+ cp --no-preserve=mode /etc/temp-ceph/ceph.conf /etc/ceph/ceph.conf
+ python3 -c '
import configparser
config = configparser.ConfigParser()
config.read('\''/etc/ceph/ceph.conf'\'')
if not config.has_section('\''global'\''):
config['\''global'\''] = {}
if not config.has_option('\''global'\'','\''fsid'\''):
config['\''global'\'']['\''fsid'\''] = '\''a7a469e6-2749-4ad7-9d7b-b5a790b93f96'\''
with open('\''/etc/ceph/ceph.conf'\'', '\''w'\'') as configfile:
config.write(configfile)
'
+ ceph -n client.admin auth get-or-create osd.0 mon 'allow profile osd' mgr 'allow profile osd' osd 'allow *' -k /etc/ceph/admin-keyring-store/keyring
[osd.0]
key = AQBkf/piq5gmNhAATvHyIShR2qpsG1A8Lp5BUQ==
+ [[ raw == \l\v\m ]]
++ mktemp
+ OSD_LIST=/tmp/tmp.LLr10BTmmg
+ ceph-volume raw list /dev/sdb
stderr: lsblk: /dev/sdb: not a block device
Traceback (most recent call last):
File "/usr/sbin/ceph-volume", line 11, in <module>
load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__
self.main(self.argv)
File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
return f(*a, **kw)
File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main
terminal.dispatch(self.mapper, subcommand_args)
File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
instance.main()
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main
terminal.dispatch(self.mapper, self.argv)
File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
instance.main()
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/list.py", line 174, in main
self.list(args)
File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
return func(*a, **kw)
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/list.py", line 130, in list
report = self.generate(args.device)
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/list.py", line 99, in generate
info_devices.append(disk.lsblk(dev, abspath=True))
File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 245, in lsblk
abspath=abspath)
File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 337, in lsblk_all
raise RuntimeError(f"Error: {err}")
RuntimeError: Error: ['lsblk: /dev/sdb: not a block device']
As seen above, it can’t find /dev/sdb which indeed doesn’t exist. It should be /dev/sda and I don’t know why. So, I decided to not use useAllDevices: true and use manual selecting devices by names instead. I edited the CephCluster to not use all devices and added storage.nodes[] to it and preparing to manually remove the OSDs and readd them because according to https://github.com/rook/rook/discussions/13357#discussioncomment-7823653 changing that doesn’t do anything to existing cluster. But, magically just by editing the CephCluster fixed the osd-0 problem, it weirdly picked the right device now. But, after a couple of minutes, osd-2 ended up in the same state as osd-0 which is finding a non existing /dev/sdb instead of /dev/sda and fails to start.
Expected behavior:
OSDs should be up just like before doing the upgrade.
How to reproduce it (minimal and precise):
File(s) to submit:
- Cluster CR (custom resource), typically called
cluster.yaml, if necessary https://github.com/budimanjojo/home-cluster/blob/53838a50898326c793529988dee402206574272e/cluster/apps/rook-ceph/rook-ceph/cluster/helmrelease.yaml
Logs to submit:
-
Operator’s logs, if necessary
-
Crashing pod(s) logs, if necessary
To get logs, use
kubectl -n <namespace> logs <pod name>When pasting logs, always surround them with backticks or use theinsert codebutton from the Github UI. Read GitHub documentation if you need help.
Cluster Status to submit:
-
Output of kubectl commands, if necessary
To get the health of the cluster, use
kubectl rook-ceph healthTo get the status of the cluster, usekubectl rook-ceph ceph statusFor more details, see the Rook kubectl Plugin
cluster:
id: a7a469e6-2749-4ad7-9d7b-b5a790b93f96
health: HEALTH_WARN
1 osds down
1 host (1 osds) down
Degraded data redundancy: 5941/17823 objects degraded (33.333%), 56 pgs degraded, 113 pgs undersized
services:
mon: 3 daemons, quorum n,q,r (age 100m)
mgr: a(active, since 6h)
osd: 3 osds: 2 up (since 31m), 3 in (since 34m)
data:
pools: 9 pools, 113 pgs
objects: 5.94k objects, 19 GiB
usage: 53 GiB used, 297 GiB / 350 GiB avail
pgs: 5941/17823 objects degraded (33.333%)
57 active+undersized
56 active+undersized+degraded
io:
client: 101 KiB/s wr, 0 op/s rd, 2 op/s wr
Environment:
- OS (e.g. from /etc/os-release): Talos v1.6.0
- Kernel (e.g.
uname -a): 6.1.67 - Cloud provider or hardware configuration:
- Rook version (use
rook versioninside of a Rook Pod): - Storage backend version (e.g. for ceph do
ceph -v): ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) - Kubernetes version (use
kubectl version): v1.28.5 - Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift):
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 39 (28 by maintainers)
I just restarted our cluster without hitting the issue. I can confirm that the issue is fixed
We hit this too!
Using explicit node and device configuration like
ROOK_BLOCK_PATHandblocksymlink set to/dev/sd*.All nodes running
Using Rook Operator
v1.14.4and Clusterv1.14.4with Cephv18.2.2-20240521installed via Helm.@budimanjojo - the “expected to keep a symlink to the device” is likely in reference to the
/dev/sdatype names can point to different device upon reboot. These are pretty much assigned by who responds first to device discovery. If a host has multiple devices, then there is no guarantee that/dev/sdawill point to the same device on next reboot.My apologies if you know this, but I’ll write for whoever else comes across this… Linux automatically creates symlinks for you which have consistent names and point the appropriate device. Nothing you need to do other than use the correct path. There are multiple ones you can use located:
The
by-idis the one I like to use, for me the the device and each partition on/dev/sdahas multiple referenced symlinks:So I can use
/dev/disk/by-id/ata-Samsung_SSD_860_EVO_M.2_1TB_S415NS0R506801Nor/dev/disk/by-id/wwn-0x5002538e3150c98ato always reference the same device. Shouldsdaandsdbswap locations no issues, these symlinks generated at boot will point to the correct one.@cameronraysmith I moved your comment to #14436, if we could follow up there for the fix, thanks for reporting.
I assumed rook is doing so currently because that’s how i interpreted the doc section
Storage Selection Settings -> devices…Especially the udev-part …
this will not change after rebootsI’ll take a look if that can be done.
ROOK_BLOCK_PATH has to be possible root cause. In case of
updatescenario, theROOK_BLOCK_PATHis fetched from the currently running OSD deployment. So even though the/dev/by-id/*is now pointing to, say,/dev/sdb, rook operator will setROOK_BLOCK_PATHto, say,/dev/sdeby reading it from the currently running OSD deployment.So we might have to change how
ROOK_BLOCK_PATHis set in case ofUpdatescenario.@sethjones our workaround is to delete the deployments and let the operator recreate them with the right device name
@Rudi3 I ended up with setting
useAllDevices: falseand define the devices like so: https://github.com/budimanjojo/home-cluster/blob/8398f1abea3dc431522386f45f75729a8ea9ae8c/cluster/apps/rook-ceph/rook-ceph/cluster/helmrelease.yaml#L88-L97