rook: `useAllDevices: true` targets the wrong device after upgrading to `v1.13.2`

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior:

I upgraded to v1.13.2 and suddenly one of my 3 nodes OSD (osd-0) is not starting up with Init:Error status. I did k -n rook-ceph logs rook-ceph-osd-0-xxxx -c activate and got this:

+ OSD_ID=0
+ CEPH_FSID=a7a469e6-2749-4ad7-9d7b-b5a790b93f96
+ OSD_UUID=1716a1ea-68db-4748-8393-549fcdf4fc47
+ OSD_STORE_FLAG=--bluestore
+ OSD_DATA_DIR=/var/lib/ceph/osd/ceph-0
+ CV_MODE=raw
+ DEVICE=/dev/sdb
+ cp --no-preserve=mode /etc/temp-ceph/ceph.conf /etc/ceph/ceph.conf
+ python3 -c '
import configparser

config = configparser.ConfigParser()
config.read('\''/etc/ceph/ceph.conf'\'')

if not config.has_section('\''global'\''):
    config['\''global'\''] = {}

if not config.has_option('\''global'\'','\''fsid'\''):
    config['\''global'\'']['\''fsid'\''] = '\''a7a469e6-2749-4ad7-9d7b-b5a790b93f96'\''

with open('\''/etc/ceph/ceph.conf'\'', '\''w'\'') as configfile:
    config.write(configfile)
'
+ ceph -n client.admin auth get-or-create osd.0 mon 'allow profile osd' mgr 'allow profile osd' osd 'allow *' -k /etc/ceph/admin-keyring-store/keyring
[osd.0]
        key = AQBkf/piq5gmNhAATvHyIShR2qpsG1A8Lp5BUQ==
+ [[ raw == \l\v\m ]]
++ mktemp
+ OSD_LIST=/tmp/tmp.LLr10BTmmg
+ ceph-volume raw list /dev/sdb
 stderr: lsblk: /dev/sdb: not a block device
Traceback (most recent call last):
  File "/usr/sbin/ceph-volume", line 11, in <module>
    load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
  File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__
    self.main(self.argv)
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
    return f(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main
    terminal.dispatch(self.mapper, subcommand_args)
  File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
    instance.main()
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main
    terminal.dispatch(self.mapper, self.argv)
  File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
    instance.main()
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/list.py", line 174, in main
    self.list(args)
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/list.py", line 130, in list
    report = self.generate(args.device)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/list.py", line 99, in generate
    info_devices.append(disk.lsblk(dev, abspath=True))
  File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 245, in lsblk
    abspath=abspath)
  File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 337, in lsblk_all
    raise RuntimeError(f"Error: {err}")
RuntimeError: Error: ['lsblk: /dev/sdb: not a block device']

As seen above, it can’t find /dev/sdb which indeed doesn’t exist. It should be /dev/sda and I don’t know why. So, I decided to not use useAllDevices: true and use manual selecting devices by names instead. I edited the CephCluster to not use all devices and added storage.nodes[] to it and preparing to manually remove the OSDs and readd them because according to https://github.com/rook/rook/discussions/13357#discussioncomment-7823653 changing that doesn’t do anything to existing cluster. But, magically just by editing the CephCluster fixed the osd-0 problem, it weirdly picked the right device now. But, after a couple of minutes, osd-2 ended up in the same state as osd-0 which is finding a non existing /dev/sdb instead of /dev/sda and fails to start.

Expected behavior:

OSDs should be up just like before doing the upgrade.

How to reproduce it (minimal and precise):

File(s) to submit:

Logs to submit:

  • Operator’s logs, if necessary

  • Crashing pod(s) logs, if necessary

    To get logs, use kubectl -n <namespace> logs <pod name> When pasting logs, always surround them with backticks or use the insert code button from the Github UI. Read GitHub documentation if you need help.

Cluster Status to submit:

  • Output of kubectl commands, if necessary

    To get the health of the cluster, use kubectl rook-ceph health To get the status of the cluster, use kubectl rook-ceph ceph status For more details, see the Rook kubectl Plugin

  cluster:
    id:     a7a469e6-2749-4ad7-9d7b-b5a790b93f96
    health: HEALTH_WARN
            1 osds down
            1 host (1 osds) down
            Degraded data redundancy: 5941/17823 objects degraded (33.333%), 56 pgs degraded, 113 pgs undersized

  services:
    mon: 3 daemons, quorum n,q,r (age 100m)
    mgr: a(active, since 6h)
    osd: 3 osds: 2 up (since 31m), 3 in (since 34m)

  data:
    pools:   9 pools, 113 pgs
    objects: 5.94k objects, 19 GiB
    usage:   53 GiB used, 297 GiB / 350 GiB avail
    pgs:     5941/17823 objects degraded (33.333%)
             57 active+undersized
             56 active+undersized+degraded

  io:
    client:   101 KiB/s wr, 0 op/s rd, 2 op/s wr

Environment:

  • OS (e.g. from /etc/os-release): Talos v1.6.0
  • Kernel (e.g. uname -a): 6.1.67
  • Cloud provider or hardware configuration:
  • Rook version (use rook version inside of a Rook Pod):
  • Storage backend version (e.g. for ceph do ceph -v): ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)
  • Kubernetes version (use kubectl version): v1.28.5
  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift):

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 39 (28 by maintainers)

Most upvoted comments

I just restarted our cluster without hitting the issue. I can confirm that the issue is fixed

We hit this too!

Using explicit node and device configuration like

  storage:
    useAllNodes: false
    useAllDevices: false
    nodes:
      - name: node-01
        devices:
          - name: /dev/disk/by-id/scsi-3600508b1001c8f492048c5852615315a
      - name: node-02
        devices:
          - name: /dev/disk/by-id/scsi-3600508b1001c7b9d2c062a3c6ecde98b
      - name: node-03
        devices:
          - name: /dev/disk/by-id/scsi-3600508b1001c2b2d7712ce1cab7447f8

ROOK_BLOCK_PATH and block symlink set to /dev/sd*.

All nodes running

NAME="Fedora Linux"
VERSION="40.20240519.3.0 (CoreOS)"
PRETTY_NAME="Fedora CoreOS 40.20240519.3.0"
CPE_NAME="cpe:/o:fedoraproject:fedora:40"
VARIANT="CoreOS"
VARIANT_ID=coreos
OSTREE_VERSION='40.20240519.3.0'

Using Rook Operator v1.14.4 and Cluster v1.14.4 with Ceph v18.2.2-20240521 installed via Helm.

@budimanjojo - the “expected to keep a symlink to the device” is likely in reference to the /dev/sda type names can point to different device upon reboot. These are pretty much assigned by who responds first to device discovery. If a host has multiple devices, then there is no guarantee that /dev/sda will point to the same device on next reboot.

My apologies if you know this, but I’ll write for whoever else comes across this… Linux automatically creates symlinks for you which have consistent names and point the appropriate device. Nothing you need to do other than use the correct path. There are multiple ones you can use located:

$ ls -l /dev/disk/

drwxr-xr-x 2 root root 540 Jan 16 16:03 by-id
drwxr-xr-x 2 root root 140 Jan 16 16:03 by-label
drwxr-xr-x 2 root root 200 Jan 16 16:03 by-partuuid
drwxr-xr-x 2 root root 340 Jan 16 16:03 by-path
drwxr-xr-x 2 root root 160 Jan 16 16:03 by-uuid

The by-id is the one I like to use, for me the the device and each partition on /dev/sda has multiple referenced symlinks:

$ ls -l /dev/disk/by-id/ | grep "sda"
lrwxrwxrwx 1 root root  9 Jan 16 16:03 ata-Samsung_SSD_860_EVO_M.2_1TB_S415NS0R506801N -> ../../sda
lrwxrwxrwx 1 root root 10 Jan 16 16:03 ata-Samsung_SSD_860_EVO_M.2_1TB_S415NS0R506801N-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jan 16 16:03 ata-Samsung_SSD_860_EVO_M.2_1TB_S415NS0R506801N-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 Jan 16 16:03 ata-Samsung_SSD_860_EVO_M.2_1TB_S415NS0R506801N-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 16 16:03 ata-Samsung_SSD_860_EVO_M.2_1TB_S415NS0R506801N-part5 -> ../../sda5
lrwxrwxrwx 1 root root  9 Jan 16 16:03 wwn-0x5002538e3150c98a -> ../../sda
lrwxrwxrwx 1 root root 10 Jan 16 16:03 wwn-0x5002538e3150c98a-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jan 16 16:03 wwn-0x5002538e3150c98a-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 Jan 16 16:03 wwn-0x5002538e3150c98a-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 16 16:03 wwn-0x5002538e3150c98a-part5 -> ../../sda5

So I can use /dev/disk/by-id/ata-Samsung_SSD_860_EVO_M.2_1TB_S415NS0R506801N or /dev/disk/by-id/wwn-0x5002538e3150c98a to always reference the same device. Should sda and sdb swap locations no issues, these symlinks generated at boot will point to the correct one.

@cameronraysmith I moved your comment to #14436, if we could follow up there for the fix, thanks for reporting.

I assumed rook is doing so currently because that’s how i interpreted the doc section Storage Selection Settings -> devices …

name: The name of the devices and partitions (e.g., sda). The full udev path can also be specified for devices, partitions, and logical volumes (e.g. /dev/disk/by-id/ata-ST4000DM004-XXXX - this will not change after reboots).

Especially the udev-part … this will not change after reboots

I’ll take a look if that can be done.

ROOK_BLOCK_PATH has to be possible root cause. In case of update scenario, the ROOK_BLOCK_PATH is fetched from the currently running OSD deployment. So even though the /dev/by-id/* is now pointing to, say, /dev/sdb, rook operator will set ROOK_BLOCK_PATH to, say, /dev/sde by reading it from the currently running OSD deployment.

So we might have to change how ROOK_BLOCK_PATH is set in case of Update scenario.

@sethjones our workaround is to delete the deployments and let the operator recreate them with the right device name