rook: Adding another OSD to an lvm metadata device fails

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior: OSD preparation job fails, metadataDevice is not found. Expected behavior: Shrink old db lvms and create new ones for new osds. How to reproduce it (minimal and precise):

Use a metadataDevice for a couple osds then try adding another. File(s) to submit:

  • Cluster CR (custom resource), typically called cluster.yaml, if necessary … storage: config: metadataDevice: /dev/nvme3n1 flappingRestartIntervalHours: 0 store: {} useAllDevices: true useAllNodes: true waitTimeoutForHealthyOSDInMinutes: 10

Logs to submit:

  • Crashing pod(s) logs (prepare osds) 2024-01-26 18:18:20.592522 D | sys: lsblk output: “SIZE="3840755982336" ROTA="0" RO="0" TYPE="disk" PKNAME="" NAME="/dev/nvme3n1" KNAME="/dev/nvme3n1" MOUNTPOINT="" FSTYPE="LVM2_member"” 2024-01-26T18:18:20.592567853Z 2024-01-26 18:18:20.592536 D | exec: Running command: sgdisk --print /dev/nvme3n1 2024-01-26T18:18:20.594129868Z 2024-01-26 18:18:20.594111 D | exec: Running command: udevadm info --query=property /dev/nvme3n1 2024-01-26 18:18:20.596822 D | sys: udevadm info output: “DEVLINKS=/dev/disk/by-id/lvm-pv-uuid-IdXk2O-et0v-xPzX-N3Ek-OTf5-OGch-68J1EW /dev/disk/by-path/pci-0000:84:00.0-nvme-1 /dev/disk/by-id/nvme-Micron_7450_MTFDKCC3T8TFR_22293A1C4543 /dev/disk/by-id/nvme-eui.000000000000000100a075223a1c4543 /dev/disk/by-id/nvme-Micron_7450_MTFDKCC3T8TFR_22293A1C4543_1\nDEVNAME=/dev/nvme3n1\nDEVPATH=/devices/pci0000:80/0000:80:03.3/0000:84:00.0/nvme/nvme3/nvme3n1\nDEVTYPE=disk\nDISKSEQ=9\nDM_MULTIPATH_DEVICE_PATH=0\nID_FS_TYPE=LVM2_member\nID_FS_USAGE=raid\nID_FS_UUID=IdXk2O-et0v-xPzX-N3Ek-OTf5-OGch-68J1EW\nID_FS_UUID_ENC=IdXk2O-et0v-xPzX-N3Ek-OTf5-OGch-68J1EW\nID_FS_VERSION=LVM2 001\nID_MODEL=Micron_7450_MTFDKCC3T8TFR\nID_NSID=1\nID_PATH=pci-0000:84:00.0-nvme-1\nID_PATH_TAG=pci-0000_84_00_0-nvme-1\nID_REVISION=E2MU200\nID_SERIAL=Micron_7450_MTFDKCC3T8TFR_22293A1C4543_1\nID_SERIAL_SHORT=22293A1C4543\nID_WWN=eui.000000000000000100a075223a1c4543\nMAJOR=259\nMINOR=0\nMPATH_SBIN_PATH=/sbin\nSUBSYSTEM=block\nSYSTEMD_ALIAS=/dev/block/259:0\nSYSTEMD_READY=1\nSYSTEMD_WANTS=lvm2-pvscan@259:0.service\nTAGS=:systemd:\nUSEC_INITIALIZED=3928562” 2024-01-26 18:18:20.596834 D | exec: Running command: lsblk --noheadings --path --list --output NAME /dev/nvme3n1 2024-01-26 18:18:20.597992 I | inventory: skipping device “nvme3n1” because it has child, considering the child instead. … 2024-01-26 18:18:21.792330 I | cephosd: binary “/usr/sbin/lvm” found on the host, proceeding with osd preparation 2024-01-26 18:18:21.793933 I | cephosd: Successfully updated lvm config file “/etc/lvm/lvm.conf” 2024-01-26 18:18:21.794179 I | cephosd: configuring new LVM device sdf 2024-01-26T18:18:21.798654468Z 2024-01-26 18:18:21.798527 C | rookcmd: failed to configure devices: failed to initialize osd: metadata device /dev/nvme3n1 is not found

    To get logs, use kubectl -n <namespace> logs <pod name> When pasting logs, always surround them with backticks or use the insert code button from the Github UI. Read GitHub documentation if you need help.

I have done both metadataDevice: nvme3n1 and /dev/nvme3n1 lsblk shows the device on all nodes

Cluster Status to submit:

  • Output of kubectl commands, if necessary

    To get the health of the cluster, use kubectl rook-ceph health To get the status of the cluster, use kubectl rook-ceph ceph status For more details, see the Rook kubectl Plugin

Environment:

  • OS (e.g. from /etc/os-release): Ubuntu 22.04.3 LTS
  • Kernel (e.g. uname -a): Linux dionysus 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
  • Cloud provider or hardware configuration: 3 x86_64 nodes
  • Rook version (use rook version inside of a Rook Pod): 1.13.3
  • Storage backend version (e.g. for ceph do ceph -v): 18.2.1
  • Kubernetes version (use kubectl version): Client Version: version.Info{Major:“1”, Minor:“26”, GitVersion:“v1.26.10+rke2r1”, GitCommit:“b8609d4dd75c5d6fba4a5eaa63a5507cb39a6e99”, GitTreeState:“clean”, BuildDate:“2023-10-18T16:30:16Z”, GoVersion:“go1.20.10 X:boringcrypto”, Compiler:“gc”, Platform:“linux/amd64”} Kustomize Version: v4.5.7 Server Version: version.Info{Major:“1”, Minor:“26”, GitVersion:“v1.26.10+rke2r1”, GitCommit:“b8609d4dd75c5d6fba4a5eaa63a5507cb39a6e99”, GitTreeState:“clean”, BuildDate:“2023-10-18T16:30:16Z”, GoVersion:“go1.20.10 X:boringcrypto”, Compiler:“gc”, Platform:“linux/amd64”}
  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): RKE2
  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox): healthy

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (9 by maintainers)

Most upvoted comments

you’ll want a new issue for this @eyitayoafolabi my issue was attempting to add drives to a metadata device that is already in use. there is no resolution for this issue barring recreating all the osds used by the metadata device and instantiating them all at the same time, or the only other option is using no metadata device and storing metadata on the new device itself.

slots seem like a good workaround! Thanks guits! @sp98 can we configure that from the cluster crd?

Reopened. I’ll take a look if we can configure this from the Ceph CRD.

@sp98 when deploying OSDs with lvm batch method, if you pass a db device, ceph-volume will by default basically use all the available space on it and create as many LVs as you pass block devices. If later you want to add a new OSD, then the existing db device won’t be available as there’s no more free space on it.