rook: Adding another OSD to an lvm metadata device fails
Is this a bug report or feature request?
- Bug Report
Deviation from expected behavior: OSD preparation job fails, metadataDevice is not found. Expected behavior: Shrink old db lvms and create new ones for new osds. How to reproduce it (minimal and precise):
Use a metadataDevice for a couple osds then try adding another. File(s) to submit:
- Cluster CR (custom resource), typically called
cluster.yaml, if necessary … storage: config: metadataDevice: /dev/nvme3n1 flappingRestartIntervalHours: 0 store: {} useAllDevices: true useAllNodes: true waitTimeoutForHealthyOSDInMinutes: 10
Logs to submit:
-
Crashing pod(s) logs (prepare osds) 2024-01-26 18:18:20.592522 D | sys: lsblk output: “SIZE="3840755982336" ROTA="0" RO="0" TYPE="disk" PKNAME="" NAME="/dev/nvme3n1" KNAME="/dev/nvme3n1" MOUNTPOINT="" FSTYPE="LVM2_member"” 2024-01-26T18:18:20.592567853Z 2024-01-26 18:18:20.592536 D | exec: Running command: sgdisk --print /dev/nvme3n1 2024-01-26T18:18:20.594129868Z 2024-01-26 18:18:20.594111 D | exec: Running command: udevadm info --query=property /dev/nvme3n1 2024-01-26 18:18:20.596822 D | sys: udevadm info output: “DEVLINKS=/dev/disk/by-id/lvm-pv-uuid-IdXk2O-et0v-xPzX-N3Ek-OTf5-OGch-68J1EW /dev/disk/by-path/pci-0000:84:00.0-nvme-1 /dev/disk/by-id/nvme-Micron_7450_MTFDKCC3T8TFR_22293A1C4543 /dev/disk/by-id/nvme-eui.000000000000000100a075223a1c4543 /dev/disk/by-id/nvme-Micron_7450_MTFDKCC3T8TFR_22293A1C4543_1\nDEVNAME=/dev/nvme3n1\nDEVPATH=/devices/pci0000:80/0000:80:03.3/0000:84:00.0/nvme/nvme3/nvme3n1\nDEVTYPE=disk\nDISKSEQ=9\nDM_MULTIPATH_DEVICE_PATH=0\nID_FS_TYPE=LVM2_member\nID_FS_USAGE=raid\nID_FS_UUID=IdXk2O-et0v-xPzX-N3Ek-OTf5-OGch-68J1EW\nID_FS_UUID_ENC=IdXk2O-et0v-xPzX-N3Ek-OTf5-OGch-68J1EW\nID_FS_VERSION=LVM2 001\nID_MODEL=Micron_7450_MTFDKCC3T8TFR\nID_NSID=1\nID_PATH=pci-0000:84:00.0-nvme-1\nID_PATH_TAG=pci-0000_84_00_0-nvme-1\nID_REVISION=E2MU200\nID_SERIAL=Micron_7450_MTFDKCC3T8TFR_22293A1C4543_1\nID_SERIAL_SHORT=22293A1C4543\nID_WWN=eui.000000000000000100a075223a1c4543\nMAJOR=259\nMINOR=0\nMPATH_SBIN_PATH=/sbin\nSUBSYSTEM=block\nSYSTEMD_ALIAS=/dev/block/259:0\nSYSTEMD_READY=1\nSYSTEMD_WANTS=lvm2-pvscan@259:0.service\nTAGS=:systemd:\nUSEC_INITIALIZED=3928562” 2024-01-26 18:18:20.596834 D | exec: Running command: lsblk --noheadings --path --list --output NAME /dev/nvme3n1 2024-01-26 18:18:20.597992 I | inventory: skipping device “nvme3n1” because it has child, considering the child instead. … 2024-01-26 18:18:21.792330 I | cephosd: binary “/usr/sbin/lvm” found on the host, proceeding with osd preparation 2024-01-26 18:18:21.793933 I | cephosd: Successfully updated lvm config file “/etc/lvm/lvm.conf” 2024-01-26 18:18:21.794179 I | cephosd: configuring new LVM device sdf 2024-01-26T18:18:21.798654468Z 2024-01-26 18:18:21.798527 C | rookcmd: failed to configure devices: failed to initialize osd: metadata device /dev/nvme3n1 is not found
To get logs, use
kubectl -n <namespace> logs <pod name>When pasting logs, always surround them with backticks or use theinsert codebutton from the Github UI. Read GitHub documentation if you need help.
I have done both metadataDevice: nvme3n1 and /dev/nvme3n1 lsblk shows the device on all nodes
Cluster Status to submit:
-
Output of kubectl commands, if necessary
To get the health of the cluster, use
kubectl rook-ceph healthTo get the status of the cluster, usekubectl rook-ceph ceph statusFor more details, see the Rook kubectl Plugin
Environment:
- OS (e.g. from /etc/os-release): Ubuntu 22.04.3 LTS
- Kernel (e.g.
uname -a): Linux dionysus 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux - Cloud provider or hardware configuration: 3 x86_64 nodes
- Rook version (use
rook versioninside of a Rook Pod): 1.13.3 - Storage backend version (e.g. for ceph do
ceph -v): 18.2.1 - Kubernetes version (use
kubectl version): Client Version: version.Info{Major:“1”, Minor:“26”, GitVersion:“v1.26.10+rke2r1”, GitCommit:“b8609d4dd75c5d6fba4a5eaa63a5507cb39a6e99”, GitTreeState:“clean”, BuildDate:“2023-10-18T16:30:16Z”, GoVersion:“go1.20.10 X:boringcrypto”, Compiler:“gc”, Platform:“linux/amd64”} Kustomize Version: v4.5.7 Server Version: version.Info{Major:“1”, Minor:“26”, GitVersion:“v1.26.10+rke2r1”, GitCommit:“b8609d4dd75c5d6fba4a5eaa63a5507cb39a6e99”, GitTreeState:“clean”, BuildDate:“2023-10-18T16:30:16Z”, GoVersion:“go1.20.10 X:boringcrypto”, Compiler:“gc”, Platform:“linux/amd64”} - Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): RKE2
- Storage backend status (e.g. for Ceph use
ceph healthin the Rook Ceph toolbox): healthy
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15 (9 by maintainers)
you’ll want a new issue for this @eyitayoafolabi my issue was attempting to add drives to a metadata device that is already in use. there is no resolution for this issue barring recreating all the osds used by the metadata device and instantiating them all at the same time, or the only other option is using no metadata device and storing metadata on the new device itself.
Reopened. I’ll take a look if we can configure this from the Ceph CRD.
@sp98 when deploying OSDs with
lvm batchmethod, if you pass a db device, ceph-volume will by default basically use all the available space on it and create as many LVs as you pass block devices. If later you want to add a new OSD, then the existing db device won’t be available as there’s no more free space on it.