rook: nsenter can't find vgcreate on NixOS host during osd preparation
Is this a bug report or feature request?
- Bug Report
Deviation from expected behavior: nsenter can’t find vgcreate on NixOS host during creating osd even though it found lvm binary earlier. Relevant parts of log:
2023-03-31 08:02:35.172428 D | exec: Running command: nsenter --mount=/rootfs/proc/1/ns/mnt -- /sbin/lvm --help
2023-03-31 08:02:35.174921 D | cephosd: failed to call nsenter. failed to execute nsenter. output: nsenter: failed to execute /sbin/lvm: No such file or directory: exit status 127
2023-03-31 08:02:35.174965 D | cephosd: failed to lookup binary path "/rootfs/sbin/lvm" on the host rootfs. stat /rootfs/sbin/lvm: no such file or directory
2023-03-31 08:02:35.174976 D | exec: Running command: nsenter --mount=/rootfs/proc/1/ns/mnt -- /run/current-system/sw/bin/lvm --help
2023-03-31 08:02:35.181346 I | cephosd: successfully called nsenter
2023-03-31 08:02:35.181403 I | cephosd: binary "/run/current-system/sw/bin/lvm" found on the host, proceeding with osd preparation
2023-03-31 08:02:37.144028 D | exec: Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts vgcreate --force --yes ceph-3efbc83a-4e9e-4f4b-8306-ed68a832aea9 /dev/sdc
2023-03-31 08:02:37.144040 D | exec: stderr: nsenter: failed to execute vgcreate: No such file or directory
Expected behavior: osd preparation finishes successfully
How to reproduce it (minimal and precise): Attempt to create a new ceph rook cluster on NixOS hosts.
File(s) to submit:
- Cluster CR (custom resource), typically called
cluster.yaml, if necessary
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
name: rook-ceph
namespace: rook-ceph
spec:
cephVersion:
image: quay.io/ceph/ceph:v17.2.5
dataDirHostPath: /var/lib/rook
mon:
count: 3
allowMultiplePerNode: false
dashboard:
enabled: true
# cluster level storage configuration and selection
storage:
useAllNodes: false
useAllDevices: false
config:
metadataDevice: "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0"
nodes:
- name: "k8s-worker0"
devices:
- name: "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0"
- name: "k8s-worker1"
devices:
- name: "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0"
- name: "k8s-worker2"
devices:
- name: "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0"
Logs to submit:
- Crashing pod log
2023-03-31 08:02:33.921210 I | cephcmd: desired devices to configure osds: [{Name:/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0 OSDsPerDevice:1 MetadataDevice: DatabaseSizeMB:0 DeviceClass: InitialWeight: IsFilter:false IsDevicePathFilter:false}]
2023-03-31 08:02:33.922943 I | rookcmd: starting Rook v1.11.2 with arguments '/rook/rook ceph osd provision'
2023-03-31 08:02:33.923039 I | rookcmd: flag values: --cluster-id=669af55d-a945-4527-b1cd-b7507d0ca305, --cluster-name=rook-ceph, --data-device-filter=, --data-device-path-filter=, --data-devices=[{"id":"/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0","storeConfig":{"osdsPerDevice":1}}], --encrypted-device=false, --force-format=false, --help=false, --location=, --log-level=DEBUG, --metadata-device=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0, --node-name=k8s-worker0, --operator-image=, --osd-crush-device-class=, --osd-crush-initial-weight=, --osd-database-size=0, --osd-wal-size=576, --osds-per-device=1, --pvc-backed-osd=false, --service-account=
2023-03-31 08:02:33.923081 I | op-mon: parsing mon endpoints: c=10.108.165.4:6789,a=10.103.208.212:6789,b=10.98.195.86:6789
2023-03-31 08:02:33.937068 I | op-osd: CRUSH location=root=default host=k8s-worker0
2023-03-31 08:02:33.937103 I | cephcmd: crush location of osd: root=default host=k8s-worker0
2023-03-31 08:02:33.937117 D | exec: Running command: dmsetup version
2023-03-31 08:02:33.941079 I | cephosd: Library version: 1.02.181-RHEL8 (2021-10-20)
Driver version: 4.45.0
2023-03-31 08:02:33.954227 D | cephclient: No ceph configuration override to merge as "rook-config-override" configmap is empty
2023-03-31 08:02:33.954251 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2023-03-31 08:02:33.954360 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2023-03-31 08:02:33.954456 D | cephclient: config file @ /etc/ceph/ceph.conf:
[global]
fsid = 9f59d54c-79e1-4af1-9b60-fa2e6099216c
mon initial members = a b c
mon host = [v2:10.103.208.212:3300,v1:10.103.208.212:6789],[v2:10.98.195.86:3300,v1:10.98.195.86:6789],[v2:10.108.165.4:3300,v1:10.108.165.4:6789]
[client.admin]
keyring = /var/lib/rook/rook-ceph/client.admin.keyring
2023-03-31 08:02:33.954470 I | cephosd: discovering hardware
2023-03-31 08:02:33.954479 D | exec: Running command: lsblk --all --noheadings --list --output KNAME
2023-03-31 08:02:33.959467 D | exec: Running command: lsblk /dev/loop0 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:33.961579 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:33.961600 W | inventory: skipping device "loop0". exit status 32
2023-03-31 08:02:33.961610 D | exec: Running command: lsblk /dev/loop1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:33.963820 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:33.963835 W | inventory: skipping device "loop1". exit status 32
2023-03-31 08:02:33.963842 D | exec: Running command: lsblk /dev/loop2 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:33.966328 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:33.966341 W | inventory: skipping device "loop2". exit status 32
2023-03-31 08:02:33.966347 D | exec: Running command: lsblk /dev/loop3 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:33.968560 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:33.968622 W | inventory: skipping device "loop3". exit status 32
2023-03-31 08:02:33.968648 D | exec: Running command: lsblk /dev/loop4 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:33.970807 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:33.970847 W | inventory: skipping device "loop4". exit status 32
2023-03-31 08:02:33.970869 D | exec: Running command: lsblk /dev/loop5 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:33.972859 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:33.972925 W | inventory: skipping device "loop5". exit status 32
2023-03-31 08:02:33.972960 D | exec: Running command: lsblk /dev/loop6 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:33.975144 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:33.975196 W | inventory: skipping device "loop6". exit status 32
2023-03-31 08:02:33.975215 D | exec: Running command: lsblk /dev/loop7 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:33.976972 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:33.977005 W | inventory: skipping device "loop7". exit status 32
2023-03-31 08:02:33.977019 D | exec: Running command: lsblk /dev/sda --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:33.980905 D | sys: lsblk output: "SIZE=\"32212254720\" ROTA=\"0\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sda\" KNAME=\"/dev/sda\" MOUNTPOINT=\"\" FSTYPE=\"\""
2023-03-31 08:02:33.980977 D | exec: Running command: sgdisk --print /dev/sda
2023-03-31 08:02:33.986171 D | exec: Running command: udevadm info --query=property /dev/sda
2023-03-31 08:02:33.993797 D | sys: udevadm info output: "DEVLINKS=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0 /dev/disk/by-diskseq/1\nDEVNAME=/dev/sda\nDEVPATH=/devices/pci0000:00/0000:00:15.0/0000:03:00.0/host0/target0:0:0/0:0:0:0/block/sda\nDEVTYPE=disk\nDISKSEQ=1\nID_BUS=scsi\nID_MODEL=Virtual_disk\nID_MODEL_ENC=Virtual\\x20disk\\x20\\x20\\x20\\x20\nID_PART_TABLE_TYPE=gpt\nID_PART_TABLE_UUID=ad68627d-d202-4197-8112-ef84e0590042\nID_PATH=pci-0000:03:00.0-scsi-0:0:0:0\nID_PATH_TAG=pci-0000_03_00_0-scsi-0_0_0_0\nID_REVISION=2.0\nID_SCSI=1\nID_TYPE=disk\nID_VENDOR=VMware\nID_VENDOR_ENC=VMware\\x20\\x20\nMAJOR=8\nMINOR=0\nPATH=/nix/store/r70pwkr42m6mi4yqdziw6jccgf12g9mk-udev-path/bin:/nix/store/r70pwkr42m6mi4yqdziw6jccgf12g9mk-udev-path/sbin\nSUBSYSTEM=block\nTAGS=:systemd:\nUSEC_INITIALIZED=800068"
2023-03-31 08:02:33.994547 D | exec: Running command: lsblk --noheadings --path --list --output NAME /dev/sda
2023-03-31 08:02:33.998429 I | inventory: skipping device "sda" because it has child, considering the child instead.
2023-03-31 08:02:33.998522 D | exec: Running command: lsblk /dev/sda1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:34.002761 D | sys: lsblk output: "SIZE=\"535822336\" ROTA=\"0\" RO=\"0\" TYPE=\"part\" PKNAME=\"/dev/sda\" NAME=\"/dev/sda1\" KNAME=\"/dev/sda1\" MOUNTPOINT=\"/rootfs/boot\" FSTYPE=\"vfat\""
2023-03-31 08:02:34.002806 D | exec: Running command: udevadm info --query=property /dev/sda1
2023-03-31 08:02:34.009391 D | sys: udevadm info output: "DEVLINKS=/dev/disk/by-label/ESP /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0-part1 /dev/disk/by-uuid/21A2-6235 /dev/disk/by-partlabel/ESP /dev/disk/by-partuuid/78cce762-7002-4c0e-89e0-9e99d7835eee\nDEVNAME=/dev/sda1\nDEVPATH=/devices/pci0000:00/0000:00:15.0/0000:03:00.0/host0/target0:0:0/0:0:0:0/block/sda/sda1\nDEVTYPE=partition\nDISKSEQ=1\nID_BUS=scsi\nID_FS_LABEL=ESP\nID_FS_LABEL_ENC=ESP\nID_FS_TYPE=vfat\nID_FS_USAGE=filesystem\nID_FS_UUID=21A2-6235\nID_FS_UUID_ENC=21A2-6235\nID_FS_VERSION=FAT32\nID_MODEL=Virtual_disk\nID_MODEL_ENC=Virtual\\x20disk\\x20\\x20\\x20\\x20\nID_PART_ENTRY_DISK=8:0\nID_PART_ENTRY_NAME=ESP\nID_PART_ENTRY_NUMBER=1\nID_PART_ENTRY_OFFSET=2048\nID_PART_ENTRY_SCHEME=gpt\nID_PART_ENTRY_SIZE=1046528\nID_PART_ENTRY_TYPE=c12a7328-f81f-11d2-ba4b-00a0c93ec93b\nID_PART_ENTRY_UUID=78cce762-7002-4c0e-89e0-9e99d7835eee\nID_PART_TABLE_TYPE=gpt\nID_PART_TABLE_UUID=ad68627d-d202-4197-8112-ef84e0590042\nID_PATH=pci-0000:03:00.0-scsi-0:0:0:0\nID_PATH_TAG=pci-0000_03_00_0-scsi-0_0_0_0\nID_REVISION=2.0\nID_SCSI=1\nID_TYPE=disk\nID_VENDOR=VMware\nID_VENDOR_ENC=VMware\\x20\\x20\nMAJOR=8\nMINOR=1\nPARTN=1\nPARTNAME=ESP\nPATH=/nix/store/r70pwkr42m6mi4yqdziw6jccgf12g9mk-udev-path/bin:/nix/store/r70pwkr42m6mi4yqdziw6jccgf12g9mk-udev-path/sbin\nSUBSYSTEM=block\nTAGS=:systemd:\nUSEC_INITIALIZED=851773"
2023-03-31 08:02:34.009429 D | exec: Running command: lsblk /dev/sda2 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:34.012980 D | sys: lsblk output: "SIZE=\"31674335232\" ROTA=\"0\" RO=\"0\" TYPE=\"part\" PKNAME=\"/dev/sda\" NAME=\"/dev/sda2\" KNAME=\"/dev/sda2\" MOUNTPOINT=\"/rootfs\" FSTYPE=\"ext4\""
2023-03-31 08:02:34.013038 D | exec: Running command: udevadm info --query=property /dev/sda2
2023-03-31 08:02:34.017635 D | sys: udevadm info output: "DEVLINKS=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0-part2 /dev/disk/by-label/nixos /dev/disk/by-uuid/c399c955-79d2-4333-b43a-a4c32cca79cc /dev/disk/by-partuuid/653f15ce-7f2a-497d-a98f-32465b8b7b8d /dev/root /dev/disk/by-partlabel/nixos\nDEVNAME=/dev/sda2\nDEVPATH=/devices/pci0000:00/0000:00:15.0/0000:03:00.0/host0/target0:0:0/0:0:0:0/block/sda/sda2\nDEVTYPE=partition\nDISKSEQ=1\nID_BUS=scsi\nID_FS_LABEL=nixos\nID_FS_LABEL_ENC=nixos\nID_FS_TYPE=ext4\nID_FS_USAGE=filesystem\nID_FS_UUID=c399c955-79d2-4333-b43a-a4c32cca79cc\nID_FS_UUID_ENC=c399c955-79d2-4333-b43a-a4c32cca79cc\nID_FS_VERSION=1.0\nID_MODEL=Virtual_disk\nID_MODEL_ENC=Virtual\\x20disk\\x20\\x20\\x20\\x20\nID_PART_ENTRY_DISK=8:0\nID_PART_ENTRY_NAME=nixos\nID_PART_ENTRY_NUMBER=2\nID_PART_ENTRY_OFFSET=1048576\nID_PART_ENTRY_SCHEME=gpt\nID_PART_ENTRY_SIZE=61863936\nID_PART_ENTRY_TYPE=0fc63daf-8483-4772-8e79-3d69d8477de4\nID_PART_ENTRY_UUID=653f15ce-7f2a-497d-a98f-32465b8b7b8d\nID_PART_TABLE_TYPE=gpt\nID_PART_TABLE_UUID=ad68627d-d202-4197-8112-ef84e0590042\nID_PATH=pci-0000:03:00.0-scsi-0:0:0:0\nID_PATH_TAG=pci-0000_03_00_0-scsi-0_0_0_0\nID_REVISION=2.0\nID_SCSI=1\nID_TYPE=disk\nID_VENDOR=VMware\nID_VENDOR_ENC=VMware\\x20\\x20\nMAJOR=8\nMINOR=2\nPARTN=2\nPARTNAME=nixos\nPATH=/nix/store/r70pwkr42m6mi4yqdziw6jccgf12g9mk-udev-path/bin:/nix/store/r70pwkr42m6mi4yqdziw6jccgf12g9mk-udev-path/sbin\nSUBSYSTEM=block\nTAGS=:systemd:\nUSEC_INITIALIZED=861510"
2023-03-31 08:02:34.017716 D | exec: Running command: lsblk /dev/sdb --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:34.021136 D | sys: lsblk output: "SIZE=\"107374182400\" ROTA=\"0\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sdb\" KNAME=\"/dev/sdb\" MOUNTPOINT=\"\" FSTYPE=\"\""
2023-03-31 08:02:34.021218 D | exec: Running command: sgdisk --print /dev/sdb
2023-03-31 08:02:34.023569 D | exec: Running command: udevadm info --query=property /dev/sdb
2023-03-31 08:02:34.029499 D | sys: udevadm info output: "DEVLINKS=/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0 /dev/disk/by-diskseq/2\nDEVNAME=/dev/sdb\nDEVPATH=/devices/pci0000:00/0000:00:15.0/0000:03:00.0/host0/target0:0:1/0:0:1:0/block/sdb\nDEVTYPE=disk\nDISKSEQ=2\nID_BUS=scsi\nID_MODEL=Virtual_disk\nID_MODEL_ENC=Virtual\\x20disk\\x20\\x20\\x20\\x20\nID_PATH=pci-0000:03:00.0-scsi-0:0:1:0\nID_PATH_TAG=pci-0000_03_00_0-scsi-0_0_1_0\nID_REVISION=2.0\nID_SCSI=1\nID_TYPE=disk\nID_VENDOR=VMware\nID_VENDOR_ENC=VMware\\x20\\x20\nMAJOR=8\nMINOR=16\nPATH=/nix/store/r70pwkr42m6mi4yqdziw6jccgf12g9mk-udev-path/bin:/nix/store/r70pwkr42m6mi4yqdziw6jccgf12g9mk-udev-path/sbin\nSUBSYSTEM=block\nTAGS=:systemd:\nUSEC_INITIALIZED=792556"
2023-03-31 08:02:34.029523 D | exec: Running command: lsblk --noheadings --path --list --output NAME /dev/sdb
2023-03-31 08:02:34.031673 D | exec: Running command: lsblk /dev/sdc --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:34.034617 D | sys: lsblk output: "SIZE=\"214748364800\" ROTA=\"1\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sdc\" KNAME=\"/dev/sdc\" MOUNTPOINT=\"\" FSTYPE=\"\""
2023-03-31 08:02:34.034648 D | exec: Running command: sgdisk --print /dev/sdc
2023-03-31 08:02:34.037536 D | exec: Running command: udevadm info --query=property /dev/sdc
2023-03-31 08:02:34.042123 D | sys: udevadm info output: "DEVLINKS=/dev/disk/by-diskseq/3 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0\nDEVNAME=/dev/sdc\nDEVPATH=/devices/pci0000:00/0000:00:15.0/0000:03:00.0/host0/target0:0:2/0:0:2:0/block/sdc\nDEVTYPE=disk\nDISKSEQ=3\nID_BUS=scsi\nID_MODEL=Virtual_disk\nID_MODEL_ENC=Virtual\\x20disk\\x20\\x20\\x20\\x20\nID_PATH=pci-0000:03:00.0-scsi-0:0:2:0\nID_PATH_TAG=pci-0000_03_00_0-scsi-0_0_2_0\nID_REVISION=2.0\nID_SCSI=1\nID_TYPE=disk\nID_VENDOR=VMware\nID_VENDOR_ENC=VMware\\x20\\x20\nMAJOR=8\nMINOR=32\nPATH=/nix/store/r70pwkr42m6mi4yqdziw6jccgf12g9mk-udev-path/bin:/nix/store/r70pwkr42m6mi4yqdziw6jccgf12g9mk-udev-path/sbin\nSUBSYSTEM=block\nTAGS=:systemd:\nUSEC_INITIALIZED=795335"
2023-03-31 08:02:34.042168 D | exec: Running command: lsblk --noheadings --path --list --output NAME /dev/sdc
2023-03-31 08:02:34.043936 D | exec: Running command: lsblk /dev/nbd0 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:34.045574 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:34.045598 W | inventory: skipping device "nbd0". exit status 32
2023-03-31 08:02:34.045610 D | exec: Running command: lsblk /dev/nbd1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:34.047447 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:34.047465 W | inventory: skipping device "nbd1". exit status 32
2023-03-31 08:02:34.047473 D | exec: Running command: lsblk /dev/nbd2 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:34.052661 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:34.052685 W | inventory: skipping device "nbd2". exit status 32
2023-03-31 08:02:34.052697 D | exec: Running command: lsblk /dev/nbd3 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:34.054555 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:34.054573 W | inventory: skipping device "nbd3". exit status 32
2023-03-31 08:02:34.054584 D | exec: Running command: lsblk /dev/nbd4 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:34.057429 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:34.057448 W | inventory: skipping device "nbd4". exit status 32
2023-03-31 08:02:34.057456 D | exec: Running command: lsblk /dev/nbd5 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:34.059525 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:34.059544 W | inventory: skipping device "nbd5". exit status 32
2023-03-31 08:02:34.059553 D | exec: Running command: lsblk /dev/nbd6 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:34.062080 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:34.062095 W | inventory: skipping device "nbd6". exit status 32
2023-03-31 08:02:34.062102 D | exec: Running command: lsblk /dev/nbd7 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:34.064070 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:34.064085 W | inventory: skipping device "nbd7". exit status 32
2023-03-31 08:02:34.064091 D | exec: Running command: lsblk /dev/nbd8 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:34.066043 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:34.066056 W | inventory: skipping device "nbd8". exit status 32
2023-03-31 08:02:34.066062 D | exec: Running command: lsblk /dev/nbd9 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:34.067663 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:34.067676 W | inventory: skipping device "nbd9". exit status 32
2023-03-31 08:02:34.067682 D | exec: Running command: lsblk /dev/nbd10 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:34.069633 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:34.069645 W | inventory: skipping device "nbd10". exit status 32
2023-03-31 08:02:34.069651 D | exec: Running command: lsblk /dev/nbd11 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:34.071153 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:34.071179 W | inventory: skipping device "nbd11". exit status 32
2023-03-31 08:02:34.071187 D | exec: Running command: lsblk /dev/nbd12 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:34.073354 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:34.073371 W | inventory: skipping device "nbd12". exit status 32
2023-03-31 08:02:34.073379 D | exec: Running command: lsblk /dev/nbd13 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:34.074925 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:34.074943 W | inventory: skipping device "nbd13". exit status 32
2023-03-31 08:02:34.074949 D | exec: Running command: lsblk /dev/nbd14 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:34.077063 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:34.077082 W | inventory: skipping device "nbd14". exit status 32
2023-03-31 08:02:34.077093 D | exec: Running command: lsblk /dev/nbd15 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:34.078690 E | sys: failed to execute lsblk. output: .
2023-03-31 08:02:34.078707 W | inventory: skipping device "nbd15". exit status 32
2023-03-31 08:02:34.078710 D | inventory: discovered disks are:
2023-03-31 08:02:34.078746 D | inventory: &{Name:sda1 Parent:sda HasChildren:false DevLinks:/dev/disk/by-label/ESP /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0-part1 /dev/disk/by-uuid/21A2-6235 /dev/disk/by-partlabel/ESP /dev/disk/by-partuuid/78cce762-7002-4c0e-89e0-9e99d7835eee Size:535822336 UUID: Serial: Type:part Rotational:false Readonly:false Partitions:[] Filesystem:vfat Mountpoint:boot Vendor:VMware Model:Virtual_disk WWN: WWNVendorExtension: Empty:false CephVolumeData: RealPath:/dev/sda1 KernelName:sda1 Encrypted:false}
2023-03-31 08:02:34.078761 D | inventory: &{Name:sda2 Parent:sda HasChildren:false DevLinks:/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0-part2 /dev/disk/by-label/nixos /dev/disk/by-uuid/c399c955-79d2-4333-b43a-a4c32cca79cc /dev/disk/by-partuuid/653f15ce-7f2a-497d-a98f-32465b8b7b8d /dev/root /dev/disk/by-partlabel/nixos Size:31674335232 UUID: Serial: Type:part Rotational:false Readonly:false Partitions:[] Filesystem:ext4 Mountpoint:rootfs Vendor:VMware Model:Virtual_disk WWN: WWNVendorExtension: Empty:false CephVolumeData: RealPath:/dev/sda2 KernelName:sda2 Encrypted:false}
2023-03-31 08:02:34.078771 D | inventory: &{Name:sdb Parent: HasChildren:false DevLinks:/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0 /dev/disk/by-diskseq/2 Size:107374182400 UUID:b43d6ab6-230c-46d7-9d1d-2a83de48e2d1 Serial: Type:disk Rotational:false Readonly:false Partitions:[] Filesystem: Mountpoint: Vendor:VMware Model:Virtual_disk WWN: WWNVendorExtension: Empty:false CephVolumeData: RealPath:/dev/sdb KernelName:sdb Encrypted:false}
2023-03-31 08:02:34.078781 D | inventory: &{Name:sdc Parent: HasChildren:false DevLinks:/dev/disk/by-diskseq/3 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0 Size:214748364800 UUID:3dc7d3e7-a2b6-4db3-9573-db30a26a177c Serial: Type:disk Rotational:true Readonly:false Partitions:[] Filesystem: Mountpoint: Vendor:VMware Model:Virtual_disk WWN: WWNVendorExtension: Empty:false CephVolumeData: RealPath:/dev/sdc KernelName:sdc Encrypted:false}
2023-03-31 08:02:34.078784 I | cephosd: creating and starting the osds
2023-03-31 08:02:34.078795 D | cephosd: desiredDevices are [{Name:/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0 OSDsPerDevice:1 MetadataDevice: DatabaseSizeMB:0 DeviceClass: InitialWeight: IsFilter:false IsDevicePathFilter:false}]
2023-03-31 08:02:34.078798 D | cephosd: context.Devices are:
2023-03-31 08:02:34.078807 D | cephosd: &{Name:sda1 Parent:sda HasChildren:false DevLinks:/dev/disk/by-label/ESP /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0-part1 /dev/disk/by-uuid/21A2-6235 /dev/disk/by-partlabel/ESP /dev/disk/by-partuuid/78cce762-7002-4c0e-89e0-9e99d7835eee Size:535822336 UUID: Serial: Type:part Rotational:false Readonly:false Partitions:[] Filesystem:vfat Mountpoint:boot Vendor:VMware Model:Virtual_disk WWN: WWNVendorExtension: Empty:false CephVolumeData: RealPath:/dev/sda1 KernelName:sda1 Encrypted:false}
2023-03-31 08:02:34.078817 D | cephosd: &{Name:sda2 Parent:sda HasChildren:false DevLinks:/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0-part2 /dev/disk/by-label/nixos /dev/disk/by-uuid/c399c955-79d2-4333-b43a-a4c32cca79cc /dev/disk/by-partuuid/653f15ce-7f2a-497d-a98f-32465b8b7b8d /dev/root /dev/disk/by-partlabel/nixos Size:31674335232 UUID: Serial: Type:part Rotational:false Readonly:false Partitions:[] Filesystem:ext4 Mountpoint:rootfs Vendor:VMware Model:Virtual_disk WWN: WWNVendorExtension: Empty:false CephVolumeData: RealPath:/dev/sda2 KernelName:sda2 Encrypted:false}
2023-03-31 08:02:34.078826 D | cephosd: &{Name:sdb Parent: HasChildren:false DevLinks:/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0 /dev/disk/by-diskseq/2 Size:107374182400 UUID:b43d6ab6-230c-46d7-9d1d-2a83de48e2d1 Serial: Type:disk Rotational:false Readonly:false Partitions:[] Filesystem: Mountpoint: Vendor:VMware Model:Virtual_disk WWN: WWNVendorExtension: Empty:false CephVolumeData: RealPath:/dev/sdb KernelName:sdb Encrypted:false}
2023-03-31 08:02:34.078835 D | cephosd: &{Name:sdc Parent: HasChildren:false DevLinks:/dev/disk/by-diskseq/3 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0 Size:214748364800 UUID:3dc7d3e7-a2b6-4db3-9573-db30a26a177c Serial: Type:disk Rotational:true Readonly:false Partitions:[] Filesystem: Mountpoint: Vendor:VMware Model:Virtual_disk WWN: WWNVendorExtension: Empty:false CephVolumeData: RealPath:/dev/sdc KernelName:sdc Encrypted:false}
2023-03-31 08:02:34.078839 I | cephosd: skipping device "sda1" with mountpoint "boot"
2023-03-31 08:02:34.078842 I | cephosd: skipping device "sda2" with mountpoint "rootfs"
2023-03-31 08:02:34.078846 I | cephosd: old lsblk can't detect bluestore signature, so try to detect here
2023-03-31 08:02:34.079019 D | exec: Running command: lsblk /dev/sdb --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:34.082629 D | sys: lsblk output: "SIZE=\"107374182400\" ROTA=\"0\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sdb\" KNAME=\"/dev/sdb\" MOUNTPOINT=\"\" FSTYPE=\"\""
2023-03-31 08:02:34.082660 D | exec: Running command: ceph-volume inventory --format json /dev/sdb
2023-03-31 08:02:34.407353 I | cephosd: device "sdb" is available.
2023-03-31 08:02:34.407398 I | cephosd: skipping device "sdb" that does not match the device filter/list ([{/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0 1 0 false false}]). <nil>
2023-03-31 08:02:34.407406 I | cephosd: old lsblk can't detect bluestore signature, so try to detect here
2023-03-31 08:02:34.407644 D | exec: Running command: lsblk /dev/sdc --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-03-31 08:02:34.411168 D | sys: lsblk output: "SIZE=\"214748364800\" ROTA=\"1\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sdc\" KNAME=\"/dev/sdc\" MOUNTPOINT=\"\" FSTYPE=\"\""
2023-03-31 08:02:34.411206 D | exec: Running command: ceph-volume inventory --format json /dev/sdc
2023-03-31 08:02:34.733547 I | cephosd: device "sdc" is available.
2023-03-31 08:02:34.733574 I | cephosd: "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0" found in the desired devices (matched by link: "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0")
2023-03-31 08:02:34.733580 I | cephosd: device "sdc" is selected by the device filter/name "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0"
2023-03-31 08:02:34.742052 I | cephosd: configuring osd devices: {"Entries":{"sdc":{"Data":-1,"Metadata":null,"Config":{"Name":"/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0","OSDsPerDevice":1,"MetadataDevice":"","DatabaseSizeMB":0,"DeviceClass":"hdd","InitialWeight":"","IsFilter":false,"IsDevicePathFilter":false},"PersistentDevicePaths":["/dev/disk/by-diskseq/3","/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0"],"DeviceInfo":{"name":"sdc","parent":"","hasChildren":false,"devLinks":"/dev/disk/by-diskseq/3 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0","size":214748364800,"uuid":"3dc7d3e7-a2b6-4db3-9573-db30a26a177c","serial":"","type":"disk","rotational":true,"readOnly":false,"Partitions":null,"filesystem":"","mountpoint":"","vendor":"VMware","model":"Virtual_disk","wwn":"","wwnVendorExtension":"","empty":false,"real-path":"/dev/sdc","kernel-name":"sdc"}}}}
2023-03-31 08:02:34.742115 I | cephclient: getting or creating ceph auth key "client.bootstrap-osd"
2023-03-31 08:02:34.742132 D | exec: Running command: ceph auth get-or-create-key client.bootstrap-osd mon allow profile bootstrap-osd --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --name=client.admin --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json
2023-03-31 08:02:35.170692 D | cephosd: won't use raw mode since there is a metadata device "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0"
2023-03-31 08:02:35.170730 D | exec: Running command: nsenter --mount=/rootfs/proc/1/ns/mnt -- /usr/sbin/lvm --help
2023-03-31 08:02:35.172374 D | cephosd: failed to call nsenter. failed to execute nsenter. output: nsenter: failed to execute /usr/sbin/lvm: No such file or directory: exit status 127
2023-03-31 08:02:35.172417 D | cephosd: failed to lookup binary path "/rootfs/usr/sbin/lvm" on the host rootfs. stat /rootfs/usr/sbin/lvm: no such file or directory
2023-03-31 08:02:35.172428 D | exec: Running command: nsenter --mount=/rootfs/proc/1/ns/mnt -- /sbin/lvm --help
2023-03-31 08:02:35.174921 D | cephosd: failed to call nsenter. failed to execute nsenter. output: nsenter: failed to execute /sbin/lvm: No such file or directory: exit status 127
2023-03-31 08:02:35.174965 D | cephosd: failed to lookup binary path "/rootfs/sbin/lvm" on the host rootfs. stat /rootfs/sbin/lvm: no such file or directory
2023-03-31 08:02:35.174976 D | exec: Running command: nsenter --mount=/rootfs/proc/1/ns/mnt -- /run/current-system/sw/bin/lvm --help
2023-03-31 08:02:35.181346 I | cephosd: successfully called nsenter
2023-03-31 08:02:35.181403 I | cephosd: binary "/run/current-system/sw/bin/lvm" found on the host, proceeding with osd preparation
2023-03-31 08:02:35.185077 I | cephosd: Successfully updated lvm config file "/etc/lvm/lvm.conf"
2023-03-31 08:02:35.185269 I | cephosd: configuring new LVM device sdc
2023-03-31 08:02:35.185319 I | cephosd: "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0" found in the desired devices (matched by link: "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0")
2023-03-31 08:02:35.185345 I | cephosd: using sdb as metadataDevice for device /dev/sdc and let ceph-volume lvm batch decide how to create volumes
2023-03-31 08:02:35.185382 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log lvm batch --prepare --bluestore --yes --osds-per-device 1 --crush-device-class hdd /dev/sdc --db-devices /dev/sdb --report
2023-03-31 08:02:35.529659 D | exec:
2023-03-31 08:02:35.529703 D | exec: Total OSDs: 1
2023-03-31 08:02:35.529707 D | exec:
2023-03-31 08:02:35.529712 D | exec: Type Path LV Size % of device
2023-03-31 08:02:35.529714 D | exec: ----------------------------------------------------------------------------------------------------
2023-03-31 08:02:35.529717 D | exec: data /dev/sdc 200.00 GB 100.00%
2023-03-31 08:02:35.529719 D | exec: block_db /dev/sdb 100.00 GB 100.00%
2023-03-31 08:02:35.529725 D | exec: --> passed data devices: 1 physical, 0 LVM
2023-03-31 08:02:35.529732 D | exec: --> relative data size: 1.0
2023-03-31 08:02:35.529735 D | exec: --> passed block_db devices: 1 physical, 0 LVM
2023-03-31 08:02:35.562139 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log lvm batch --prepare --bluestore --yes --osds-per-device 1 --crush-device-class hdd /dev/sdc --db-devices /dev/sdb --report --format json
2023-03-31 08:02:35.931988 D | cephosd: ceph-volume reports: [{"data": "/dev/sdc", "data_size": "200.00 GB", "encryption": "None", "block_db": "/dev/sdb", "block_db_size": "100.00 GB"}]
2023-03-31 08:02:35.932131 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log lvm batch --prepare --bluestore --yes --osds-per-device 1 --crush-device-class hdd /dev/sdc --db-devices /dev/sdb
2023-03-31 08:02:37.143891 D | exec: --> passed data devices: 1 physical, 0 LVM
2023-03-31 08:02:37.143991 D | exec: --> relative data size: 1.0
2023-03-31 08:02:37.143999 D | exec: --> passed block_db devices: 1 physical, 0 LVM
2023-03-31 08:02:37.144006 D | exec: Running command: /usr/bin/ceph-authtool --gen-print-key
2023-03-31 08:02:37.144015 D | exec: Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8253c683-1f00-45fa-b028-6e5ae11f339b
2023-03-31 08:02:37.144028 D | exec: Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts vgcreate --force --yes ceph-3efbc83a-4e9e-4f4b-8306-ed68a832aea9 /dev/sdc
2023-03-31 08:02:37.144040 D | exec: stderr: nsenter: failed to execute vgcreate: No such file or directory
2023-03-31 08:02:37.144046 D | exec: --> Was unable to complete a new OSD, will rollback changes
2023-03-31 08:02:37.144051 D | exec: Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it
2023-03-31 08:02:37.144056 D | exec: stderr: purged osd.0
2023-03-31 08:02:37.146537 D | exec: Traceback (most recent call last):
2023-03-31 08:02:37.146578 D | exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare
2023-03-31 08:02:37.146586 D | exec: self.prepare()
2023-03-31 08:02:37.146592 D | exec: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
2023-03-31 08:02:37.146597 D | exec: return func(*a, **kw)
2023-03-31 08:02:37.146602 D | exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 363, in prepare
2023-03-31 08:02:37.146608 D | exec: block_lv = self.prepare_data_device('block', osd_fsid)
2023-03-31 08:02:37.146612 D | exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 221, in prepare_data_device
2023-03-31 08:02:37.146617 D | exec: **kwargs)
2023-03-31 08:02:37.146622 D | exec: File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 978, in create_lv
2023-03-31 08:02:37.146627 D | exec: vg = create_vg(device, name_prefix='ceph')
2023-03-31 08:02:37.146632 D | exec: File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 663, in create_vg
2023-03-31 08:02:37.146637 D | exec: run_on_host=True
2023-03-31 08:02:37.146642 D | exec: File "/usr/lib/python3.6/site-packages/ceph_volume/process.py", line 147, in run
2023-03-31 08:02:37.146646 D | exec: raise RuntimeError(msg)
2023-03-31 08:02:37.146652 D | exec: RuntimeError: command returned non-zero exit status: 127
2023-03-31 08:02:37.146657 D | exec:
2023-03-31 08:02:37.146662 D | exec: During handling of the above exception, another exception occurred:
2023-03-31 08:02:37.146670 D | exec:
2023-03-31 08:02:37.146678 D | exec: Traceback (most recent call last):
2023-03-31 08:02:37.146684 D | exec: File "/usr/sbin/ceph-volume", line 11, in <module>
2023-03-31 08:02:37.146693 D | exec: load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
2023-03-31 08:02:37.146700 D | exec: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__
2023-03-31 08:02:37.146708 D | exec: self.main(self.argv)
2023-03-31 08:02:37.146716 D | exec: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
2023-03-31 08:02:37.146723 D | exec: return f(*a, **kw)
2023-03-31 08:02:37.146730 D | exec: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main
2023-03-31 08:02:37.146738 D | exec: terminal.dispatch(self.mapper, subcommand_args)
2023-03-31 08:02:37.146745 D | exec: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
2023-03-31 08:02:37.146753 D | exec: instance.main()
2023-03-31 08:02:37.146760 D | exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main
2023-03-31 08:02:37.146769 D | exec: terminal.dispatch(self.mapper, self.argv)
2023-03-31 08:02:37.146776 D | exec: File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
2023-03-31 08:02:37.146785 D | exec: instance.main()
2023-03-31 08:02:37.146792 D | exec: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
2023-03-31 08:02:37.146800 D | exec: return func(*a, **kw)
2023-03-31 08:02:37.146805 D | exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 441, in main
2023-03-31 08:02:37.146810 D | exec: self._execute(plan)
2023-03-31 08:02:37.146815 D | exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 457, in _execute
2023-03-31 08:02:37.146819 D | exec: p.safe_prepare(argparse.Namespace(**args))
2023-03-31 08:02:37.146824 D | exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 256, in safe_prepare
2023-03-31 08:02:37.146831 D | exec: rollback_osd(self.args, self.osd_id)
2023-03-31 08:02:37.146839 D | exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 35, in rollback_osd
2023-03-31 08:02:37.146846 D | exec: Zap(['--destroy', '--osd-id', osd_id]).main()
2023-03-31 08:02:37.146853 D | exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main
2023-03-31 08:02:37.146861 D | exec: self.zap_osd()
2023-03-31 08:02:37.146868 D | exec: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
2023-03-31 08:02:37.146876 D | exec: return func(*a, **kw)
2023-03-31 08:02:37.146881 D | exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 301, in zap_osd
2023-03-31 08:02:37.146885 D | exec: devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid)
2023-03-31 08:02:37.146895 D | exec: File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices
2023-03-31 08:02:37.146901 D | exec: '%s' % osd_id or osd_fsid)
2023-03-31 08:02:37.146905 D | exec: RuntimeError: Unable to find any LV for zapping OSD: 0
2023-03-31 08:02:37.188525 C | rookcmd: failed to configure devices: failed to initialize osd: failed ceph-volume: exit status 1
Cluster Status to submit:
-
Output of krew commands, if necessary
To get the health of the cluster, use
kubectl rook-ceph healthTo get the status of the cluster, usekubectl rook-ceph ceph statusFor more details, see the Rook Krew Plugin
Environment:
- OS (e.g. from /etc/os-release):
NixOS 22.11 (Raccoon) - Kernel (e.g.
uname -a):Linux k8s-master0 5.15.102 #1-NixOS SMP Mon Mar 13 09:20:37 UTC 2023 x86_64 GNU/Linux - Cloud provider or hardware configuration: VMWare virtual machine
- Rook version (use
rook versioninside of a Rook Pod): 1.11 - Storage backend version (e.g. for ceph do
ceph -v): - Kubernetes version (use
kubectl version): Server Version: v1.25.8 - Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): self hosted
- Storage backend status (e.g. for Ceph use
ceph healthin the Rook Ceph toolbox):
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 17 (3 by maintainers)
FWIW here is how I got past the errors described in this issue to get NixOS host + rook-ceph + encrypted OSDs working:
csiRBDPlugin*values, see the the rook-ceph prereqs page.PATHenv var in the OSD pods, using the following configmap: