rook: Half of my disks are detected when OSDs are provisioned, the other half are skipped due to insufficient space
Is this a bug report or feature request?
- Bug Report
Deviation from expected behavior: The provisioner skips these disks.
Expected behavior: I would expect the OSD provisioner to detect these disks and add to the ceph cluster.
How to reproduce it (minimal and precise):
File(s) to submit:
I am using a non-modified version of ./deploy/examples/cluster.yaml from this repository.
Logs to submit:
Disks: (Disks sdf-sdk should be picked up my ceph as per my current understanding, but no action is taken upon them. 5 of these disks are 16TBB HDDs, and one is a 4TB SSD.
webdog@terranova:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 4K 1 loop /snap/bare/5
loop1 7:1 0 63.2M 1 loop /snap/core20/1738
loop2 7:2 0 91.7M 1 loop /snap/gtk-common-themes/1535
loop3 7:3 0 346.3M 1 loop /snap/gnome-3-38-2004/119
loop4 7:4 0 49.6M 1 loop /snap/snapd/17883
loop5 7:5 0 346.3M 1 loop /snap/gnome-3-38-2004/115
loop6 7:6 0 46M 1 loop /snap/snap-store/638
loop7 7:7 0 54.2M 1 loop /snap/snap-store/558
loop8 7:8 0 49.8M 1 loop /snap/snapd/17950
loop9 7:9 0 63.3M 1 loop /snap/core20/1778
sda 8:0 0 3.7T 0 disk
├─sda2 8:2 0 48G 0 part
└─sda3 8:3 0 6.3M 0 part
sdb 8:16 0 3.7T 0 disk
├─sdb2 8:18 0 48G 0 part
└─sdb3 8:19 0 6.4M 0 part
sdc 8:32 0 3.7T 0 disk
├─sdc2 8:34 0 48G 0 part
└─sdc3 8:35 0 6.6M 0 part
sdd 8:48 0 3.7T 0 disk
├─sdd2 8:50 0 48G 0 part
└─sdd3 8:51 0 6.6M 0 part
sde 8:64 0 3.7T 0 disk
├─sde2 8:66 0 48G 0 part
└─sde3 8:67 0 6.2M 0 part
sdf 8:80 1 14.6T 0 disk
sdg 8:96 1 3.7T 0 disk
sdh 8:112 1 14.6T 0 disk
sdi 8:128 1 14.6T 0 disk
sdj 8:144 1 14.6T 0 disk
sdk 8:160 1 14.6T 0 disk
sdl 8:176 0 250G 0 disk /var/lib/kubelet/pods/bbc452f7-9d3c-4b9f-9e68-52643fd4978a/volumes/kubernetes.io~iscsi/plex-iscsi
nvme1n1 259:0 0 3.7T 0 disk
├─nvme1n1p1 259:1 0 1.8T 0 part
├─nvme1n1p2 259:2 0 119.2G 0 part
├─nvme1n1p3 259:3 0 119.2G 0 part
└─nvme1n1p4 259:4 0 1.6T 0 part
nvme0n1 259:5 0 3.7T 0 disk
├─nvme0n1p1 259:6 0 4.7G 0 part /boot/efi
└─nvme0n1p2 259:7 0 3.6T 0 part /
Note: Prior to install, /var/lib/rook was deleted from each worker node, and both wipefs -a and sgdisk --zap-all were run on all disks/
. . .
2023-01-24 16:52:25.768676 D | exec: Running command: lsblk /dev/sdf --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-01-24 16:52:25.771030 D | sys: lsblk output: "SIZE=\"16000900661248\" ROTA=\"1\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sdf\" KNAME=\"/dev/sdf\" MOUNTPOINT=\"\" FSTYPE=\"\""
2023-01-24 16:52:25.771041 D | exec: Running command: ceph-volume inventory --format json /dev/sdf
2023-01-24 16:52:26.255472 I | cephosd: skipping device "sdf": ["Insufficient space (<5GB)"].
2023-01-24 16:52:26.255486 I | cephosd: old lsblk can't detect bluestore signature, so try to detect here
2023-01-24 16:52:26.256997 D | exec: Running command: lsblk /dev/sdg --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-01-24 16:52:26.260105 D | sys: lsblk output: "SIZE=\"4000787030016\" ROTA=\"0\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sdg\" KNAME=\"/dev/sdg\" MOUNTPOINT=\"\" FSTYPE=\"\""
2023-01-24 16:52:26.260115 D | exec: Running command: ceph-volume inventory --format json /dev/sdg
2023-01-24 16:52:26.701314 I | cephosd: skipping device "sdg": ["Insufficient space (<5GB)"].
2023-01-24 16:52:26.701330 I | cephosd: old lsblk can't detect bluestore signature, so try to detect here
2023-01-24 16:52:26.701976 D | exec: Running command: lsblk /dev/sdh --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-01-24 16:52:26.704919 D | sys: lsblk output: "SIZE=\"16000900661248\" ROTA=\"1\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sdh\" KNAME=\"/dev/sdh\" MOUNTPOINT=\"\" FSTYPE=\"\""
2023-01-24 16:52:26.704928 D | exec: Running command: ceph-volume inventory --format json /dev/sdh
2023-01-24 16:52:27.192919 I | cephosd: skipping device "sdh": ["Insufficient space (<5GB)"].
2023-01-24 16:52:27.192932 I | cephosd: old lsblk can't detect bluestore signature, so try to detect here**
2023-01-24 16:52:27.193551 D | exec: Running command: lsblk /dev/sdi --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-01-24 16:52:27.196421 D | sys: lsblk output: "SIZE=\"16000900661248\" ROTA=\"1\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sdi\" KNAME=\"/dev/sdi\" MOUNTPOINT=\"\" FSTYPE=\"\""
2023-01-24 16:52:27.196433 D | exec: Running command: ceph-volume inventory --format json /dev/sdi
2023-01-24 16:52:27.705274 I | cephosd: skipping device "sdi": ["Insufficient space (<5GB)"].**
2023-01-24 16:52:27.705293 I | cephosd: old lsblk can't detect bluestore signature, so try to detect here
2023-01-24 16:52:27.705924 D | exec: Running command: lsblk /dev/sdj --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-01-24 16:52:27.709346 D | sys: lsblk output: "SIZE=\"16000900661248\" ROTA=\"1\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sdj\" KNAME=\"/dev/sdj\" MOUNTPOINT=\"\" FSTYPE=\"\""
2023-01-24 16:52:27.709362 D | exec: Running command: ceph-volume inventory --format json /dev/sdj
2023-01-24 16:52:28.157907 I | cephosd: skipping device "sdj": ["Insufficient space (<5GB)"].
2023-01-24 16:52:28.157920 I | cephosd: old lsblk can't detect bluestore signature, so try to detect here**
2023-01-24 16:52:28.158579 D | exec: Running command: lsblk /dev/sdk --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-01-24 16:52:28.161507 D | sys: lsblk output: "SIZE=\"16000900661248\" ROTA=\"1\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sdk\" KNAME=\"/dev/sdk\" MOUNTPOINT=\"\" FSTYPE=\"\""
2023-01-24 16:52:28.161521 D | exec: Running command: ceph-volume inventory --format json /dev/sdk
2023-01-24 16:52:28.656615 I | cephosd: skipping device "sdk": ["Insufficient space (<5GB)"].
2023-01-24 16:52:28.656629 I | cephosd: skipping device "sdl" with mountpoint "plex-iscsi"
2023-01-24 16:52:28.656633 I | cephosd: skipping device "nvme1n1p1" because it contains a filesystem "zfs_member"
2023-01-24 16:52:28.656635 I | cephosd: skipping device "nvme1n1p2" because it contains a filesystem "swap"
2023-01-24 16:52:28.656636 I | cephosd: skipping device "nvme1n1p3" because it contains a filesystem "ext4"
2023-01-24 16:52:28.656638 I | cephosd: skipping device "nvme1n1p4" because it contains a filesystem "ext4"
2023-01-24 16:52:28.656640 I | cephosd: skipping device "nvme0n1p1" with mountpoint "efi"
2023-01-24 16:52:28.656641 I | cephosd: skipping device "nvme0n1p2" with mountpoint "rootfs"
2023-01-24 16:52:28.678692 I | cephosd: configuring osd devices: {"Entries":{}}
2023-01-24 16:52:28.678704 I | cephosd: no new devices to configure. returning devices already configured with ceph-volume.
2023-01-24 16:52:28.678836 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log lvm list --format json
2023-01-24 16:52:29.028295 D | cephosd: {}
2023-01-24 16:52:29.028313 I | cephosd: 0 ceph-volume lvm osd devices configured on this node
2023-01-24 16:52:29.028325 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log raw list --format json
2023-01-24 16:52:29.944378 D | cephosd: {
"a129b0c2-c4da-4104-9c1e-374511664997": {
"ceph_fsid": "87b4e26e-497a-410c-8da3-f3bc09171b56",
"device": "/dev/sdc",
"osd_id": 4,
"osd_uuid": "a129b0c2-c4da-4104-9c1e-374511664997",
"type": "bluestore"
},
"afcc5fe1-2f59-4c8a-9e4c-8d1ba4233c2b": {
"ceph_fsid": "87b4e26e-497a-410c-8da3-f3bc09171b56",
"device": "/dev/sdb",
"osd_id": 3,
"osd_uuid": "afcc5fe1-2f59-4c8a-9e4c-8d1ba4233c2b",
"type": "bluestore"
},
"b632d16e-50ed-4abd-87ad-6aa3b7264200": {
"ceph_fsid": "87b4e26e-497a-410c-8da3-f3bc09171b56",
"device": "/dev/sde",
"osd_id": 1,
"osd_uuid": "b632d16e-50ed-4abd-87ad-6aa3b7264200",
"type": "bluestore"
},
"eefa0c99-4d6e-470a-915c-b980620573e3": {
"ceph_fsid": "87b4e26e-497a-410c-8da3-f3bc09171b56",
"device": "/dev/sda",
"osd_id": 2,
"osd_uuid": "eefa0c99-4d6e-470a-915c-b980620573e3",
"type": "bluestore"
},
"f76f5c1b-63cd-4599-88da-8b395e9edebc": {
"ceph_fsid": "87b4e26e-497a-410c-8da3-f3bc09171b56",
"device": "/dev/sdd",
"osd_id": 5,
"osd_uuid": "f76f5c1b-63cd-4599-88da-8b395e9edebc",
"type": "bluestore"
}
}
2023-01-24 16:52:29.944465 D | exec: Running command: lsblk /dev/sde --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-01-24 16:52:29.948562 D | sys: lsblk output: "SIZE=\"4000787030016\" ROTA=\"0\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sde\" KNAME=\"/dev/sde\" MOUNTPOINT=\"\" FSTYPE=\"ceph_bluestore\""
2023-01-24 16:52:29.948586 D | exec: Running command: sgdisk --print /dev/sde
2023-01-24 16:52:29.950467 I | cephosd: setting device class "ssd" for device "/dev/sde"
2023-01-24 16:52:29.950477 D | exec: Running command: lsblk /dev/sda --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-01-24 16:52:29.953892 D | sys: lsblk output: "SIZE=\"4000787030016\" ROTA=\"0\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sda\" KNAME=\"/dev/sda\" MOUNTPOINT=\"\" FSTYPE=\"ceph_bluestore\""
2023-01-24 16:52:29.953911 D | exec: Running command: sgdisk --print /dev/sda
2023-01-24 16:52:29.955724 I | cephosd: setting device class "ssd" for device "/dev/sda"
2023-01-24 16:52:29.955745 D | exec: Running command: lsblk /dev/sdd --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-01-24 16:52:29.958773 D | sys: lsblk output: "SIZE=\"4000787030016\" ROTA=\"0\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sdd\" KNAME=\"/dev/sdd\" MOUNTPOINT=\"\" FSTYPE=\"ceph_bluestore\""
2023-01-24 16:52:29.958792 D | exec: Running command: sgdisk --print /dev/sdd
2023-01-24 16:52:29.959941 I | cephosd: setting device class "ssd" for device "/dev/sdd"
2023-01-24 16:52:29.959949 D | exec: Running command: lsblk /dev/sdc --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-01-24 16:52:29.962328 D | sys: lsblk output: "SIZE=\"4000787030016\" ROTA=\"0\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sdc\" KNAME=\"/dev/sdc\" MOUNTPOINT=\"\" FSTYPE=\"ceph_bluestore\""
2023-01-24 16:52:29.962343 D | exec: Running command: sgdisk --print /dev/sdc
2023-01-24 16:52:29.963484 I | cephosd: setting device class "ssd" for device "/dev/sdc"
2023-01-24 16:52:29.963492 D | exec: Running command: lsblk /dev/sdb --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-01-24 16:52:29.966237 D | sys: lsblk output: "SIZE=\"4000787030016\" ROTA=\"0\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sdb\" KNAME=\"/dev/sdb\" MOUNTPOINT=\"\" FSTYPE=\"ceph_bluestore\""
2023-01-24 16:52:29.966255 D | exec: Running command: sgdisk --print /dev/sdb
2023-01-24 16:52:29.967578 I | cephosd: setting device class "ssd" for device "/dev/sdb"
2023-01-24 16:52:29.967588 I | cephosd: 5 ceph-volume raw osd devices configured on this node
Environment:
- OS (e.g. from /etc/os-release):
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
-
Kernel (e.g.
uname -a):Linux terranova 5.15.0-58-generic #64~20.04.1-Ubuntu SMP Fri Jan 6 16:42:31 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux -
Cloud provider or hardware configuration: Bare-metal Kubernetes installation, worker node
- AMD Ryzen 5950x
- 128gb DDR4 RAM
- GPU: Radeon 3090ti
-
Rook version (use
rook versioninside of a Rook Pod): I am using thev1.10.0branch to deploy. However,rookis not found as a command on the pods when deployed. -
Storage backend version (e.g. for ceph do
ceph -v):ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) -
Kubernetes version (use
kubectl version):v1.25.5 -
Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): Bare metal deployed with Kubespray
-
Storage backend status (e.g. for Ceph use
ceph healthin the Rook Ceph toolbox):
bash-4.4$ ceph health
HEALTH_WARN mon c is low on available space; Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded, 1 pg undersized
I am unclear as to why these disks are not being added to the cluster.
-
Is there a limit of OSDs per host? I noticed you can customize OSDs per device, but I didn’t see a similar item for hosts.
-
Could this be related to https://github.com/rook/rook/issues/11474 – Note that these are not removable disks, and are not part of any LVM, vgroup. They were previously part of a zfs cluster, but have been since wiped many times over as described above.
Thanks! Apologies in advance if this is a configuration issue 😃
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 1
- Comments: 31 (16 by maintainers)
I think we are slightly deviating from the main topic.
To summarize:
WTF? Why does ceph/rook ignore these devices? I rent root servers where I have almost no way to know beforehand what kind of harddisk is used. Now I have to rent a lot of them and cancel the ones with the wrong type just because ceph doesn’t like it? Or is there a real reason why that doesn’t work?