rook: Half of my disks are detected when OSDs are provisioned, the other half are skipped due to insufficient space

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior: The provisioner skips these disks.

Expected behavior: I would expect the OSD provisioner to detect these disks and add to the ceph cluster.

How to reproduce it (minimal and precise):

File(s) to submit:

I am using a non-modified version of ./deploy/examples/cluster.yaml from this repository.

Logs to submit:

Disks: (Disks sdf-sdk should be picked up my ceph as per my current understanding, but no action is taken upon them. 5 of these disks are 16TBB HDDs, and one is a 4TB SSD.

webdog@terranova:~$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0         7:0    0     4K  1 loop /snap/bare/5
loop1         7:1    0  63.2M  1 loop /snap/core20/1738
loop2         7:2    0  91.7M  1 loop /snap/gtk-common-themes/1535
loop3         7:3    0 346.3M  1 loop /snap/gnome-3-38-2004/119
loop4         7:4    0  49.6M  1 loop /snap/snapd/17883
loop5         7:5    0 346.3M  1 loop /snap/gnome-3-38-2004/115
loop6         7:6    0    46M  1 loop /snap/snap-store/638
loop7         7:7    0  54.2M  1 loop /snap/snap-store/558
loop8         7:8    0  49.8M  1 loop /snap/snapd/17950
loop9         7:9    0  63.3M  1 loop /snap/core20/1778
sda           8:0    0   3.7T  0 disk
├─sda2        8:2    0    48G  0 part
└─sda3        8:3    0   6.3M  0 part
sdb           8:16   0   3.7T  0 disk
├─sdb2        8:18   0    48G  0 part
└─sdb3        8:19   0   6.4M  0 part
sdc           8:32   0   3.7T  0 disk
├─sdc2        8:34   0    48G  0 part
└─sdc3        8:35   0   6.6M  0 part
sdd           8:48   0   3.7T  0 disk
├─sdd2        8:50   0    48G  0 part
└─sdd3        8:51   0   6.6M  0 part
sde           8:64   0   3.7T  0 disk
├─sde2        8:66   0    48G  0 part
└─sde3        8:67   0   6.2M  0 part
sdf           8:80   1  14.6T  0 disk
sdg           8:96   1   3.7T  0 disk
sdh           8:112  1  14.6T  0 disk
sdi           8:128  1  14.6T  0 disk
sdj           8:144  1  14.6T  0 disk
sdk           8:160  1  14.6T  0 disk
sdl           8:176  0   250G  0 disk /var/lib/kubelet/pods/bbc452f7-9d3c-4b9f-9e68-52643fd4978a/volumes/kubernetes.io~iscsi/plex-iscsi
nvme1n1     259:0    0   3.7T  0 disk
├─nvme1n1p1 259:1    0   1.8T  0 part
├─nvme1n1p2 259:2    0 119.2G  0 part
├─nvme1n1p3 259:3    0 119.2G  0 part
└─nvme1n1p4 259:4    0   1.6T  0 part
nvme0n1     259:5    0   3.7T  0 disk
├─nvme0n1p1 259:6    0   4.7G  0 part /boot/efi
└─nvme0n1p2 259:7    0   3.6T  0 part /

Note: Prior to install, /var/lib/rook was deleted from each worker node, and both wipefs -a and sgdisk --zap-all were run on all disks/


. . . 

2023-01-24 16:52:25.768676 D | exec: Running command: lsblk /dev/sdf --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-01-24 16:52:25.771030 D | sys: lsblk output: "SIZE=\"16000900661248\" ROTA=\"1\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sdf\" KNAME=\"/dev/sdf\" MOUNTPOINT=\"\" FSTYPE=\"\""
2023-01-24 16:52:25.771041 D | exec: Running command: ceph-volume inventory --format json /dev/sdf
2023-01-24 16:52:26.255472 I | cephosd: skipping device "sdf": ["Insufficient space (<5GB)"].
2023-01-24 16:52:26.255486 I | cephosd: old lsblk can't detect bluestore signature, so try to detect here


2023-01-24 16:52:26.256997 D | exec: Running command: lsblk /dev/sdg --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-01-24 16:52:26.260105 D | sys: lsblk output: "SIZE=\"4000787030016\" ROTA=\"0\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sdg\" KNAME=\"/dev/sdg\" MOUNTPOINT=\"\" FSTYPE=\"\""
2023-01-24 16:52:26.260115 D | exec: Running command: ceph-volume inventory --format json /dev/sdg
2023-01-24 16:52:26.701314 I | cephosd: skipping device "sdg": ["Insufficient space (<5GB)"].
2023-01-24 16:52:26.701330 I | cephosd: old lsblk can't detect bluestore signature, so try to detect here


2023-01-24 16:52:26.701976 D | exec: Running command: lsblk /dev/sdh --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-01-24 16:52:26.704919 D | sys: lsblk output: "SIZE=\"16000900661248\" ROTA=\"1\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sdh\" KNAME=\"/dev/sdh\" MOUNTPOINT=\"\" FSTYPE=\"\""
2023-01-24 16:52:26.704928 D | exec: Running command: ceph-volume inventory --format json /dev/sdh
2023-01-24 16:52:27.192919 I | cephosd: skipping device "sdh": ["Insufficient space (<5GB)"].
2023-01-24 16:52:27.192932 I | cephosd: old lsblk can't detect bluestore signature, so try to detect here**


2023-01-24 16:52:27.193551 D | exec: Running command: lsblk /dev/sdi --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-01-24 16:52:27.196421 D | sys: lsblk output: "SIZE=\"16000900661248\" ROTA=\"1\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sdi\" KNAME=\"/dev/sdi\" MOUNTPOINT=\"\" FSTYPE=\"\""
2023-01-24 16:52:27.196433 D | exec: Running command: ceph-volume inventory --format json /dev/sdi
2023-01-24 16:52:27.705274 I | cephosd: skipping device "sdi": ["Insufficient space (<5GB)"].**
2023-01-24 16:52:27.705293 I | cephosd: old lsblk can't detect bluestore signature, so try to detect here


2023-01-24 16:52:27.705924 D | exec: Running command: lsblk /dev/sdj --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-01-24 16:52:27.709346 D | sys: lsblk output: "SIZE=\"16000900661248\" ROTA=\"1\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sdj\" KNAME=\"/dev/sdj\" MOUNTPOINT=\"\" FSTYPE=\"\""
2023-01-24 16:52:27.709362 D | exec: Running command: ceph-volume inventory --format json /dev/sdj
2023-01-24 16:52:28.157907 I | cephosd: skipping device "sdj": ["Insufficient space (<5GB)"].
2023-01-24 16:52:28.157920 I | cephosd: old lsblk can't detect bluestore signature, so try to detect here**

2023-01-24 16:52:28.158579 D | exec: Running command: lsblk /dev/sdk --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-01-24 16:52:28.161507 D | sys: lsblk output: "SIZE=\"16000900661248\" ROTA=\"1\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sdk\" KNAME=\"/dev/sdk\" MOUNTPOINT=\"\" FSTYPE=\"\""

2023-01-24 16:52:28.161521 D | exec: Running command: ceph-volume inventory --format json /dev/sdk
2023-01-24 16:52:28.656615 I | cephosd: skipping device "sdk": ["Insufficient space (<5GB)"].


2023-01-24 16:52:28.656629 I | cephosd: skipping device "sdl" with mountpoint "plex-iscsi"
2023-01-24 16:52:28.656633 I | cephosd: skipping device "nvme1n1p1" because it contains a filesystem "zfs_member"
2023-01-24 16:52:28.656635 I | cephosd: skipping device "nvme1n1p2" because it contains a filesystem "swap"
2023-01-24 16:52:28.656636 I | cephosd: skipping device "nvme1n1p3" because it contains a filesystem "ext4"
2023-01-24 16:52:28.656638 I | cephosd: skipping device "nvme1n1p4" because it contains a filesystem "ext4"
2023-01-24 16:52:28.656640 I | cephosd: skipping device "nvme0n1p1" with mountpoint "efi"
2023-01-24 16:52:28.656641 I | cephosd: skipping device "nvme0n1p2" with mountpoint "rootfs"
2023-01-24 16:52:28.678692 I | cephosd: configuring osd devices: {"Entries":{}}
2023-01-24 16:52:28.678704 I | cephosd: no new devices to configure. returning devices already configured with ceph-volume.
2023-01-24 16:52:28.678836 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log lvm list  --format json
2023-01-24 16:52:29.028295 D | cephosd: {}
2023-01-24 16:52:29.028313 I | cephosd: 0 ceph-volume lvm osd devices configured on this node
2023-01-24 16:52:29.028325 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log raw list --format json
2023-01-24 16:52:29.944378 D | cephosd: {
    "a129b0c2-c4da-4104-9c1e-374511664997": {
        "ceph_fsid": "87b4e26e-497a-410c-8da3-f3bc09171b56",
        "device": "/dev/sdc",
        "osd_id": 4,
        "osd_uuid": "a129b0c2-c4da-4104-9c1e-374511664997",
        "type": "bluestore"
    },
    "afcc5fe1-2f59-4c8a-9e4c-8d1ba4233c2b": {
        "ceph_fsid": "87b4e26e-497a-410c-8da3-f3bc09171b56",
        "device": "/dev/sdb",
        "osd_id": 3,
        "osd_uuid": "afcc5fe1-2f59-4c8a-9e4c-8d1ba4233c2b",
        "type": "bluestore"
    },
    "b632d16e-50ed-4abd-87ad-6aa3b7264200": {
        "ceph_fsid": "87b4e26e-497a-410c-8da3-f3bc09171b56",
        "device": "/dev/sde",
        "osd_id": 1,
        "osd_uuid": "b632d16e-50ed-4abd-87ad-6aa3b7264200",
        "type": "bluestore"
    },
    "eefa0c99-4d6e-470a-915c-b980620573e3": {
        "ceph_fsid": "87b4e26e-497a-410c-8da3-f3bc09171b56",
        "device": "/dev/sda",
        "osd_id": 2,
        "osd_uuid": "eefa0c99-4d6e-470a-915c-b980620573e3",
        "type": "bluestore"
    },
    "f76f5c1b-63cd-4599-88da-8b395e9edebc": {
        "ceph_fsid": "87b4e26e-497a-410c-8da3-f3bc09171b56",
        "device": "/dev/sdd",
        "osd_id": 5,
        "osd_uuid": "f76f5c1b-63cd-4599-88da-8b395e9edebc",
        "type": "bluestore"
    }
}
2023-01-24 16:52:29.944465 D | exec: Running command: lsblk /dev/sde --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-01-24 16:52:29.948562 D | sys: lsblk output: "SIZE=\"4000787030016\" ROTA=\"0\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sde\" KNAME=\"/dev/sde\" MOUNTPOINT=\"\" FSTYPE=\"ceph_bluestore\""
2023-01-24 16:52:29.948586 D | exec: Running command: sgdisk --print /dev/sde
2023-01-24 16:52:29.950467 I | cephosd: setting device class "ssd" for device "/dev/sde"
2023-01-24 16:52:29.950477 D | exec: Running command: lsblk /dev/sda --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-01-24 16:52:29.953892 D | sys: lsblk output: "SIZE=\"4000787030016\" ROTA=\"0\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sda\" KNAME=\"/dev/sda\" MOUNTPOINT=\"\" FSTYPE=\"ceph_bluestore\""
2023-01-24 16:52:29.953911 D | exec: Running command: sgdisk --print /dev/sda
2023-01-24 16:52:29.955724 I | cephosd: setting device class "ssd" for device "/dev/sda"
2023-01-24 16:52:29.955745 D | exec: Running command: lsblk /dev/sdd --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-01-24 16:52:29.958773 D | sys: lsblk output: "SIZE=\"4000787030016\" ROTA=\"0\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sdd\" KNAME=\"/dev/sdd\" MOUNTPOINT=\"\" FSTYPE=\"ceph_bluestore\""
2023-01-24 16:52:29.958792 D | exec: Running command: sgdisk --print /dev/sdd
2023-01-24 16:52:29.959941 I | cephosd: setting device class "ssd" for device "/dev/sdd"
2023-01-24 16:52:29.959949 D | exec: Running command: lsblk /dev/sdc --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-01-24 16:52:29.962328 D | sys: lsblk output: "SIZE=\"4000787030016\" ROTA=\"0\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sdc\" KNAME=\"/dev/sdc\" MOUNTPOINT=\"\" FSTYPE=\"ceph_bluestore\""
2023-01-24 16:52:29.962343 D | exec: Running command: sgdisk --print /dev/sdc
2023-01-24 16:52:29.963484 I | cephosd: setting device class "ssd" for device "/dev/sdc"
2023-01-24 16:52:29.963492 D | exec: Running command: lsblk /dev/sdb --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME,MOUNTPOINT,FSTYPE
2023-01-24 16:52:29.966237 D | sys: lsblk output: "SIZE=\"4000787030016\" ROTA=\"0\" RO=\"0\" TYPE=\"disk\" PKNAME=\"\" NAME=\"/dev/sdb\" KNAME=\"/dev/sdb\" MOUNTPOINT=\"\" FSTYPE=\"ceph_bluestore\""
2023-01-24 16:52:29.966255 D | exec: Running command: sgdisk --print /dev/sdb
2023-01-24 16:52:29.967578 I | cephosd: setting device class "ssd" for device "/dev/sdb"
2023-01-24 16:52:29.967588 I | cephosd: 5 ceph-volume raw osd devices configured on this node

Environment:

  • OS (e.g. from /etc/os-release):
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
  • Kernel (e.g. uname -a): Linux terranova 5.15.0-58-generic #64~20.04.1-Ubuntu SMP Fri Jan 6 16:42:31 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

  • Cloud provider or hardware configuration: Bare-metal Kubernetes installation, worker node

    • AMD Ryzen 5950x
    • 128gb DDR4 RAM
    • GPU: Radeon 3090ti
  • Rook version (use rook version inside of a Rook Pod): I am using the v1.10.0 branch to deploy. However, rook is not found as a command on the pods when deployed.

  • Storage backend version (e.g. for ceph do ceph -v): ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)

  • Kubernetes version (use kubectl version): v1.25.5

  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): Bare metal deployed with Kubespray

  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):

bash-4.4$ ceph health
HEALTH_WARN mon c is low on available space; Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded, 1 pg undersized

I am unclear as to why these disks are not being added to the cluster.

  • Is there a limit of OSDs per host? I noticed you can customize OSDs per device, but I didn’t see a similar item for hosts.

  • Could this be related to https://github.com/rook/rook/issues/11474 – Note that these are not removable disks, and are not part of any LVM, vgroup. They were previously part of a zfs cluster, but have been since wiped many times over as described above.

Thanks! Apologies in advance if this is a configuration issue 😃

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 1
  • Comments: 31 (16 by maintainers)

Most upvoted comments

I think we are slightly deviating from the main topic.

To summarize:

  • current stable releases of Ceph don’t include the required fixes for hotpluggable devices support
  • for devel/testing purposes only you can use a non official image
  • for production environments I strongly advise against using non official images for obvious reasons. I get it can be frustrating/annoying but you have to wait for new released versions of Ceph.

WTF? Why does ceph/rook ignore these devices? I rent root servers where I have almost no way to know beforehand what kind of harddisk is used. Now I have to rent a lot of them and cancel the ones with the wrong type just because ceph doesn’t like it? Or is there a real reason why that doesn’t work?