rook: OSD Resize Increases Used Capacity Not Available Capacity

** Previous bug report for same issue. Only this time with a different OS, VMware controller, Kubernetes version, Rook version, and Ceph version. Where is the bug?! https://github.com/rook/rook/issues/12511**

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior:

After resizing the underlying disk at the hypervisor and OS level resizing the OSD increases cluster total capacity and used capacity.

Expected behavior:

After resizing the underlying disk at the hypervisor and OS level resizing the OSD increases cluster total capacity and available capacity.

How to reproduce it (minimal and precise):

Build a Kubernetes cluster from virtual machines where Rook consumes virtual disks as OSDs. Resize a virtual disk used as an OSD at the hypervisor level. Resize the disk at the OS level. Resize the disk at the Ceph level (e.g., restart the OSD pod).

File(s) to submit:

image image

  • Cluster CR (custom resource), typically called cluster.yaml, if necessary: rook-ceph.yaml.txt

Logs to submit:

Cluster Status to submit:

  cluster:
    id:     e1ebb901-75ad-4b7c-90d9-69edf914c04e
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum a,b,d (age 2d)
    mgr: b(active, since 90m), standbys: a
    mds: 1/1 daemons up, 1 hot standby
    osd: 8 osds: 8 up (since 91m), 8 in (since 2h)
    rgw: 2 daemons active (2 hosts, 1 zones)
 
  data:
    volumes: 1/1 healthy
    pools:   12 pools, 265 pgs
    objects: 493.10k objects, 748 GiB
    usage:   1.3 TiB used, 1.5 TiB / 2.8 TiB avail
    pgs:     265 active+clean
 
  io:
    client:   103 KiB/s rd, 1.4 MiB/s wr, 22 op/s rd, 14 op/s wr

Environment:

  • OS (e.g. from /etc/os-release): Talos (v1.6.4)
  • Kernel (e.g. uname -a): 6.1.74-talos
  • Cloud provider or hardware configuration: VMware
  • Rook version (use rook version inside of a Rook Pod): v1.13.3
  • Storage backend version (e.g. for ceph do ceph -v): v18.2.2
  • Kubernetes version (use kubectl version): v1.29.1
  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): Talos
  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox): HEALTH_OK

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 29 (12 by maintainers)

Most upvoted comments

I checked rn. Ceph was upgraded to 18.2.4. I resized my volumes, but the changes were not picked up (as opposed to showing up as used). For the changes to take effect, the osd pods had to be restarted:

kubectl rollout restart deployment rook-ceph-osd-1 -n rook-ceph

Restarting the operator does not do this automatically. Is this expected? If so, I think it should be documented.

At the time the OSDs were upgraded to v18.2.4, they would have been restarted. After that, Rook wouldn’t restart them unless there is a change in the pod spec, such as another version update or setting change related to the OSD. Restarting the operator is not expected to restart all the daemons.

@travisn really appreciate your help. I’m closing this one out.

From the linked ceph tracker, since https://github.com/ceph/ceph/pull/55777 was merged to reef this is expected to be fixed in v18.2.3.

Good to see that the workaround with kill -9 in the pod fixed it.

The weight of osd.3 looks about half the size of the other OSDs, which would explain why the PGs are not balanced. But since the raw size is the same for all the OSDs, I’m not sure why the weight would be smaller. It seems the weight is the same as before the resize? Try ceph osd crush reweight to adjust it.

I spun up an AWS test cluster and confirmed that I see the same behavior…

The initial cluster has three OSDs:

sh-4.4$ ceph osd df
ID  CLASS  WEIGHT   REWEIGHT  SIZE    RAW USE  DATA     OMAP  META    AVAIL   %USE  VAR   PGS  STATUS
 1    ssd  0.00980   1.00000  10 GiB   27 MiB  724 KiB   0 B  26 MiB  10 GiB  0.26  1.00    1      up
 2    ssd  0.00980   1.00000  10 GiB   27 MiB  724 KiB   0 B  26 MiB  10 GiB  0.26  1.00    1      up
 0    ssd  0.00980   1.00000  10 GiB   27 MiB  720 KiB   0 B  26 MiB  10 GiB  0.26  1.00    1      up
                       TOTAL  30 GiB   81 MiB  2.1 MiB   0 B  79 MiB  30 GiB  0.26                  

After resizing, the OSD config is:

sh-4.4$ ceph osd df
ID  CLASS  WEIGHT   REWEIGHT  SIZE    RAW USE  DATA     OMAP     META    AVAIL   %USE   VAR   PGS  STATUS
 1    ssd  0.00980   1.00000  15 GiB  5.0 GiB  976 KiB    1 KiB  27 MiB  10 GiB  33.51  1.00    1      up
 2    ssd  0.00980   1.00000  15 GiB  5.0 GiB  976 KiB    1 KiB  27 MiB  10 GiB  33.51  1.00    1      up
 0    ssd  0.00980   1.00000  15 GiB  5.0 GiB  972 KiB    1 KiB  27 MiB  10 GiB  33.51  1.00    1      up
                       TOTAL  45 GiB   15 GiB  2.9 MiB  3.5 KiB  80 MiB  30 GiB  33.51                   

This would be a Ceph issue. Rook doesn’t have any influence on the sizes the OSDs that are reported, and I don’t see how the OSD resize could influence that incorrect raw size. Would you mind opening a Ceph tracker for this?