rook: OSD Resize Increases Used Capacity Not Available Capacity
** Previous bug report for same issue. Only this time with a different OS, VMware controller, Kubernetes version, Rook version, and Ceph version. Where is the bug?! https://github.com/rook/rook/issues/12511**
Is this a bug report or feature request?
- Bug Report
Deviation from expected behavior:
After resizing the underlying disk at the hypervisor and OS level resizing the OSD increases cluster total capacity and used capacity.
Expected behavior:
After resizing the underlying disk at the hypervisor and OS level resizing the OSD increases cluster total capacity and available capacity.
How to reproduce it (minimal and precise):
Build a Kubernetes cluster from virtual machines where Rook consumes virtual disks as OSDs. Resize a virtual disk used as an OSD at the hypervisor level. Resize the disk at the OS level. Resize the disk at the Ceph level (e.g., restart the OSD pod).
File(s) to submit:
- Cluster CR (custom resource), typically called
cluster.yaml, if necessary: rook-ceph.yaml.txt
Logs to submit:
- Operator’s logs, if necessary: Explore-logs-2024-04-19 09_53_04.txt
- Crashing pod(s) logs, if necessary: N/A
Cluster Status to submit:
cluster:
id: e1ebb901-75ad-4b7c-90d9-69edf914c04e
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,d (age 2d)
mgr: b(active, since 90m), standbys: a
mds: 1/1 daemons up, 1 hot standby
osd: 8 osds: 8 up (since 91m), 8 in (since 2h)
rgw: 2 daemons active (2 hosts, 1 zones)
data:
volumes: 1/1 healthy
pools: 12 pools, 265 pgs
objects: 493.10k objects, 748 GiB
usage: 1.3 TiB used, 1.5 TiB / 2.8 TiB avail
pgs: 265 active+clean
io:
client: 103 KiB/s rd, 1.4 MiB/s wr, 22 op/s rd, 14 op/s wr
Environment:
- OS (e.g. from /etc/os-release): Talos (v1.6.4)
- Kernel (e.g.
uname -a): 6.1.74-talos - Cloud provider or hardware configuration: VMware
- Rook version (use
rook versioninside of a Rook Pod): v1.13.3 - Storage backend version (e.g. for ceph do
ceph -v): v18.2.2 - Kubernetes version (use
kubectl version): v1.29.1 - Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): Talos
- Storage backend status (e.g. for Ceph use
ceph healthin the Rook Ceph toolbox): HEALTH_OK
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 29 (12 by maintainers)
At the time the OSDs were upgraded to v18.2.4, they would have been restarted. After that, Rook wouldn’t restart them unless there is a change in the pod spec, such as another version update or setting change related to the OSD. Restarting the operator is not expected to restart all the daemons.
@travisn really appreciate your help. I’m closing this one out.
From the linked ceph tracker, since https://github.com/ceph/ceph/pull/55777 was merged to reef this is expected to be fixed in v18.2.3.
Good to see that the workaround with
kill -9in the pod fixed it.The
weightof osd.3 looks about half the size of the other OSDs, which would explain why the PGs are not balanced. But since the raw size is the same for all the OSDs, I’m not sure why the weight would be smaller. It seems the weight is the same as before the resize? Tryceph osd crush reweightto adjust it.I spun up an AWS test cluster and confirmed that I see the same behavior…
The initial cluster has three OSDs:
After resizing, the OSD config is:
This would be a Ceph issue. Rook doesn’t have any influence on the sizes the OSDs that are reported, and I don’t see how the OSD resize could influence that incorrect raw size. Would you mind opening a Ceph tracker for this?