rook: OSD does not start after disk resize due to expand-bluefs container crashing on Hetzner Cloud

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior: OSD pod stays at Init:CrashLoopBackOff

Expected behavior: OSD pod starting properly.

How to reproduce it (minimal and precise):

  1. Get a healthy rook cluster up and running with 3 OSDs (all nodes on arm64 arch)
  2. Increase the size of a disk
  3. Restart the osd pod where the disk was resized
  4. Observe the crash

Logs to submit:

inferring bluefs devices from bluestore path
2025-04-24T18:00:03.419+0000 ffffb71ed040 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0/block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
2025-04-24T18:00:03.719+0000 ffffb71ed040 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0/block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
2025-04-24T18:00:03.719+0000 ffffb71ed040 -1 bluestore(/var/lib/ceph/osd/ceph-0) _check_main_bdev_label not all labels read properly
/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/AVAILABLE_ARCH/arm64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.2/rpm/el9/BUILD/ceph-19.2.2/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::expand_devices(std::ostream&)' thread ffffb71ed040 time 2025-04-24T18:00:04.019618+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/AVAILABLE_ARCH/arm64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.2/rpm/el9/BUILD/ceph-19.2.2/src/os/bluestore/BlueStore.cc: 8844: FAILED ceph_assert(r == 0)
 ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x134) [0xffffb83b4f6c]
 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0xffffb83b50f4]
 3: (BlueStore::expand_devices(std::ostream&)+0x5a4) [0xaaaad6e42f54]
 4: main()
 5: /lib64/libc.so.6(+0x272c0) [0xffffb7d9c2c0]
 6: __libc_start_main()
 7: _start()
*** Caught signal (Aborted) **
 in thread ffffb71ed040 thread_name:ceph-bluestore-
2025-04-24T18:00:04.015+0000 ffffb71ed040 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/AVAILABLE_ARCH/arm64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.2/rpm/el9/BUILD/ceph-19.2.2/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::expand_devices(std::ostream&)' thread ffffb71ed040 time 2025-04-24T18:00:04.019618+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/AVAILABLE_ARCH/arm64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.2/rpm/el9/BUILD/ceph-19.2.2/src/os/bluestore/BlueStore.cc: 8844: FAILED ceph_assert(r == 0)

 ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x134) [0xffffb83b4f6c]
 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0xffffb83b50f4]
 3: (BlueStore::expand_devices(std::ostream&)+0x5a4) [0xaaaad6e42f54]
 4: main()
 5: /lib64/libc.so.6(+0x272c0) [0xffffb7d9c2c0]
 6: __libc_start_main()
 7: _start()

 ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)
 1: __kernel_rt_sigreturn()
 2: /lib64/libc.so.6(+0x82678) [0xffffb7df7678]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18c) [0xffffb83b4fc4]
 6: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0xffffb83b50f4]
 7: (BlueStore::expand_devices(std::ostream&)+0x5a4) [0xaaaad6e42f54]
 8: main()
 9: /lib64/libc.so.6(+0x272c0) [0xffffb7d9c2c0]
 10: __libc_start_main()
 11: _start()
2025-04-24T18:00:04.019+0000 ffffb71ed040 -1 *** Caught signal (Aborted) **
 in thread ffffb71ed040 thread_name:ceph-bluestore-

 ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)
 1: __kernel_rt_sigreturn()
 2: /lib64/libc.so.6(+0x82678) [0xffffb7df7678]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18c) [0xffffb83b4fc4]
 6: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0xffffb83b50f4]
 7: (BlueStore::expand_devices(std::ostream&)+0x5a4) [0xaaaad6e42f54]
 8: main()
 9: /lib64/libc.so.6(+0x272c0) [0xffffb7d9c2c0]
 10: __libc_start_main()
 11: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

   -39> 2025-04-24T18:00:03.419+0000 ffffb71ed040 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0/block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
   -38> 2025-04-24T18:00:03.719+0000 ffffb71ed040 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0/block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
   -37> 2025-04-24T18:00:03.719+0000 ffffb71ed040 -1 bluestore(/var/lib/ceph/osd/ceph-0) _check_main_bdev_label not all labels read properly
   -36> 2025-04-24T18:00:04.015+0000 ffffb71ed040 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/AVAILABLE_ARCH/arm64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.2/rpm/el9/BUILD/ceph-19.2.2/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::expand_devices(std::ostream&)' thread ffffb71ed040 time 2025-04-24T18:00:04.019618+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/AVAILABLE_ARCH/arm64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.2/rpm/el9/BUILD/ceph-19.2.2/src/os/bluestore/BlueStore.cc: 8844: FAILED ceph_assert(r == 0)

 ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x134) [0xffffb83b4f6c]
 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0xffffb83b50f4]
 3: (BlueStore::expand_devices(std::ostream&)+0x5a4) [0xaaaad6e42f54]
 4: main()
 5: /lib64/libc.so.6(+0x272c0) [0xffffb7d9c2c0]
 6: __libc_start_main()
 7: _start()

   -35> 2025-04-24T18:00:04.019+0000 ffffb71ed040 -1 *** Caught signal (Aborted) **
 in thread ffffb71ed040 thread_name:ceph-bluestore-

 ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)
 1: __kernel_rt_sigreturn()
 2: /lib64/libc.so.6(+0x82678) [0xffffb7df7678]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18c) [0xffffb83b4fc4]
 6: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0xffffb83b50f4]
 7: (BlueStore::expand_devices(std::ostream&)+0x5a4) [0xaaaad6e42f54]
 8: main()
 9: /lib64/libc.so.6(+0x272c0) [0xffffb7d9c2c0]
 10: __libc_start_main()
 11: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

    -9> 2025-04-24T18:00:03.419+0000 ffffb71ed040 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0/block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
    -4> 2025-04-24T18:00:03.719+0000 ffffb71ed040 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0/block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
    -3> 2025-04-24T18:00:03.719+0000 ffffb71ed040 -1 bluestore(/var/lib/ceph/osd/ceph-0) _check_main_bdev_label not all labels read properly
    -1> 2025-04-24T18:00:04.015+0000 ffffb71ed040 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/AVAILABLE_ARCH/arm64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.2/rpm/el9/BUILD/ceph-19.2.2/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::expand_devices(std::ostream&)' thread ffffb71ed040 time 2025-04-24T18:00:04.019618+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/arm64/AVAILABLE_ARCH/arm64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.2/rpm/el9/BUILD/ceph-19.2.2/src/os/bluestore/BlueStore.cc: 8844: FAILED ceph_assert(r == 0)

 ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x134) [0xffffb83b4f6c]
 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0xffffb83b50f4]
 3: (BlueStore::expand_devices(std::ostream&)+0x5a4) [0xaaaad6e42f54]
 4: main()
 5: /lib64/libc.so.6(+0x272c0) [0xffffb7d9c2c0]
 6: __libc_start_main()
 7: _start()

     0> 2025-04-24T18:00:04.019+0000 ffffb71ed040 -1 *** Caught signal (Aborted) **
 in thread ffffb71ed040 thread_name:ceph-bluestore-

 ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)
 1: __kernel_rt_sigreturn()
 2: /lib64/libc.so.6(+0x82678) [0xffffb7df7678]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18c) [0xffffb83b4fc4]
 6: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0xffffb83b50f4]
 7: (BlueStore::expand_devices(std::ostream&)+0x5a4) [0xaaaad6e42f54]
 8: main()
 9: /lib64/libc.so.6(+0x272c0) [0xffffb7d9c2c0]
 10: __libc_start_main()
 11: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Environment:

  • OS (e.g. from /etc/os-release): NixOS 25.05 (Warbler)
  • Kernel (e.g. uname -a): 6.12.23
  • Cloud provider or hardware configuration: Hetzner VPS
  • Rook version (use rook version inside of a Rook Pod): docker.io/rook/ceph:v1.17.0
  • Storage backend version (e.g. for ceph do ceph -v): quay.io/ceph/ceph:v19.2.2
  • Kubernetes version (use kubectl version): 1.32.2
  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): default / vanilla
  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 3
  • Comments: 17 (6 by maintainers)

Most upvoted comments

To summarize, it appears this OSD issue affects Ceph v19.2.1 and v19.2.2. The fix is expected in v19.2.3.

@m4r1vs Can you add a link to your tracker here? If the fix is merged to the squid branch, it could be tested with with the tag quay.ceph.io/ceph-ci/ceph:squid.

@travisn I did some more testing and it seems i’m unable to reproduce this on Ceph 18.2.6, so seems like something that got introduced with 19.

Edit: reproduced also on 19.2.1 wasn’t yet able to reproduce on 19.2.0

Downgrading to 19.2.0 from 19.2.2 also works for me, should I open an issue over at the ceph repository?

@travisn I did some more testing and it seems i’m unable to reproduce this on Ceph 18.2.6, so seems like something that got introduced with 19.

Edit: reproduced also on 19.2.1 wasn’t yet able to reproduce on 19.2.0

yes, that’s right: https://tracker.ceph.com/issues/71180

And thanks for the reference to the image, I think I can test it tomorrow sometime

only registered users are able to create an issue on their tracker?

Correct. In addition, account creation process is not automated. If your application is not accepted for several days, I suggest you to ask about this in the Ceph slack.

@m4r1vs Yes, sounds good to open a Ceph tracker, perhaps the issue will be more obvious to them.

Ok interesting to see you also repro also on Hetzner cloud though on amd64, not just arm64 as indicated in the title (i’ll update that).