rook: Ceph-main tests failing to create OSDs on daily CI smoke suite

This bug seems similar to https://github.com/rook/rook/issues/15282

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior: The smoke suite is failing when run against the ceph-main in the daily CI. For example, see this failure

This failure started failing on Feb 14th. The operator and osd logs can be found in the artifacts under ceph-smoke-suite-squid-artifact.

The OSD prepare log shows that the OSD is failing to be provisioned by c-v with the following error:

2025-02-14 00:30:01.763996 D | exec: Running command: stdbuf -oL ceph-volume raw prepare --bluestore --data /dev/sdb1 --crush-device-class hdd
2025-02-14 00:30:04.275150 I | cephosd: Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 16b4711c-9ac3-4124-ab9a-9404e76ad31b
Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
Running command: /usr/bin/chown -R ceph:ceph /dev/sdb1
Running command: /usr/bin/ln -s /dev/sdb1 /var/lib/ceph/osd/ceph-0/block
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap
 stderr: got monmap epoch 3
--> Creating keyring file for osd.0
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 16b4711c-9ac3-4124-ab9a-9404e76ad31b --setuser ceph --setgroup ceph
 stderr: 2025-02-14T00:30:03.219+0000 7f447e63f8c0 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label /var/lib/ceph/osd/ceph-0//block data at 0, unable to decode label
 stderr: 2025-02-14T00:30:03.219+0000 7f447e63f8c0 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label /var/lib/ceph/osd/ceph-0//block data at 40000000, unable to decode label
 stderr: 2025-02-14T00:30:03.219+0000 7f447e63f8c0 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label /var/lib/ceph/osd/ceph-0//block data at 280000000, unable to decode label
 stderr: 2025-02-14T00:30:03.219+0000 7f447e63f8c0 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) No valid bdev label found
 stderr: 2025-02-14T00:30:03.275+0000 7f447e63f8c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
--> ceph-volume raw dmcrypt prepare successful for: /dev/sdb1
2025-02-14 00:30:04.275439 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log lvm list  --format json
2025-02-14 00:30:04.665029 D | cephosd: {}
2025-02-14 00:30:04.665060 I | cephosd: 0 ceph-volume lvm osd devices configured on this node
2025-02-14 00:30:04.665081 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log raw list --format json
2025-02-14 00:30:09.511047 D | cephosd: {}
2025-02-14 00:30:09.511071 I | cephosd: 0 ceph-volume raw osd devices configured on this node
2025-02-14 00:30:09.511079 W | cephosd: skipping OSD configuration as no devices matched the storage settings for this node "test-prefix-this-is-a-very-long-hostname-fv-az774-68"

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 32 (11 by maintainers)

Most upvoted comments

https://github.com/ceph/ceph/pull/62009 just merged. It still did not pass teuthology, so some extra fixes might be needed in future, but it should unblock rook.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

but it should unblock rook.

And ramen, thank you for this fix!

Fix: https://github.com/ceph/ceph/pull/62009 Due to urgency, it will pass make check, check if shaman build properly and merged skipping teuthology. ETA: 4h

@aclamk @travisn ceph-volume raw list implies a call to ceph-bluestore-tool like the following in the case of rook CI:

ceph-bluestore-tool show-label --bdev_aio_poll_ms=1 --dev /dev/sda --dev /dev/sda1 --dev /dev/sda14 --dev /dev/sda15 --dev /dev/sdb --dev /dev/sdb1 --dev /dev/nbd0 --dev /dev/nbd1 --dev /dev/nbd2 --dev /dev/nbd3 --dev /dev/nbd4 --dev /dev/nbd5 --dev /dev/nbd6 --dev /dev/nbd7 --dev /dev/nbd8 --dev /dev/nbd9 --dev /dev/nbd10 --dev /dev/nbd11 --dev /dev/nbd1
2 --dev /dev/nbd13 --dev /dev/nbd14 --dev /dev/nbd15

when running that command in a rook environment we can see it crashes like this (sorry for the long output):

[root@rook-ceph-osd-prepare-1fde8db402101a3cda7e6f8f91c5d8d4-vbwsj /]# ceph-bluestore-tool show-label --bdev_aio_poll_ms=1 --dev /dev/sda --dev /dev/sda1 --dev /dev/sda14 --dev /dev/sda15 --dev /dev/sdb --dev /dev/sdb1 --dev /dev/nbd0 --dev /dev/nbd1 --dev /dev/nbd2 --dev /dev/nbd3 --dev /dev/nbd4 --dev /dev/nbd5 --dev /dev/nbd6 --dev /dev/nbd7 --dev /dev/nbd8 --dev /dev/nbd9 --dev /dev/nbd10 --dev /dev/nbd11 --dev /dev/nbd1
2 --dev /dev/nbd13 --dev /dev/nbd14 --dev /dev/nbd15
2025-02-25T15:31:09.647+0000 7f6329c28c00 -1 bluestore(/dev/sda) _read_bdev_label /dev/sda data at 0, unable to decode label
2025-02-25T15:31:09.647+0000 7f6329c28c00 -1 bluestore(/dev/sda) _read_bdev_label /dev/sda data at 40000000, unable to decode label
2025-02-25T15:31:09.647+0000 7f6329c28c00 -1 bluestore(/dev/sda) _read_bdev_label /dev/sda data at 280000000, unable to decode label
2025-02-25T15:31:09.647+0000 7f6329c28c00 -1 bluestore(/dev/sda) No valid bdev label found
unable to read label for /dev/sda: (2) No such file or directory
2025-02-25T15:31:09.687+0000 7f6329c28c00 -1 bluestore(/dev/sda1) _read_bdev_label /dev/sda1 data at 0, unable to decode label
2025-02-25T15:31:09.687+0000 7f6329c28c00 -1 bluestore(/dev/sda1) _read_bdev_label /dev/sda1 data at 40000000, unable to decode label
2025-02-25T15:31:09.687+0000 7f6329c28c00 -1 bluestore(/dev/sda1) _read_bdev_label /dev/sda1 data at 280000000, unable to decode label
2025-02-25T15:31:09.687+0000 7f6329c28c00 -1 bluestore(/dev/sda1) No valid bdev label found
unable to read label for /dev/sda1: (2) No such file or directory
2025-02-25T15:31:09.727+0000 7f6329c28c00 -1 bluestore(/dev/sda14) _read_bdev_label /dev/sda14 data at 0, unable to decode label
2025-02-25T15:31:09.727+0000 7f6329c28c00 -1 bluestore(/dev/sda14) No valid bdev label found
unable to read label for /dev/sda14: (2) No such file or directory
2025-02-25T15:31:09.775+0000 7f6329c28c00 -1 bluestore(/dev/sda15) _read_bdev_label /dev/sda15 data at 0, unable to decode label
2025-02-25T15:31:09.775+0000 7f6329c28c00 -1 bluestore(/dev/sda15) No valid bdev label found
unable to read label for /dev/sda15: (2) No such file or directory
2025-02-25T15:31:09.811+0000 7f6329c28c00 -1 bluestore(/dev/sdb) _read_bdev_label /dev/sdb data at 0, unable to decode label
2025-02-25T15:31:09.811+0000 7f6329c28c00 -1 bluestore(/dev/sdb) _read_bdev_label /dev/sdb data at 40000000, unable to decode label
2025-02-25T15:31:09.811+0000 7f6329c28c00 -1 bluestore(/dev/sdb) _read_bdev_label /dev/sdb data at 280000000, unable to decode label
2025-02-25T15:31:09.811+0000 7f6329c28c00 -1 bluestore(/dev/sdb) No valid bdev label found
unable to read label for /dev/sdb: (2) No such file or directory
/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-7819-g290894cc/rpm/el9/BUILD/ceph-19.3.0-7819-g290894cc/src/blk/kernel/KernelDevice.cc: In function 'virtual int KernelDevice::aio_read(uint64_t, uint64_t, ceph::bufferlist*, IOContext*)' thread 7f6329c28c00 time 2025-02-25T15:31:09.927527+0000
/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-7819-g290894cc/rpm/el9/BUILD/ceph-19.3.0-7819-g290894cc/src/blk/kernel/KernelDevice.cc: 1411: FAILED ceph_assert(is_valid_io(off, len))
2025-02-25T15:31:09.923+0000 7f6329c28c00 -1 bluestore(/dev/nbd0) No valid bdev label found
2025-02-25T15:31:09.923+0000 7f6329c28c00 -1 bdev is_valid_io 0~1000 block_size 1000 size 0
 ceph version 19.3.0-7819-g290894cc (290894cc762024be7774bdda6a02815a1e08555e) squid (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11e) [0x7f632adf1654]
 2: /usr/lib64/ceph/libceph-common.so.2(+0x183813) [0x7f632adf1813]
 3: (KernelDevice::aio_read(unsigned long, unsigned long, ceph::buffer::v15_2_0::list*, IOContext*)+0x99e) [0x55698e8a7d4e]
 4: (KernelDevice::_aio_stop()+0x143) [0x55698e89ad13]
 5: (KernelDevice::close()+0x1ad) [0x55698e89e34d]
 6: (BlueStore::read_bdev_label(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bluestore_bdev_label_t*, std::vector<unsigned long, std::allocator<unsigned long> >*, bool*, long*)+0xc2) [0x55698e732502]
 7: main()
 8: /lib64/libc.so.6(+0x295d0) [0x7f632a7655d0]
 9: __libc_start_main()
 10: _start()
*** Caught signal (Aborted) **
 in thread 7f6329c28c00 thread_name:ceph-bluestore-
2025-02-25T15:31:09.923+0000 7f6329c28c00 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-7819-g290894cc/rpm/el9/BUILD/ceph-19.3.0-7819-g290894cc/src/blk/kernel/KernelDevice.cc: In function 'virtual int KernelDevice::aio_read(uint64_t, uint64_t, ceph::bufferlist*, IOContext*)' thread 7f6329c28c00 time 2025-02-25T1
5:31:09.927527+0000
/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-7819-g290894cc/rpm/el9/BUILD/ceph-19.3.0-7819-g290894cc/src/blk/kernel/KernelDevice.cc: 1411: FAILED ceph_assert(is_valid_io(off, len))
 
 ceph version 19.3.0-7819-g290894cc (290894cc762024be7774bdda6a02815a1e08555e) squid (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11e) [0x7f632adf1654]
 2: /usr/lib64/ceph/libceph-common.so.2(+0x183813) [0x7f632adf1813]
 3: (KernelDevice::aio_read(unsigned long, unsigned long, ceph::buffer::v15_2_0::list*, IOContext*)+0x99e) [0x55698e8a7d4e]
 4: (KernelDevice::_aio_stop()+0x143) [0x55698e89ad13]
 5: (KernelDevice::close()+0x1ad) [0x55698e89e34d]
 6: (BlueStore::read_bdev_label(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bluestore_bdev_label_t*, std::vector<unsigned long, std::allocator<unsigned long> >*, bool*, long*)+0xc2) [0x55698e732502]
 7: main()
 8: /lib64/libc.so.6(+0x295d0) [0x7f632a7655d0]
 9: __libc_start_main()
 10: _start()
 
 ceph version 19.3.0-7819-g290894cc (290894cc762024be7774bdda6a02815a1e08555e) squid (dev)                                                                                                                                                                                                                                                                                                                                          [64/392]
 1: /lib64/libc.so.6(+0x3ea60) [0x7f632a77aa60]
 2: /lib64/libc.so.6(+0x8c0fc) [0x7f632a7c80fc]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x178) [0x7f632adf16ae]
 6: /usr/lib64/ceph/libceph-common.so.2(+0x183813) [0x7f632adf1813]
 7: (KernelDevice::aio_read(unsigned long, unsigned long, ceph::buffer::v15_2_0::list*, IOContext*)+0x99e) [0x55698e8a7d4e]
 8: (KernelDevice::_aio_stop()+0x143) [0x55698e89ad13]
 9: (KernelDevice::close()+0x1ad) [0x55698e89e34d]
 10: (BlueStore::read_bdev_label(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bluestore_bdev_label_t*, std::vector<unsigned long, std::allocator<unsigned long> >*, bool*, long*)+0xc2) [0x55698e732502]
 11: main()
 12: /lib64/libc.so.6(+0x295d0) [0x7f632a7655d0]
 13: __libc_start_main()
 14: _start()
2025-02-25T15:31:09.923+0000 7f6329c28c00 -1 *** Caught signal (Aborted) **
 in thread 7f6329c28c00 thread_name:ceph-bluestore-
 
 ceph version 19.3.0-7819-g290894cc (290894cc762024be7774bdda6a02815a1e08555e) squid (dev)
 1: /lib64/libc.so.6(+0x3ea60) [0x7f632a77aa60]
 2: /lib64/libc.so.6(+0x8c0fc) [0x7f632a7c80fc]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x178) [0x7f632adf16ae]
 6: /usr/lib64/ceph/libceph-common.so.2(+0x183813) [0x7f632adf1813]
 7: (KernelDevice::aio_read(unsigned long, unsigned long, ceph::buffer::v15_2_0::list*, IOContext*)+0x99e) [0x55698e8a7d4e]
 8: (KernelDevice::_aio_stop()+0x143) [0x55698e89ad13]
 9: (KernelDevice::close()+0x1ad) [0x55698e89e34d]
 10: (BlueStore::read_bdev_label(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bluestore_bdev_label_t*, std::vector<unsigned long, std::allocator<unsigned long> >*, bool*, long*)+0xc2) [0x55698e732502]
 11: main()
 12: /lib64/libc.so.6(+0x295d0) [0x7f632a7655d0]
 13: __libc_start_main()
 14: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
 
   -73> 2025-02-25T15:31:09.647+0000 7f6329c28c00 -1 bluestore(/dev/sda) _read_bdev_label /dev/sda data at 0, unable to decode label
   -72> 2025-02-25T15:31:09.647+0000 7f6329c28c00 -1 bluestore(/dev/sda) _read_bdev_label /dev/sda data at 40000000, unable to decode label
   -71> 2025-02-25T15:31:09.647+0000 7f6329c28c00 -1 bluestore(/dev/sda) _read_bdev_label /dev/sda data at 280000000, unable to decode label
   -70> 2025-02-25T15:31:09.647+0000 7f6329c28c00 -1 bluestore(/dev/sda) No valid bdev label found
   -69> 2025-02-25T15:31:09.687+0000 7f6329c28c00 -1 bluestore(/dev/sda1) _read_bdev_label /dev/sda1 data at 0, unable to decode label
   -68> 2025-02-25T15:31:09.687+0000 7f6329c28c00 -1 bluestore(/dev/sda1) _read_bdev_label /dev/sda1 data at 40000000, unable to decode label
   -67> 2025-02-25T15:31:09.687+0000 7f6329c28c00 -1 bluestore(/dev/sda1) _read_bdev_label /dev/sda1 data at 280000000, unable to decode label
   -66> 2025-02-25T15:31:09.687+0000 7f6329c28c00 -1 bluestore(/dev/sda1) No valid bdev label found
   -65> 2025-02-25T15:31:09.727+0000 7f6329c28c00 -1 bluestore(/dev/sda14) _read_bdev_label /dev/sda14 data at 0, unable to decode label
   -64> 2025-02-25T15:31:09.727+0000 7f6329c28c00 -1 bluestore(/dev/sda14) No valid bdev label found
   -63> 2025-02-25T15:31:09.775+0000 7f6329c28c00 -1 bluestore(/dev/sda15) _read_bdev_label /dev/sda15 data at 0, unable to decode label
   -62> 2025-02-25T15:31:09.775+0000 7f6329c28c00 -1 bluestore(/dev/sda15) No valid bdev label found
   -61> 2025-02-25T15:31:09.811+0000 7f6329c28c00 -1 bluestore(/dev/sdb) _read_bdev_label /dev/sdb data at 0, unable to decode label
   -60> 2025-02-25T15:31:09.811+0000 7f6329c28c00 -1 bluestore(/dev/sdb) _read_bdev_label /dev/sdb data at 40000000, unable to decode label
   -59> 2025-02-25T15:31:09.811+0000 7f6329c28c00 -1 bluestore(/dev/sdb) _read_bdev_label /dev/sdb data at 280000000, unable to decode label
   -58> 2025-02-25T15:31:09.811+0000 7f6329c28c00 -1 bluestore(/dev/sdb) No valid bdev label found
   -57> 2025-02-25T15:31:09.923+0000 7f6329c28c00 -1 bluestore(/dev/nbd0) No valid bdev label found
   -56> 2025-02-25T15:31:09.923+0000 7f6329c28c00 -1 bdev is_valid_io 0~1000 block_size 1000 size 0
   -55> 2025-02-25T15:31:09.923+0000 7f6329c28c00 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-7819-g290894cc/rpm/el9/BUILD/ceph-19.3.0-7819-g290894cc/src/blk/kernel/KernelDevice.cc: In function 'virtual int KernelDevice::aio_read(uint64_t, uint64_t, ceph::bufferlist*, IOContext*)' thread 7f6329c28c00 time 2025
-02-25T15:31:09.927527+0000
/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-7819-g290894cc/rpm/el9/BUILD/ceph-19.3.0-7819-g290894cc/src/blk/kernel/KernelDevice.cc: 1411: FAILED ceph_assert(is_valid_io(off, len))
 
 ceph version 19.3.0-7819-g290894cc (290894cc762024be7774bdda6a02815a1e08555e) squid (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11e) [0x7f632adf1654]
 2: /usr/lib64/ceph/libceph-common.so.2(+0x183813) [0x7f632adf1813]
 3: (KernelDevice::aio_read(unsigned long, unsigned long, ceph::buffer::v15_2_0::list*, IOContext*)+0x99e) [0x55698e8a7d4e]
 4: (KernelDevice::_aio_stop()+0x143) [0x55698e89ad13]
 5: (KernelDevice::close()+0x1ad) [0x55698e89e34d]
 6: (BlueStore::read_bdev_label(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bluestore_bdev_label_t*, std::vector<unsigned long, std::allocator<unsigned long> >*, bool*, long*)+0xc2) [0x55698e732502]
 7: main()
 8: /lib64/libc.so.6(+0x295d0) [0x7f632a7655d0]
 9: __libc_start_main()
 10: _start()
 
   -54> 2025-02-25T15:31:09.923+0000 7f6329c28c00 -1 *** Caught signal (Aborted) **
 in thread 7f6329c28c00 thread_name:ceph-bluestore-
 
 ceph version 19.3.0-7819-g290894cc (290894cc762024be7774bdda6a02815a1e08555e) squid (dev)
 1: /lib64/libc.so.6(+0x3ea60) [0x7f632a77aa60]
 2: /lib64/libc.so.6(+0x8c0fc) [0x7f632a7c80fc]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x178) [0x7f632adf16ae]
 6: /usr/lib64/ceph/libceph-common.so.2(+0x183813) [0x7f632adf1813]
 7: (KernelDevice::aio_read(unsigned long, unsigned long, ceph::buffer::v15_2_0::list*, IOContext*)+0x99e) [0x55698e8a7d4e]
 8: (KernelDevice::_aio_stop()+0x143) [0x55698e89ad13]
 9: (KernelDevice::close()+0x1ad) [0x55698e89e34d]
 10: (BlueStore::read_bdev_label(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bluestore_bdev_label_t*, std::vector<unsigned long, std::allocator<unsigned long> >*, bool*, long*)+0xc2) [0x55698e732502]
 11: main()
 12: /lib64/libc.so.6(+0x295d0) [0x7f632a7655d0]
 13: __libc_start_main()
 14: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
 
   -44> 2025-02-25T15:31:09.647+0000 7f6329c28c00 -1 bluestore(/dev/sda) _read_bdev_label /dev/sda data at 0, unable to decode label
   -43> 2025-02-25T15:31:09.647+0000 7f6329c28c00 -1 bluestore(/dev/sda) _read_bdev_label /dev/sda data at 40000000, unable to decode label
   -42> 2025-02-25T15:31:09.647+0000 7f6329c28c00 -1 bluestore(/dev/sda) _read_bdev_label /dev/sda data at 280000000, unable to decode label
   -41> 2025-02-25T15:31:09.647+0000 7f6329c28c00 -1 bluestore(/dev/sda) No valid bdev label found
   -36> 2025-02-25T15:31:09.687+0000 7f6329c28c00 -1 bluestore(/dev/sda1) _read_bdev_label /dev/sda1 data at 0, unable to decode label
   -35> 2025-02-25T15:31:09.687+0000 7f6329c28c00 -1 bluestore(/dev/sda1) _read_bdev_label /dev/sda1 data at 40000000, unable to decode label
   -34> 2025-02-25T15:31:09.687+0000 7f6329c28c00 -1 bluestore(/dev/sda1) _read_bdev_label /dev/sda1 data at 280000000, unable to decode label
   -33> 2025-02-25T15:31:09.687+0000 7f6329c28c00 -1 bluestore(/dev/sda1) No valid bdev label found
   -28> 2025-02-25T15:31:09.727+0000 7f6329c28c00 -1 bluestore(/dev/sda14) _read_bdev_label /dev/sda14 data at 0, unable to decode label
   -27> 2025-02-25T15:31:09.727+0000 7f6329c28c00 -1 bluestore(/dev/sda14) No valid bdev label found
   -22> 2025-02-25T15:31:09.775+0000 7f6329c28c00 -1 bluestore(/dev/sda15) _read_bdev_label /dev/sda15 data at 0, unable to decode label
   -21> 2025-02-25T15:31:09.775+0000 7f6329c28c00 -1 bluestore(/dev/sda15) No valid bdev label found
   -16> 2025-02-25T15:31:09.811+0000 7f6329c28c00 -1 bluestore(/dev/sdb) _read_bdev_label /dev/sdb data at 0, unable to decode label
   -15> 2025-02-25T15:31:09.811+0000 7f6329c28c00 -1 bluestore(/dev/sdb) _read_bdev_label /dev/sdb data at 40000000, unable to decode label
   -14> 2025-02-25T15:31:09.811+0000 7f6329c28c00 -1 bluestore(/dev/sdb) _read_bdev_label /dev/sdb data at 280000000, unable to decode label
   -13> 2025-02-25T15:31:09.811+0000 7f6329c28c00 -1 bluestore(/dev/sdb) No valid bdev label found
    -4> 2025-02-25T15:31:09.923+0000 7f6329c28c00 -1 bluestore(/dev/nbd0) No valid bdev label found
    -2> 2025-02-25T15:31:09.923+0000 7f6329c28c00 -1 bdev is_valid_io 0~1000 block_size 1000 size 0
    -1> 2025-02-25T15:31:09.923+0000 7f6329c28c00 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-7819-g290894cc/rpm/el9/BUILD/ceph-19.3.0-7819-g290894cc/src/blk/kernel/KernelDevice.cc: In function 'virtual int KernelDevice::aio_read(uint64_t, uint64_t, ceph::bufferlist*, IOContext*)' thread 7f6329c28c00 time 2025
-02-25T15:31:09.927527+0000
/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-7819-g290894cc/rpm/el9/BUILD/ceph-19.3.0-7819-g290894cc/src/blk/kernel/KernelDevice.cc: 1411: FAILED ceph_assert(is_valid_io(off, len))
 
 ceph version 19.3.0-7819-g290894cc (290894cc762024be7774bdda6a02815a1e08555e) squid (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11e) [0x7f632adf1654]
 2: /usr/lib64/ceph/libceph-common.so.2(+0x183813) [0x7f632adf1813]
 3: (KernelDevice::aio_read(unsigned long, unsigned long, ceph::buffer::v15_2_0::list*, IOContext*)+0x99e) [0x55698e8a7d4e]
 4: (KernelDevice::_aio_stop()+0x143) [0x55698e89ad13]
 5: (KernelDevice::close()+0x1ad) [0x55698e89e34d]
 6: (BlueStore::read_bdev_label(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bluestore_bdev_label_t*, std::vector<unsigned long, std::allocator<unsigned long> >*, bool*, long*)+0xc2) [0x55698e732502]
 7: main()
 8: /lib64/libc.so.6(+0x295d0) [0x7f632a7655d0]
 9: __libc_start_main()
 10: _start()
 
     0> 2025-02-25T15:31:09.923+0000 7f6329c28c00 -1 *** Caught signal (Aborted) **
 in thread 7f6329c28c00 thread_name:ceph-bluestore-
 
 ceph version 19.3.0-7819-g290894cc (290894cc762024be7774bdda6a02815a1e08555e) squid (dev)
 1: /lib64/libc.so.6(+0x3ea60) [0x7f632a77aa60]
 2: /lib64/libc.so.6(+0x8c0fc) [0x7f632a7c80fc]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x178) [0x7f632adf16ae]
 6: /usr/lib64/ceph/libceph-common.so.2(+0x183813) [0x7f632adf1813]
 7: (KernelDevice::aio_read(unsigned long, unsigned long, ceph::buffer::v15_2_0::list*, IOContext*)+0x99e) [0x55698e8a7d4e]
 8: (KernelDevice::_aio_stop()+0x143) [0x55698e89ad13]
 9: (KernelDevice::close()+0x1ad) [0x55698e89e34d]
 10: (BlueStore::read_bdev_label(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bluestore_bdev_label_t*, std::vector<unsigned long, std::allocator<unsigned long> >*, bool*, long*)+0xc2) [0x55698e732502]
 11: main()
 12: /lib64/libc.so.6(+0x295d0) [0x7f632a7655d0]
 13: __libc_start_main()
 14: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
 
Aborted (core dumped)