rook: Ceph-main tests failing to create OSDs on daily CI smoke suite
This bug seems similar to https://github.com/rook/rook/issues/15282
Is this a bug report or feature request?
- Bug Report
Deviation from expected behavior: The smoke suite is failing when run against the ceph-main in the daily CI. For example, see this failure
This failure started failing on Feb 14th. The operator and osd logs can be found in the artifacts under ceph-smoke-suite-squid-artifact.
The OSD prepare log shows that the OSD is failing to be provisioned by c-v with the following error:
2025-02-14 00:30:01.763996 D | exec: Running command: stdbuf -oL ceph-volume raw prepare --bluestore --data /dev/sdb1 --crush-device-class hdd
2025-02-14 00:30:04.275150 I | cephosd: Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 16b4711c-9ac3-4124-ab9a-9404e76ad31b
Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
Running command: /usr/bin/chown -R ceph:ceph /dev/sdb1
Running command: /usr/bin/ln -s /dev/sdb1 /var/lib/ceph/osd/ceph-0/block
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap
stderr: got monmap epoch 3
--> Creating keyring file for osd.0
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 16b4711c-9ac3-4124-ab9a-9404e76ad31b --setuser ceph --setgroup ceph
stderr: 2025-02-14T00:30:03.219+0000 7f447e63f8c0 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label /var/lib/ceph/osd/ceph-0//block data at 0, unable to decode label
stderr: 2025-02-14T00:30:03.219+0000 7f447e63f8c0 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label /var/lib/ceph/osd/ceph-0//block data at 40000000, unable to decode label
stderr: 2025-02-14T00:30:03.219+0000 7f447e63f8c0 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label /var/lib/ceph/osd/ceph-0//block data at 280000000, unable to decode label
stderr: 2025-02-14T00:30:03.219+0000 7f447e63f8c0 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) No valid bdev label found
stderr: 2025-02-14T00:30:03.275+0000 7f447e63f8c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
--> ceph-volume raw dmcrypt prepare successful for: /dev/sdb1
2025-02-14 00:30:04.275439 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log lvm list --format json
2025-02-14 00:30:04.665029 D | cephosd: {}
2025-02-14 00:30:04.665060 I | cephosd: 0 ceph-volume lvm osd devices configured on this node
2025-02-14 00:30:04.665081 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log raw list --format json
2025-02-14 00:30:09.511047 D | cephosd: {}
2025-02-14 00:30:09.511071 I | cephosd: 0 ceph-volume raw osd devices configured on this node
2025-02-14 00:30:09.511079 W | cephosd: skipping OSD configuration as no devices matched the storage settings for this node "test-prefix-this-is-a-very-long-hostname-fv-az774-68"
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 32 (11 by maintainers)
https://github.com/ceph/ceph/pull/62009 just merged. It still did not pass teuthology, so some extra fixes might be needed in future, but it should unblock rook.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.
And ramen, thank you for this fix!
Fix: https://github.com/ceph/ceph/pull/62009 Due to urgency, it will pass make check, check if shaman build properly and merged skipping teuthology. ETA: 4h
@aclamk @travisn
ceph-volume raw listimplies a call toceph-bluestore-toollike the following in the case of rook CI:when running that command in a rook environment we can see it crashes like this (sorry for the long output):