rook: ARM64 OSD crash on v18.2.4

Is this a bug report or feature request?

  • Bug Report

No errors in the logs the OSD pod crashes, specifically the osd container. My environment consists of both intel and arm osd nodes only the arm ones crash. Reverting back to v18.2.2 resolves the issue.

When reviewing the changes in v18.2.2 I noticed this oddity and wondering if maybe it is the source of the issue.

in “images/ceph/Makefile”

Image Build Options

ifeq ($(GOARCH),amd64)
CEPH_VERSION ?= v18.2.2-20240311
CEPH_VERSION ?= v18.2.4-20240723
else
CEPH_VERSION ?= v18.2.2-20240311
CEPH_VERSION ?= v18.2.4-20240724
endif
REGISTRY_NAME = quay.io

Is the timestamp version supposed to be 1 day later on the non-amd64 builds, previously that was not the case.

I do not have more details on the crash, I saw nothing in the describe other then just “Error” no other reasons given and the logs for the osd container never populated before the crash.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 5
  • Comments: 26 (6 by maintainers)

Commits related to this issue

Most upvoted comments

After upgrading rook to 1.16.2 (now running ceph 19.2.0), I can confirm that there is no more issue with ARM64 OSD (tested on Turing PI RK1 / Rockchip RK3588)

same on a raspberry pi 4b. Edit: rolling back fixed the issue

I’m able to repro this issue with OSDs as well in minikube on my mac. It seems there is an issue with the ceph arm64 image, so we’ll need to open a ceph tracker for that. The rook operator is working fine, it’s just inside the osd prepare pod and osd daemon pod that we see the issue, which is purely running the ceph image. The Rook operator v1.14.9 is also able to successfully start v18.2.2 OSDs.

The image build options you were looking at are expected. Notice that in the amd64 and arm64 tags have a different date. The difference is just an artifact of how the build may span the date change.

In upgraded OSDs, I’m also seeing that there is no message of any form in the main osd container, it just fails.

For the new OSDs, the ceph_volume.log shows the failure of the ceph-osd --osd-objectstore bluestore --mkfs command. Here is a snippet, and also see the full log attached.

[2024-07-26 15:13:28,876][ceph_volume.main][INFO  ] Running command: ceph-volume  raw prepare --bluestore --data /dev/vdb --crush-device-class hdd
...
[2024-07-26 15:13:29,645][ceph_volume.util.prepare][INFO  ] Creating keyring file for osd.0
[2024-07-26 15:13:29,645][ceph_volume.process][INFO  ] Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-0/keyring --create-keyring --name osd.0 --add-key AQAZvaNmFEGlAhAASMGN2pvV5vMwk8u/L5i/sA==
[2024-07-26 15:13:29,657][ceph_volume.process][INFO  ] Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
[2024-07-26 15:13:29,666][ceph_volume.process][INFO  ] Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
[2024-07-26 15:13:29,676][ceph_volume.process][INFO  ] Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /v
[2024-07-26 15:13:29,692][ceph_volume.devices.raw.prepare][ERROR ] raw prepare was unable to complete
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/ceph_volume/devices/raw/prepare.py", line 80, in safe_prepare
    self.prepare()
  File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/usr/lib/python3.9/site-packages/ceph_volume/devices/raw/prepare.py", line 118, in prepare
    prepare_bluestore(
  File "/usr/lib/python3.9/site-packages/ceph_volume/devices/raw/prepare.py", line 53, in prepare_bluestore
    prepare_utils.osd_mkfs_bluestore(
  File "/usr/lib/python3.9/site-packages/ceph_volume/util/prepare.py", line 459, in osd_mkfs_bluestore
    raise RuntimeError('Command failed with exit code %s: %s' % (returncode, ' '.join(command)))
RuntimeError: Command failed with exit code -11: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/
[2024-07-26 15:13:29,693][ceph_volume.devices.raw.prepare][INFO  ] will rollback OSD ID creation
[2024-07-26 15:13:29,693][ceph_volume.process][INFO  ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-
[2024-07-26 15:13:29,869][ceph_volume.process][INFO  ] stderr purged osd.0
[2024-07-26 15:13:29,884][ceph_volume.process][INFO  ] Running command: /usr/bin/systemctl is-active ceph-osd@0
[2024-07-26 15:13:29,887][ceph_volume.process][INFO  ] stderr System has not been booted with systemd as init system (PID 1). Can't operate.
[2024-07-26 15:13:29,887][ceph_volume.process][INFO  ] stderr Failed to connect to bus: Host is down
[2024-07-26 15:13:29,888][ceph_volume.util.system][INFO  ] Executable lvs found on the host, will use /sbin/lvs
[2024-07-26 15:13:29,888][ceph_volume.process][INFO  ] Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvs --noheadings --r
[2024-07-26 15:13:29,920][ceph_volume][ERROR ] exception caught by decorator
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/ceph_volume/devices/raw/prepare.py", line 80, in safe_prepare
    self.prepare()
  File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/usr/lib/python3.9/site-packages/ceph_volume/devices/raw/prepare.py", line 118, in prepare
    prepare_bluestore(
  File "/usr/lib/python3.9/site-packages/ceph_volume/devices/raw/prepare.py", line 53, in prepare_bluestore
    prepare_utils.osd_mkfs_bluestore(
  File "/usr/lib/python3.9/site-packages/ceph_volume/util/prepare.py", line 459, in osd_mkfs_bluestore
    raise RuntimeError('Command failed with exit code %s: %s' % (returncode, ' '.join(command)))
RuntimeError: Command failed with exit code -11: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 59, in newfunc
    return f(*a, **kw)
  File "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 153, in main
    terminal.dispatch(self.mapper, subcommand_args)
  File "/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in dispatch
    instance.main()
  File "/usr/lib/python3.9/site-packages/ceph_volume/devices/raw/main.py", line 32, in main
    terminal.dispatch(self.mapper, self.argv)
  File "/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in dispatch
    instance.main()
  File "/usr/lib/python3.9/site-packages/ceph_volume/devices/raw/prepare.py", line 160, in main
    self.safe_prepare(self.args)
  File "/usr/lib/python3.9/site-packages/ceph_volume/devices/raw/prepare.py", line 84, in safe_prepare
    rollback_osd(self.args, self.osd_id)
  File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/common.py", line 35, in rollback_osd
    Zap(['--destroy', '--osd-id', osd_id]).main()
  File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/zap.py", line 407, in main
    self.zap_osd()
  File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/zap.py", line 305, in zap_osd
    devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid)
  File "/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices
    raise RuntimeError('Unable to find any LV for zapping OSD: '
RuntimeError: Unable to find any LV for zapping OSD: 0

See full log: ceph-volume.log

I can confirm that ceph 19.2.0 is fully functional on arm64 linux

Using ceph image with id 'fd3234b9d664' and tag 'v19.2.0' created on 2024-09-27 23:52:09 +0200 CEST
quay.io/ceph/ceph@sha256:200087c35811bf28e8a8073b15fa86c07cce85c575f1ccd62d1d6ddbfdc6770a
ceph version 19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable)
➜  ~ uname -a
Linux rk01 5.10.160-rockchip #37 SMP Fri Apr 26 05:16:30 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

The only thing I needed to do is faking the grafana container image as the one provided by the cephadm setup was amd64 only. But ceph cluster is up and running on #turingpi cluster.

FYI: just a x-post of my further findings from https://tracker.ceph.com/issues/67213#note-6 it seems to be, that gperftools-libs linked and thus pulled from EPEL-9 is too old to fully support aarch64… “forcibly” upgrading this dependent lib to a more recent version apparently seems to fix the issue

not sure if it’s worth noting, but 18.2.7 still has this issue

@travisn Thanks!

Updated by Travis Nielsen 3 months ago

@rongchi sun What is the fix that is needed?

We need to make sure this is fixed in the next Reef release since v18.2.4 is broken for arm. While the issue does not repro in v19, some users still require Reef.

I wasn’t aware of this issue before! now I have the correct issue to follow! Thanks! Sorry about cluttering up this issues comments.

FYI: I’ve got exactly the same issue on turingpi rk1 compute module (Rockchip RK3588) I confirm that the workaround using v18.2.2 instead of v18.2.4 is working ok : https://gitlab.com/grunlab/rook-ceph/-/blob/1.15.4/deploy/patches/cephcluster_rook-ceph.yml?ref_type=tags#L8