rook: ARM64 OSD crash on v18.2.4
Is this a bug report or feature request?
- Bug Report
No errors in the logs the OSD pod crashes, specifically the osd container. My environment consists of both intel and arm osd nodes only the arm ones crash. Reverting back to v18.2.2 resolves the issue.
When reviewing the changes in v18.2.2 I noticed this oddity and wondering if maybe it is the source of the issue.
in “images/ceph/Makefile”
Image Build Options
ifeq ($(GOARCH),amd64)
CEPH_VERSION ?= v18.2.2-20240311
CEPH_VERSION ?= v18.2.4-20240723
else
CEPH_VERSION ?= v18.2.2-20240311
CEPH_VERSION ?= v18.2.4-20240724
endif
REGISTRY_NAME = quay.io
Is the timestamp version supposed to be 1 day later on the non-amd64 builds, previously that was not the case.
I do not have more details on the crash, I saw nothing in the describe other then just “Error” no other reasons given and the logs for the osd container never populated before the crash.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 5
- Comments: 26 (6 by maintainers)
Links to this issue
Commits related to this issue
- Use vfkit driver on darwin The new experimental vfkit driver[1] uses vfkit to start virtual machines using Apple virtualisation framework when running on Apple silicon. This driver is not available ... — committed to nirs/ramen by nirs 2 years ago
- Use vfkit driver on darwin The new experimental vfkit driver[1] uses vfkit to start virtual machines using Apple virtualisation framework when running on Apple silicon. This driver is not available ... — committed to nirs/ramen by nirs 2 years ago
- Use vfkit driver on darwin The new experimental vfkit driver[1] uses vfkit to start virtual machines using Apple virtualisation framework when running on Apple silicon. This driver is not available ... — committed to nirs/ramen by nirs 2 years ago
- Use vfkit driver on darwin The new experimental vfkit driver[1] uses vfkit to start virtual machines using Apple virtualisation framework when running on Apple silicon. This driver is not available ... — committed to nirs/ramen by nirs 2 years ago
After upgrading rook to 1.16.2 (now running ceph 19.2.0), I can confirm that there is no more issue with ARM64 OSD (tested on Turing PI RK1 / Rockchip RK3588)
same on a raspberry pi 4b. Edit: rolling back fixed the issue
I’m able to repro this issue with OSDs as well in minikube on my mac. It seems there is an issue with the ceph arm64 image, so we’ll need to open a ceph tracker for that. The rook operator is working fine, it’s just inside the osd prepare pod and osd daemon pod that we see the issue, which is purely running the ceph image. The Rook operator v1.14.9 is also able to successfully start v18.2.2 OSDs.
The image build options you were looking at are expected. Notice that in the amd64 and arm64 tags have a different date. The difference is just an artifact of how the build may span the date change.
In upgraded OSDs, I’m also seeing that there is no message of any form in the main osd container, it just fails.
For the new OSDs, the ceph_volume.log shows the failure of the
ceph-osd --osd-objectstore bluestore --mkfscommand. Here is a snippet, and also see the full log attached.See full log: ceph-volume.log
I can confirm that ceph 19.2.0 is fully functional on arm64 linux
The only thing I needed to do is faking the grafana container image as the one provided by the cephadm setup was amd64 only. But ceph cluster is up and running on #turingpi cluster.
FYI: just a x-post of my further findings from https://tracker.ceph.com/issues/67213#note-6 it seems to be, that
gperftools-libslinked and thus pulled from EPEL-9 is too old to fully support aarch64… “forcibly” upgrading this dependent lib to a more recent version apparently seems to fix the issuenot sure if it’s worth noting, but 18.2.7 still has this issue
@travisn Thanks!
I wasn’t aware of this issue before! now I have the correct issue to follow! Thanks! Sorry about cluttering up this issues comments.
FYI: I’ve got exactly the same issue on turingpi rk1 compute module (Rockchip RK3588) I confirm that the workaround using v18.2.2 instead of v18.2.4 is working ok : https://gitlab.com/grunlab/rook-ceph/-/blob/1.15.4/deploy/patches/cephcluster_rook-ceph.yml?ref_type=tags#L8