rmw_zenoh: Mismatching ros-jazzy-rmw-zenoh-cpp apt versions between amd64 and arm64 causing rmw failure

Operating System:

amd64 vs arm64

ROS version or commit hash:

jazzy

RMW implementation (if applicable):

rmw_zenoh_cpp

RMW Configuration (if applicable):

     Host 1                           Host 2
rviz (client mode) ------ zenohd (router mode) + nav2 (peer mode)

Host 1 (dev pc):

RMW_IMPLEMENTATION=rmw_zenoh_cpp
ZENOH_CONFIG_OVERRIDE='connect/endpoints=["tcp/<ROBOT_IP>:7447"];mode="client"'

Host 2 (robot):
RMW_IMPLEMENTATION=rmw_zenoh_cpp

Client library (if applicable):

rclcpp

‘ros2 doctor --report’ output

Steps to reproduce issue

apt-cache madison ros-jazzy-rmw-zenoh-cpp outputs:

  • amd64:
    • ros-jazzy-rmw-zenoh-cpp | 0.2.3-1noble.20250429.203124 | http://packages.ros.org/ros2/ubuntu noble/main amd64 Packages
  • arm64:
    • ros-jazzy-rmw-zenoh-cpp | 0.2.4-1noble.20250424.134556 | http://packages.ros.org/ros2/ubuntu noble/main arm64 Packages

Expected behavior

To be able to debug and visualize data coming from the robot on my dev pc without crashing.

Actual behavior

On my amd64 dev pc with 0.2.3-1noble.20250429.203124 version:

[rviz2-1] 2025-05-14T08:34:38.426841Z ERROR rx-0 ThreadId(04) zenohc::serialization: Failed to deserialize the payload: deserialization error
[rviz2-1] thread 'rx-0' panicked at library/core/src/panicking.rs:136:5:
[rviz2-1] panic in a function that cannot unwind
[rviz2-1] stack backtrace:
[rviz2-1]    0:     0x7a21199cdbcc - <unknown>
[rviz2-1]    1:     0x7a2119892b80 - <unknown>
[rviz2-1]    2:     0x7a21199c53a1 - <unknown>
[rviz2-1]    3:     0x7a21199cd96e - <unknown>
[rviz2-1]    4:     0x7a21199eafd0 - <unknown>
[rviz2-1]    5:     0x7a21199ebb20 - <unknown>
[rviz2-1]    6:     0x7a21199ce138 - <unknown>
[rviz2-1]    7:     0x7a21199ce0c6 - <unknown>
[rviz2-1]    8:     0x7a21199eb351 - <unknown>
[rviz2-1]    9:     0x7a21196bddb7 - <unknown>
[rviz2-1]   10:     0x7a21196bde23 - <unknown>
[rviz2-1]   11:     0x7a21196bddc6 - <unknown>
[rviz2-1]   12:     0x7a21199c7145 - <unknown>
[rviz2-1]   13:     0x7a212049caa4 - start_thread
[rviz2-1]                                at ./nptl/pthread_create.c:447:8
[rviz2-1]   14:     0x7a2120529c3c - __GI___clone3
[rviz2-1]                                at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
[rviz2-1]   15:                0x0 - <unknown>
[rviz2-1] thread caused non-unwinding panic. aborting.

On my arm64 robot with 0.2.4-1noble.20250424.134556 version(this happens to all the nodes):

[gps_driver-3] #13   Source "../sysdeps/unix/sysv/linux/aarch64/clone3.S", line 76, in thread_start [0xffffb1bfba4b]
[gps_driver-3] #12   Source "./nptl/pthread_create.c", line 447, in start_thread [0xffffb1b9595b]
[gps_driver-3] #11   Object "/opt/ros/jazzy/opt/zenoh_cpp_vendor/lib/libzenohc.so", at 0xffffaf88c723, in 
[gps_driver-3] #10   Object "/opt/ros/jazzy/opt/zenoh_cpp_vendor/lib/libzenohc.so", at 0xffffaf5a2d33, in 
[gps_driver-3] #9    Object "/opt/ros/jazzy/opt/zenoh_cpp_vendor/lib/libzenohc.so", at 0xffffaf5a2d7f, in 
[gps_driver-3] #8    Object "/opt/ros/jazzy/opt/zenoh_cpp_vendor/lib/libzenohc.so", at 0xffffaf5a2d1f, in 
[gps_driver-3] #7    Object "/opt/ros/jazzy/opt/zenoh_cpp_vendor/lib/libzenohc.so", at 0xffffaf86f4cf, in 
[gps_driver-3] #6    Object "/opt/ros/jazzy/opt/zenoh_cpp_vendor/lib/libzenohc.so", at 0xffffaf86d443, in 
[gps_driver-3] #5    Object "/opt/ros/jazzy/opt/zenoh_cpp_vendor/lib/libzenohc.so", at 0xffffaf86d4af, in 
[gps_driver-3] #4    Object "/opt/ros/jazzy/opt/zenoh_cpp_vendor/lib/libzenohc.so", at 0xffffaf86f907, in 
[gps_driver-3] #3    Object "/opt/ros/jazzy/opt/zenoh_cpp_vendor/lib/libzenohc.so", at 0xffffaf88ba87, in 
[gps_driver-3] #2    Source "./stdlib/abort.c", line 79, in abort [0xffffb1b37dff]
[gps_driver-3] #1    Source "../sysdeps/posix/raise.c", line 26, in raise [0xffffb1b4cb3b]
[gps_driver-3] #0    Source "./nptl/pthread_kill.c", line 44, in __pthread_kill_implementation [0xffffb1b97608]
[gps_driver-3] Aborted (Signal sent by tkill() 58 0)

Additional information

My robot runs on a jetson with arm64 while my dev pc runs on a x86_64 cpu. I have the latest apt releases on both. Normally, I use the following config to debug my robot:

     Host 1                           Host 2
rviz (client mode) ------ zenohd (router mode) + nav2 (peer mode)

Host 1 (dev pc):

RMW_IMPLEMENTATION=rmw_zenoh_cpp
ZENOH_CONFIG_OVERRIDE='connect/endpoints=["tcp/<ROBOT_IP>:7447"];mode="client"'

Host 2 (robot):
RMW_IMPLEMENTATION=rmw_zenoh_cpp

However, today I noticed that as soon as I try to run anything on my dev pc, rmw crashes on the robot. Upon furthur investigation I noticed the mismatch between the apt releases which I suspect might be the issue.

I am not 100% certain that this is happening due to the mismatch, but considering that this was not an issue up just a few days ago and it happens as soon as I run something on my dev pc (trying to connect to the zenoh router on the robot), I suspect that it might be due to the version differences.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 3
  • Comments: 20 (6 by maintainers)

Most upvoted comments

To summarize,

We made the decision to break write compatibility for an already released distribution like Jazzy as we felt the benefits from reducing the bandwidth were worth it, especially to enable communication with memory constrained devices like micro-controllers over zenoh-pico. This breaking change happens when migrating to rmw_zenoh 0.2.4 from 0.2.3 on Jazzy.

Moving forward, we will be extremely careful to not introduce breaking changes into a stable ROS distro.

Now about the state of rmw_zenoh binaries on Jazzy. We ended up reverting the bump to 0.2.4 for reasons explained in https://github.com/ros/rosdistro/pull/45556.

However due to some unfortunate timing between the revert and when the last sync was performed, rmw_zenoh binaries for Jazzy on arm64 and amd64 platforms ended up with different versions. This is a freak accident.

Once the sync announced this week is completed, the version discrepancy should hopefully go away.

Towards the end of this month, as the opencv challenge wraps up, we will bump rmw_zenoh binaries on Jazzy to the newer version WITH the wire breaking changes. Prior to this, we will make an announcement on discourse to give a heads up.

We would appreciate everyone’s understanding as we get over this short term inconvenience 😃

Hi @Yadunund , we are experiencing the same issue with communication between our Jetson and amd64 machine, both running Ubuntu 24.04 jazzy.

On the Jetson, we have built rmw_zenoh_cpp from source (and sourced the workspace, of course), but the machines still cannot communicate - the errors are the same as posted above.

Am I missing something else?

Thanks!

@alireza-moayyedi thanks for flagging this issue. It is indeed surprising that the version on amd64 and arm64 are different - sorry about that.

It looks like the changes from https://github.com/ros/rosdistro/pull/45556 didn’t make it to the arm64 runners on the buildfarm before the sync was performed.

I see that the downgrade to 0.2.3 is pending on arm64: https://repo.ros2.org/status_page/ros_jazzy_unv8.html?q=REGRESSION This should be fixed if @marcoag does another Jazzy sync.

In the meanwhile, on the arm64 host you could switch to the ros2-testing repo to pull binaries by following instructions here after which running sudo apt update && sudo apt install rmw-zenoh-cpp should install 0.2.3 binary on amr64. But this might have consequences as if you run an apt upgrade, it will install latest bianries of other jazzy packages too that have not yet been released. Building rmw_zenoh from source on the arm64 host might be the better way to go.

Just for the record, upgrading the package using apt doesn’t work in this case because apt doesn’t consider the lower version number an upgrade. Similarly, reinstalling the package won’t downgrade it either. To fix this, we need to remove the package and then install it again so that apt can recognize the correct version number.

sudo apt remove ros-jazzy-rmw-zenoh-cpp
sudo apt install ros-jazzy-rmw-zenoh-cpp

The next Jazzy sync was already announced. One it is completed, rmw_zenoh will be 0.2.3 on arm64 and amd64.

we are experiencing the same issue with communication between our Jetson and amd64 machine, both running Ubuntu 24.04 jazzy. On the Jetson, we have built rmw_zenoh_cpp from source (and sourced the workspace, of course), but the machines still cannot communicate - the errors are the same as posted above.

Hi @JoLichtenfeld, If you build from sources on Jetson you get a version >0.2.4 which includes the change in the attachment. If on amd64 you installed from ROS repositories, you get get a version 0.2.3 which doesn’t include this change, and thus can’t interoperate with your Jetson.

For the time being, you need to:

  • either on amd64 to build from sources also
  • either on Jetson to build from sources but from the 0.2.3 tag

Hi @alireza-moayyedi,

This issue is a consequence of #601 which changed the serialization format of attachments (timestamp, counter and publisher id). It has been merged into rolling just before the code freeze for Kilted. It has also been merged into jazzy and bloomed in version 0.2.4. Therefore it breaks interoperability with 0.2.3. However it has not yet been synced for the reason detailed in https://github.com/ros/rosdistro/pull/45556

~So I guess somehow your apt-cache madison command gets the 0.2.4 from the testing repository.~

We are truly sorry for the inconvenience this change in the jazzy release has caused. We will be more cautious in the future to maintain backward compatibility within the same release.


Edit: actually the 0.2.4 package is well present in the final repository, but only for arm64 ! I don’t know how this could have happened following the revert on https://github.com/ros/rosdistro/pull/45556 . @Yadunund , @marcoag ?