gz-sim: Crash on M1 chip instantly even using docker

Environment

OS Version: macOS Monterey with Apple M1 chip using Docker (Ubuntu 20.04)
Source or binary build: Citadel binary build
If this is a GUI or sensor rendering bug, describe your GPU and rendering system. Otherwise delete this section.

GUI Bug

  - [ ] GUI rendering error. <!-- Search for message like "[GUI] [Msg] Loading plugin [ignition-rendering-ogre2] -->

Full log after launch gazebo such as ign gazebo --verbose log.txt

Ran ign gazebo -v 3 and it has the same as above. Ran ign gazebo -v 4 and it showed 4 extra blue line. Here log while it was inside middle log like above first log: log 2.txt

I tried to do it locally and this results the exact same as this issue

So that leaves me two options: Pull gazebo from gazebo website and run docker using feagi.yml

So, I ran feagi.yml following this steps:

git clone git@github.com:feagi/feagi.git
cd feagi/docker
docker-compose -f feagi.yml build --no-cache
Once it’s complete, docker-compose -f feagi.yml up
Open Browser (VNC)
Wait for it to complete colcon build so it can show you the gazebo. You should see the error instantly once Colon build ends.

The error looks exact same as log.txt (That’s actually where I copied all from)

So, that leaves me the last option.

docker pull gazebo (This is from gazebo’s docker hub)
docker run gazebo You can see the error instantly too.

The full log (I can’t get more than that)

station1@Station1s-Mini docker % docker run gazebo                                     
Error setting socket option (IP_MULTICAST_IF).
Error setting socket option (IP_MULTICAST_IF).
Did you set the environment variable IGN_IP with a correct IP address? 
  [172.17.0.2] seems an invalid local IP address.
  Using 127.0.0.1 as hostname.
terminate called after throwing an instance of 'std::out_of_range'
  what():  vector::_M_range_check: __n (which is 0) >= this->size() (which is 0)
qemu: uncaught target signal 6 (Aborted) - core dumped

So this seems like IGN_IP issue, GPU and ogre altogether.

- Generally, mention all circumstances that might affect rendering capabilities: <!-- remove lines that do not apply to keep the list short -->
  - [ ] running on real hardware
  - [ ] running in Docker/Singularity
- Rendering system info:
   Inside docker since it's using Ubuntu 20.04
  - On Linux, provide the outputs of the following commands:
  
      ```
        root@d45c22edad72:~# bash 
        root@d45c22edad72:~# LANG=C lspci -nn | grep VGA  # might require installing pciutils
        root@d45c22edad72:~# echo "$DISPLAY"
        :1.0
        root@d45c22edad72:~# LANG=C glxinfo -B | grep -i '\(direct rendering\|opengl\|profile\)'  # might require installing mesa-utils package
        direct rendering: Yes
            Preferred profile: core (0x1)
            Max core profile version: 4.5
            Max compat profile version: 3.1
            Max GLES1 profile version: 1.1
            Max GLES[23] profile version: 3.2
        OpenGL vendor string: Mesa/X.org
        OpenGL renderer string: llvmpipe (LLVM 11.0.0, 128 bits)
        OpenGL core profile version string: 4.5 (Core Profile) Mesa 20.2.6
        OpenGL core profile shading language version string: 4.50
        OpenGL core profile context flags: (none)
        OpenGL core profile profile mask: core profile
        OpenGL version string: 3.1 Mesa 20.2.6
        OpenGL shading language version string: 1.40
        OpenGL context flags: (none)
        OpenGL ES profile version string: OpenGL ES 3.2 Mesa 20.2.6
        OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20
        root@d45c22edad72:~# ps aux | grep Xorg
        root       922  0.0  0.3 151008  7212 pts/0    Sl+  17:38   0:00 /usr/bin/qemu-x86_64 /usr/bin/grep --color=auto Xorg
        root@d45c22edad72:~# sudo env LANG=C X -version
        /usr/bin/env: ‘X’: No such file or directory
        root@d45c22edad72:~# 
      ```

On Mac OS, open a terminal and type system_profiler SPDisplaysDataType. Copy the output here.

        station1@Station1s-Mini docker % system_profiler SPDisplaysDataType
     Graphics/Displays:

    Apple M1:

      Chipset Model: Apple M1
      Type: GPU
      Bus: Built-In
      Total Number of Cores: 8
      Vendor: Apple (0x106b)
      Metal Family: Supported, Metal GPUFamily Apple 7
      Displays:
        ASUS VP28U:
          Resolution: 5120 x 2880 (5K/UHD+ - Ultra High Definition Plus)
          UI Looks like: 2560 x 1440 @ 60.00Hz
          Main Display: Yes
          Mirror: Off
          Online: Yes
          Rotation: Supported
          Automatically Adjust Brightness: Yes
      station1@Station1s-Mini docker %

Description

Expected behavior: Just able to load gazebo. ign gazebo
Actual behavior: log.txt

Addition information: I ran --platform = linux/arm64 too or anything related to this command. Everything else worked except Gazebo.

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 1
Comments: 42 (15 by maintainers)

Most upvoted comments

It’s officially working, everyone!!!

See here on M1 where I spinned my robot:

https://user-images.githubusercontent.com/65916520/172218164-31af000e-38e4-4da0-9574-aad3cbe7db9f.mov

ON M1! It’s surprisingly faster and almost same speed as my local linux (it lags on docker on linux)

Here is the solution: I found out that Gazebo can run arm64 so I switched from vnc (amd64 by default) to arm64 and I swapped amd64 to arm64 on ros2 for the FOXY ros2/gazebo bridge.

Here is what I used for docker: https://hub.docker.com/r/dorowu/ubuntu-desktop-lxde-vnc/tags (I use dockerfile so it is FROM dorowu/ubuntu-desktop-lxde-vnc:focal-arm64)

That’s it!!!

Looks like amd64 is NOT supported on Gazebo/M1. Arm64 is available on M1 only.

I believe we can build Gazebo using arm64 only

Thank you, everyone! This helped me a ton! Hope it helps you a lot too, guys! See you around in Gazebo community!

Kakcalu13 on Jun 6, 2022

There was an no member name error in src/ign-launch/src/vendor/backward.hpp

oh looks like that’s also fixed upstream in the backward repo: https://github.com/bombela/backward-cpp/blob/master/backward.hpp#L4220-L4225.

iche033 on May 5, 2022

I was able to reproduce the issue with ign-physics. Could be due to recent changes in Garden.

This patch is probably not a proper fix but it got ign-physics to build on my mac M1 machine:
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 0645982..fc958e8 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -63,6 +63,8 @@ ign_find_package(EIGEN3 REQUIRED)
 ign_find_package(sdformat13
   REQUIRED_BY sdf dartsim tpe bullet)
 
+unset(TINYXML2_LIBRARIES)
+
 #--------------------------------------
 # Find dartsim for the dartsim plugin wrapper
 ign_find_package(DART

That worked!

I had more issues to be addressed though

There was an no member name error in src/ign-launch/src/vendor/backward.hpp

ign-garden-workspace/src/ign-launch/src/vendor/backward.hpp:3938:60: error: member reference type 'struct __darwin_mcontext64 *' is a pointer; did you mean to use '->'?
    error_addr = reinterpret_cast<void *>(uctx->uc_mcontext.pc);
                                          ~~~~~~~~~~~~~~~~~^
                                                           ->
ign-garden-workspace/src/ign-launch/src/vendor/backward.hpp:3938:61: error: no member named 'pc' in '__darwin_mcontext64'
    error_addr = reinterpret_cast<void *>(uctx->uc_mcontext.pc);
                                          ~~~~~~~~~~~~~~~~~ ^
2 errors generated.

I had to replace https://github.com/gazebosim/gz-launch/blob/8c78b6c937943ba1650e887c3a82dbfaea6c154f/src/vendor/backward.hpp#L3937

with

#elif defined(__aarch64__) && !defined(__APPLE__)
     error_addr = reinterpret_cast<void *>(uctx->uc_mcontext.pc);
#elif defined(__APPLE__) && defined(__aarch64__)
    error_addr = reinterpret_cast<void *>(uctx->uc_mcontext->__ss.__pc);

After the complete build the ignition gazebo could not find the libzmq.5.dylib.

Library error for [ign-garden-workspace/install/lib/libignition-gazebo7-ign.7.0.0~pre1.dylib]: dlopen(ign-garden-workspace/install/lib/libignition-gazebo7-ign.7.0.0~pre1.dylib, 0x0009): Library not loaded: @rpath/libzmq.5.dylib
  Referenced from: ign-garden-workspace/install/lib/libignition-transport12.12.0.0~pre1.dylib
  Reason: tried: '/usr/local/lib/libzmq.5.dylib' (no such file), '/usr/lib/libzmq.5.dylib' (no such file)

I had to create a symlink of libzeq.5.dylib @/usr/local/lib/ for that reason.

jasonbyun8 on May 3, 2022

I was able to reproduce the issue with ign-physics. Could be due to recent changes in Garden.

This patch is probably not a proper fix but it got ign-physics to build on my mac M1 machine:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 0645982..fc958e8 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -63,6 +63,8 @@ ign_find_package(EIGEN3 REQUIRED)
 ign_find_package(sdformat13
   REQUIRED_BY sdf dartsim tpe bullet)
 
+unset(TINYXML2_LIBRARIES)
+
 #--------------------------------------
 # Find dartsim for the dartsim plugin wrapper
 ign_find_package(DART

iche033 on May 3, 2022

@srmainwaring @iche033 I got the ogre2 example working with my ignition-rendering7 & m1 mac; Screen Shot 2022-05-01 at 1 25 55 AM I am unable to complete the colcon build though as the build of ign-physics6 fails.

  Command failed: 2

   '/Applications/Xcode.app/Contents/Developer/usr/bin/make'

  See also

    /Users/lapyx/ign-workspace/build/ignition-physics6/test/FAKE_INSTALL-prefix/src/FAKE_INSTALL-stamp/FAKE_INSTALL-build-*.log


make[2]: *** [test/FAKE_INSTALL-prefix/src/FAKE_INSTALL-stamp/FAKE_INSTALL-build] Error 1
make[1]: *** [test/CMakeFiles/FAKE_INSTALL.dir/all] Error 2
make: *** [all] Error 2
make: INTERNAL: Exiting with 12 jobserver tokens available; should be 10!
---
Failed   <<< ignition-physics6 [53.3s, exited with code 2]

Summary: 13 packages finished [7min 58s]
  1 package failed: ignition-physics6
  14 packages had stderr output: ignition-cmake3 ignition-common5 ignition-fuel_tools8 ignition-gui7 ignition-math7 ignition-msgs9 ignition-physics6 ignition-plugin2 ignition-rendering7 ignition-sensors7 ignition-tools2 ignition-transport12 ignition-utils2 sdformat13
  2 packages not processed

Have you experienced any issue like this before?

Below is the colcon graph I got from the latest garden yaml file, and the log files are also attached.

(base) ➜  ign-workspace colcon graph
ignition-cmake3       + **************
ignition-tools2        +      *   ..**
ignition-utils2         +**.*******.**
ignition-math7           +****.*******
ignition-common5          + ** *******
ignition-msgs9             +* *   ****
ignition-fuel_tools8        +       *.
ignition-plugin2             +  ***.**
ignition-transport12          +   ****
sdformat13                     +*  **.
ignition-physics6               +   *.
ignition-rendering7              +***.
ignition-gui7                     + **
ignition-sensors7                  +*.
ignition-gazebo7                    +*
ignition-launch6                     +

FAKE_INSTALL-build-err.log FAKE_INSTALL-build-out.log

As you can read from the file, the error complains ld: library not found for -lTINYXML2_LIBRARY_/opt/homebrew/Cellar/tinyxml2/9.0.0/lib/libtinyxml2.dylib-NOTFOUND although I do have libtinyxml2.dylib in /opt/homebrew/Cellar/tinyxml2/9.0.0/lib/.

Could this be related to disabling SIP?

jasonbyun8 on May 2, 2022

@Kakcalu13 can you try exporting the following before running the simple_demo:

export OGRE2_RESOURCE_PATH=/opt/homebrew/opt/ogre2.2/lib/OGRE-2.2/OGRE

(double check I’ve got that path right). The failure is macOS not resolving an rpath. It’s trying a couple of paths from CMAKE_PREFIX_PATH but none of them contain the correct location for the brew installed ogre2.2 libraries.

srmainwaring on Mar 29, 2022

I noticed that X-code isn’t installed by default on M1

You will need Xcode and the matching Command line tools (which need to be installed from the Apple developer downloads pages).

It looks like you managed to get ignition-rendering7 built. What was the outcome of building and running the examples simple_demo and ogre2demo?

srmainwaring on Mar 29, 2022

I think colcon should show some error msgs on why ign-gui failed.

Some differences I saw between your step 9 and mine are these libraries:

brew install qt@5 qwt-qt5 ffmpeg@4

The paths setup in step 10 looks the same as mine. I had one extra export which I’m not sure if it’s actually needed

export PKG_CONFIG_PATH=/opt/homebrew/opt/ffmpeg@4/lib/pkgconfig

iche033 on Mar 28, 2022

@Kakcalu13 not quite sure what the status of you build is. In your colcon workspace have you managed to build the main branch of ignition-rendering and dependencies using:

$ cd ~/ign_ws
$ colcon build --merge-install --cmake-args -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_MACOSX_RPATH=FALSE -DCMAKE_INSTALL_NAME_DIR=$(pwd)/install/lib

After calling

$ cd ~/ign_ws
$ source ./install/setup.zsh

you should then be able to run

$ cd ~/ign_ws/src/ign-rendering/examples/simple_demo
$ mkdir build && cd build
$ cmake ..
$ make -j4
$ ./simple_demo ogre2 metal

Your cmake log files appear to be reporting missing Xcode framework libraries etc. This can happen if you upgrade Xcode or the command line tools or brew and don’t do a proper clean of all of your build and install directories (i.e. the CMakeCache is stale and subsequent builds get corrupted).

srmainwaring on Mar 28, 2022

The key step is getting ignition-rendering (main branch) built and running the examples. Once you’re at that stage ping me and I can walk you through the rest of the set up.

srmainwaring on Mar 28, 2022

@Kakcalu13 and @brolin-botsync - most of the changes required to support Ignition on macOS using Metal have now been merged into the main branches of various repos. The outstanding PR for macOS is in ignition-gazebo (this PR runs fine on macOS but there are some Linux/OpenGL threading compatibility issues to resolve for GPU enabled sensors). I believe that the formula for Ogre2.2 in osrf/simulation is also now up to date for Metal and should work on a M1 machine as a couple of the devs reviewing the changes for Metal support were using these machines. Correction: there is an open PR for the M1 patch to the ogre2.2 formula

I should update or remove the wiki notes you refer to above - they have mostly been superseded now.

For Metal support you should use the Garden branch and will need to build from source.

I’d be happy to help you both get a native installation of ignition gazebo running on macOS - it would help figure out what the platform specific installation notes should look like. Starting point is to install Garden and test that the examples in ignition-rendering are working with the ogre2 metal flags (e.g. ign-rendering/examples/build/ogre2demo ogre2 metal). If ogre2 metal rendering is running then you should be able to get the full Gazebo environment working.

I have not had much luck running ignition using virtualised versions of Ubuntu with accelerated graphics post Ignition Edifice. The main problem is a lack of OpenGL 4.3+ support in the hypervisor drivers. For example VMWare fusion will only support OpenGL 4.1 and that is not sufficient for Ogre2.2. I have not looked at running docker machines with hardware acceleration for a while, but suspect you may run into a similar issue there. Software rendering may work, but that will be an order of magnitude slower for interesting rendering environments.

srmainwaring on Mar 28, 2022