PX4-Autopilot: Slowing real time factor in SITL Gazebo

Hello,

I am doing a 3 IRIS quadcopters simulation in gazebo. I have noticed that after running simulation for 4 days at low quadcopters speed (1 m/s), and after updating to the last SITL commit in which iris SDF file has been update with new plugins for magnometers and gyroscope, that gazebo is slowing down, the real time factor is going down to 0.22 Also the communication between the SDK and the PX4 is slowing. I am even losing heart beats during take off and landing. The output from the SDK is the following :

11:06:27|Debug] MAVLink: info: DISARMED by auto disarm on land (system_impl.cpp:300)
[11:06:28|Info ] heartbeats timed out (system_impl.cpp:305)
[11:06:28|Debug] Lost 5283920058631409232 (mavsdk_impl.cpp:388)
[11:06:29|Info ] heartbeats timed out (system_impl.cpp:305)
[11:06:29|Info ] heartbeats timed out (system_impl.cpp:305)
[11:06:29|Debug] Lost 5283920058631409233 (mavsdk_impl.cpp:388)
[11:06:29|Debug] Lost 5283920058631409234 (mavsdk_impl.cpp:388)
[11:06:29|Debug] Discovered 1 component(s) (UUID: 5283920058631409232) (system_impl.cpp:557)
[11:06:29|Info ] Discovered system with UUID: 5283920058631409232
[11:06:31|Debug] Discovered 1 component(s) (UUID: 5283920058631409233) (system_impl.cpp:557)
[11:06:31|Info ] Discovered system with UUID: 5283920058631409233
[11:06:31|Debug] Discovered 1 component(s) (UUID: 5283920058631409234) (system_impl.cpp:557)
[11:06:31|Info ] Discovered system with UUID: 5283920058631409234
[11:06:32|Info ] heartbeats timed out (system_impl.cpp:305)
[11:06:32|Debug] Lost 5283920058631409232 (mavsdk_impl.cpp:388)
[11:06:34|Info ] heartbeats timed out (system_impl.cpp:305)
[11:06:34|Debug] Lost 5283920058631409233 (mavsdk_impl.cpp:388)
[11:06:34|Info ] heartbeats timed out (system_impl.cpp:305)
[11:06:34|Debug] Lost 5283920058631409234 (mavsdk_impl.cpp:388)
[11:06:34|Debug] Discovered 1 component(s) (UUID: 5283920058631409232) (system_impl.cpp:557)
[11:06:34|Info ] Discovered system with UUID: 5283920058631409232
[11:06:35|Debug] Discovered 1 component(s) (UUID: 5283920058631409233) (system_impl.cpp:557)
[11:06:35|Info ] Discovered system with UUID: 5283920058631409233
[11:06:35|Debug] Discovered 1 component(s) (UUID: 5283920058631409234) (system_impl.cpp:557)
[11:06:35|Info ] Discovered system with UUID: 5283920058631409234

I am using Arch Linux with 2 Intel® Xeon® CPU E5-2609 v4 @ 1.70GHz with 16 cores and 32 GB of RAM.

Also using htop, I have noticed that gzserver is using only 70% of only one core of CPU, there is no over consumption at any level, even though it is a little bit reduced over the four days

Does any one have an idea about what is happening??

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 36 (17 by maintainers)

Most upvoted comments

I think this issue is caused by a part of code below. src/gazebo_magnetometer_plugin.cpp mag_message_.add_magnetic_field_covariance() would be called when world update start event occurs (This event is called frequentry). So mag_message_'s contents keep growing. This part should be written in Load event, if its contents is constant. But I’m not sure if it’s constant.

@julianoes Thanks for merging, this will help me a lot

I’m still on Gazebo 10 😄. Looks like I should not upgrade! 😕

@julianoes, Well after an enormous update for my arch system. Everything needs to be rebuild. Since I have to rebuild the binaries for px4. I have to relook into it since the last working commit before this issue can not be compiled because of a broken python dependecy So, yes, I have to relook into it today.

@julianoes, It does happen with default gazebo, I only do several quadcopters vehicles simulation in gazebo. Never tried only one vehicles. I have no idea if leaving the quadcopters on the ground might reduce or not the real time factor. I will try to have more information and post it here