turtlebot4: Inconsistent Navigation on Existing Maps
Robot Model
Turtlebot4 Standard
ROS distro
Humble
Networking Configuration
Discovery Server
OS
Ubuntu 22.04
Built from source or installed?
Installed
Package version
turtlebot4_navigation version 1.0.5-1jammy.20250114.201201
All packages on both remote PC and Raspberry Pi fully-updated at time of opening this issue.
Type of issue
Navigation (SLAM, Nav2 etc.)
Expected behaviour
The robot to be able to autonomously navigate to goals provied in RViz using a pre-existing map of the environment.
Actual behaviour
This behaviour is very inconsistent. Very rarely, the everything works as expected. However typically something crashes when launching nav2.launch.py. An example error log is shown below.
The error is not always the same, but it is typically lifecycle_manager being unable to reach either controller_server or smoother_server after 4 seconds by bond. In the below instance, it was controller_server that cannot be reached.
Error messages
[INFO] [launch]: All log files can be found below /home/ed/.ros/log/2025-02-10-11-33-39-912698-Unhygienix-24041
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [controller_server-1]: process started with pid [24042]
[INFO] [smoother_server-2]: process started with pid [24044]
[INFO] [planner_server-3]: process started with pid [24046]
[INFO] [behavior_server-4]: process started with pid [24048]
[INFO] [bt_navigator-5]: process started with pid [24050]
[INFO] [waypoint_follower-6]: process started with pid [24052]
[INFO] [velocity_smoother-7]: process started with pid [24054]
[INFO] [lifecycle_manager-8]: process started with pid [24056]
[velocity_smoother-7] [INFO] [1739147620.022728425] [tb4.velocity_smoother]:
[velocity_smoother-7] velocity_smoother lifecycle node launched.
[velocity_smoother-7] Waiting on external lifecycle transitions to activate
[velocity_smoother-7] See https://design.ros2.org/articles/node_lifecycle.html for more information.
[waypoint_follower-6] [INFO] [1739147620.026065141] [tb4.waypoint_follower]:
[waypoint_follower-6] waypoint_follower lifecycle node launched.
[waypoint_follower-6] Waiting on external lifecycle transitions to activate
[waypoint_follower-6] See https://design.ros2.org/articles/node_lifecycle.html for more information.
[waypoint_follower-6] [INFO] [1739147620.026408458] [tb4.waypoint_follower]: Creating
[lifecycle_manager-8] [INFO] [1739147620.026702640] [tb4.lifecycle_manager_navigation]: Creating
[bt_navigator-5] [INFO] [1739147620.027269513] [tb4.bt_navigator]:
[bt_navigator-5] bt_navigator lifecycle node launched.
[bt_navigator-5] Waiting on external lifecycle transitions to activate
[bt_navigator-5] See https://design.ros2.org/articles/node_lifecycle.html for more information.
[bt_navigator-5] [INFO] [1739147620.027338303] [tb4.bt_navigator]: Creating
[lifecycle_manager-8] [INFO] [1739147620.027507127] [tb4.lifecycle_manager_navigation]: Creating and initializing lifecycle service clients
[controller_server-1] [INFO] [1739147620.029482791] [tb4.controller_server]:
[controller_server-1] controller_server lifecycle node launched.
[controller_server-1] Waiting on external lifecycle transitions to activate
[controller_server-1] See https://design.ros2.org/articles/node_lifecycle.html for more information.
[behavior_server-4] [INFO] [1739147620.029882550] [tb4.behavior_server]:
[behavior_server-4] behavior_server lifecycle node launched.
[behavior_server-4] Waiting on external lifecycle transitions to activate
[behavior_server-4] See https://design.ros2.org/articles/node_lifecycle.html for more information.
[controller_server-1] [INFO] [1739147620.031244657] [tb4.controller_server]: Creating controller server
[controller_server-1] [INFO] [1739147620.033090280] [tb4.local_costmap.local_costmap]:
[controller_server-1] local_costmap lifecycle node launched.
[controller_server-1] Waiting on external lifecycle transitions to activate
[controller_server-1] See https://design.ros2.org/articles/node_lifecycle.html for more information.
[smoother_server-2] [INFO] [1739147620.033249216] [tb4.smoother_server]:
[smoother_server-2] smoother_server lifecycle node launched.
[smoother_server-2] Waiting on external lifecycle transitions to activate
[smoother_server-2] See https://design.ros2.org/articles/node_lifecycle.html for more information.
[controller_server-1] [INFO] [1739147620.033317454] [tb4.local_costmap.local_costmap]: Creating Costmap
[smoother_server-2] [INFO] [1739147620.033950263] [tb4.smoother_server]: Creating smoother server
[planner_server-3] [INFO] [1739147620.036680856] [tb4.planner_server]:
[planner_server-3] planner_server lifecycle node launched.
[planner_server-3] Waiting on external lifecycle transitions to activate
[planner_server-3] See https://design.ros2.org/articles/node_lifecycle.html for more information.
[planner_server-3] [INFO] [1739147620.037522448] [tb4.planner_server]: Creating
[planner_server-3] [INFO] [1739147620.039236625] [tb4.global_costmap.global_costmap]:
[planner_server-3] global_costmap lifecycle node launched.
[planner_server-3] Waiting on external lifecycle transitions to activate
[planner_server-3] See https://design.ros2.org/articles/node_lifecycle.html for more information.
[planner_server-3] [INFO] [1739147620.039439076] [tb4.global_costmap.global_costmap]: Creating Costmap
[lifecycle_manager-8] [INFO] [1739147622.027983201] [tb4.lifecycle_manager_navigation]: Waiting for service controller_server/get_state...
[lifecycle_manager-8] [INFO] [1739147624.028178826] [tb4.lifecycle_manager_navigation]: Waiting for service controller_server/get_state...
[lifecycle_manager-8] [INFO] [1739147626.028357785] [tb4.lifecycle_manager_navigation]: Waiting for service controller_server/get_state...
[lifecycle_manager-8] [INFO] [1739147628.028672616] [tb4.lifecycle_manager_navigation]: Waiting for service controller_server/get_state...
[lifecycle_manager-8] [INFO] [1739147630.028848347] [tb4.lifecycle_manager_navigation]: Waiting for service controller_server/get_state...
[lifecycle_manager-8] [INFO] [1739147632.029022374] [tb4.lifecycle_manager_navigation]: Waiting for service controller_server/get_state...
[lifecycle_manager-8] [INFO] [1739147634.029209576] [tb4.lifecycle_manager_navigation]: Waiting for service controller_server/get_state...
[lifecycle_manager-8] [INFO] [1739147636.029453824] [tb4.lifecycle_manager_navigation]: Waiting for service controller_server/get_state...
[lifecycle_manager-8] [INFO] [1739147638.029705882] [tb4.lifecycle_manager_navigation]: Waiting for service controller_server/get_state...
[lifecycle_manager-8] [INFO] [1739147640.718887739] [tb4.lifecycle_manager_navigation]: Waiting for service smoother_server/get_state...
[lifecycle_manager-8] [INFO] [1739147642.719141579] [tb4.lifecycle_manager_navigation]: Waiting for service smoother_server/get_state...
[lifecycle_manager-8] [INFO] [1739147644.719448955] [tb4.lifecycle_manager_navigation]: Waiting for service smoother_server/get_state...
[lifecycle_manager-8] [INFO] [1739147646.719667778] [tb4.lifecycle_manager_navigation]: Waiting for service smoother_server/get_state...
[lifecycle_manager-8] [INFO] [1739147648.719862938] [tb4.lifecycle_manager_navigation]: Waiting for service smoother_server/get_state...
[lifecycle_manager-8] [INFO] [1739147650.720143566] [tb4.lifecycle_manager_navigation]: Waiting for service smoother_server/get_state...
[lifecycle_manager-8] [INFO] [1739147652.720410725] [tb4.lifecycle_manager_navigation]: Waiting for service smoother_server/get_state...
[lifecycle_manager-8] [INFO] [1739147657.841892730] [tb4.lifecycle_manager_navigation]: Waiting for service bt_navigator/get_state...
[lifecycle_manager-8] [INFO] [1739147658.891662652] [tb4.lifecycle_manager_navigation]: Starting managed nodes bringup...
[lifecycle_manager-8] [INFO] [1739147658.891753408] [tb4.lifecycle_manager_navigation]: Configuring controller_server
[controller_server-1] [INFO] [1739147658.892280146] [tb4.controller_server]: Configuring controller interface
[controller_server-1] [INFO] [1739147658.892660785] [tb4.controller_server]: getting goal checker plugins..
[controller_server-1] [INFO] [1739147658.892881374] [tb4.controller_server]: Controller frequency set to 20.0000Hz
[controller_server-1] [INFO] [1739147658.892927591] [tb4.local_costmap.local_costmap]: Configuring
[controller_server-1] [INFO] [1739147658.966383194] [tb4.local_costmap.local_costmap]: Using plugin "static_layer"
[controller_server-1] [INFO] [1739147658.993837239] [tb4.local_costmap.local_costmap]: Subscribing to the map topic (/tb4/map) with transient local durability
[controller_server-1] [INFO] [1739147658.997600881] [tb4.local_costmap.local_costmap]: Initialized plugin "static_layer"
[controller_server-1] [INFO] [1739147658.997698649] [tb4.local_costmap.local_costmap]: Using plugin "voxel_layer"
[controller_server-1] [INFO] [1739147659.046349681] [tb4.local_costmap.local_costmap]: Subscribed to Topics: scan
[controller_server-1] [INFO] [1739147659.539914932] [tb4.local_costmap.local_costmap]: Initialized plugin "voxel_layer"
[controller_server-1] [INFO] [1739147659.540056514] [tb4.local_costmap.local_costmap]: Using plugin "inflation_layer"
[controller_server-1] [INFO] [1739147659.587299825] [tb4.local_costmap.local_costmap]: Initialized plugin "inflation_layer"
[controller_server-1] [INFO] [1739147659.697181794] [tb4.controller_server]: Created progress_checker : progress_checker of type nav2_controller::SimpleProgressChecker
[controller_server-1] [INFO] [1739147659.700322598] [tb4.controller_server]: Created goal checker : general_goal_checker of type nav2_controller::SimpleGoalChecker
[controller_server-1] [INFO] [1739147659.719785391] [tb4.controller_server]: Controller Server has general_goal_checker goal checkers available.
[controller_server-1] [INFO] [1739147659.723887149] [tb4.controller_server]: Created controller : FollowPath of type dwb_core::DWBLocalPlanner
[controller_server-1] [INFO] [1739147659.824447919] [tb4.controller_server]: Setting transform_tolerance to 0.200000
[controller_server-1] [INFO] [1739147659.939537472] [tb4.local_costmap.local_costmap]: StaticLayer: Resizing static layer to 172 X 402 at 0.050000 m/pix
[controller_server-1] [INFO] [1739147660.030484326] [tb4.controller_server]: Using critic "RotateToGoal" (dwb_critics::RotateToGoalCritic)
[controller_server-1] [INFO] [1739147660.045523167] [tb4.controller_server]: Critic plugin initialized
[controller_server-1] [INFO] [1739147660.045910631] [tb4.controller_server]: Using critic "Oscillation" (dwb_critics::OscillationCritic)
[controller_server-1] [INFO] [1739147660.055796507] [tb4.controller_server]: Critic plugin initialized
[controller_server-1] [INFO] [1739147660.056134382] [tb4.controller_server]: Using critic "BaseObstacle" (dwb_critics::BaseObstacleCritic)
[controller_server-1] [INFO] [1739147660.056645938] [tb4.controller_server]: Critic plugin initialized
[controller_server-1] [INFO] [1739147660.056912225] [tb4.controller_server]: Using critic "GoalAlign" (dwb_critics::GoalAlignCritic)
[controller_server-1] [INFO] [1739147660.090788793] [tb4.controller_server]: Critic plugin initialized
[controller_server-1] [INFO] [1739147660.091454840] [tb4.controller_server]: Using critic "PathAlign" (dwb_critics::PathAlignCritic)
[controller_server-1] [INFO] [1739147660.093106174] [tb4.controller_server]: Critic plugin initialized
[controller_server-1] [INFO] [1739147660.093714075] [tb4.controller_server]: Using critic "PathDist" (dwb_critics::PathDistCritic)
[controller_server-1] [INFO] [1739147660.094728391] [tb4.controller_server]: Critic plugin initialized
[controller_server-1] [INFO] [1739147660.095309701] [tb4.controller_server]: Using critic "GoalDist" (dwb_critics::GoalDistCritic)
[controller_server-1] [INFO] [1739147660.096217918] [tb4.controller_server]: Critic plugin initialized
[controller_server-1] [INFO] [1739147660.096259174] [tb4.controller_server]: Controller Server has FollowPath controllers available.
[lifecycle_manager-8] [INFO] [1739147660.193751024] [tb4.lifecycle_manager_navigation]: Configuring smoother_server
[smoother_server-2] [INFO] [1739147660.194473525] [tb4.smoother_server]: Configuring smoother server
[smoother_server-2] [INFO] [1739147660.240800951] [tb4.smoother_server]: Created smoother : simple_smoother of type nav2_smoother::SimpleSmoother
[smoother_server-2] [INFO] [1739147660.245882476] [tb4.smoother_server]: Smoother Server has simple_smoother smoothers available.
[lifecycle_manager-8] [INFO] [1739147660.294532184] [tb4.lifecycle_manager_navigation]: Configuring planner_server
[planner_server-3] [INFO] [1739147660.294986182] [tb4.planner_server]: Configuring
[planner_server-3] [INFO] [1739147660.295066481] [tb4.global_costmap.global_costmap]: Configuring
[planner_server-3] [INFO] [1739147660.355722352] [tb4.global_costmap.global_costmap]: Using plugin "static_layer"
[planner_server-3] [INFO] [1739147660.362795783] [tb4.global_costmap.global_costmap]: Subscribing to the map topic (/tb4/map) with transient local durability
[planner_server-3] [INFO] [1739147660.375231543] [tb4.global_costmap.global_costmap]: Initialized plugin "static_layer"
[planner_server-3] [INFO] [1739147660.375448822] [tb4.global_costmap.global_costmap]: Using plugin "obstacle_layer"
[planner_server-3] [INFO] [1739147660.392342689] [tb4.global_costmap.global_costmap]: Subscribed to Topics: scan
[planner_server-3] [INFO] [1739147660.438141231] [tb4.global_costmap.global_costmap]: Initialized plugin "obstacle_layer"
[planner_server-3] [INFO] [1739147660.438265243] [tb4.global_costmap.global_costmap]: Using plugin "inflation_layer"
[planner_server-3] [INFO] [1739147660.441842522] [tb4.global_costmap.global_costmap]: Initialized plugin "inflation_layer"
[planner_server-3] [INFO] [1739147660.496202058] [tb4.planner_server]: Created global planner plugin GridBased of type nav2_navfn_planner/NavfnPlanner
[planner_server-3] [INFO] [1739147660.496302180] [tb4.planner_server]: Configuring plugin GridBased of type NavfnPlanner
[planner_server-3] [INFO] [1739147660.506536817] [tb4.planner_server]: Planner Server has GridBased planners available.
[lifecycle_manager-8] [INFO] [1739147660.675196184] [tb4.lifecycle_manager_navigation]: Configuring behavior_server
[behavior_server-4] [INFO] [1739147660.675945848] [tb4.behavior_server]: Configuring
[behavior_server-4] [INFO] [1739147660.723754918] [tb4.behavior_server]: Creating behavior plugin spin of type nav2_behaviors/Spin
[behavior_server-4] [INFO] [1739147660.724904956] [tb4.behavior_server]: Configuring spin
[behavior_server-4] [INFO] [1739147660.824232999] [tb4.behavior_server]: Creating behavior plugin backup of type nav2_behaviors/BackUp
[behavior_server-4] [INFO] [1739147660.825919481] [tb4.behavior_server]: Configuring backup
[behavior_server-4] [INFO] [1739147660.877482940] [tb4.behavior_server]: Creating behavior plugin drive_on_heading of type nav2_behaviors/DriveOnHeading
[behavior_server-4] [INFO] [1739147660.878910310] [tb4.behavior_server]: Configuring drive_on_heading
[behavior_server-4] [INFO] [1739147660.975635213] [tb4.behavior_server]: Creating behavior plugin assisted_teleop of type nav2_behaviors/AssistedTeleop
[behavior_server-4] [INFO] [1739147660.979125526] [tb4.behavior_server]: Configuring assisted_teleop
[behavior_server-4] [INFO] [1739147661.172087832] [tb4.behavior_server]: Creating behavior plugin wait of type nav2_behaviors/Wait
[behavior_server-4] [INFO] [1739147661.173516293] [tb4.behavior_server]: Configuring wait
[lifecycle_manager-8] [INFO] [1739147661.237295687] [tb4.lifecycle_manager_navigation]: Configuring bt_navigator
[bt_navigator-5] [INFO] [1739147661.237813645] [tb4.bt_navigator]: Configuring
[lifecycle_manager-8] [INFO] [1739147662.861497223] [tb4.lifecycle_manager_navigation]: Configuring waypoint_follower
[waypoint_follower-6] [INFO] [1739147662.861955057] [tb4.waypoint_follower]: Configuring
[waypoint_follower-6] [INFO] [1739147662.954726925] [tb4.waypoint_follower]: Created waypoint_task_executor : wait_at_waypoint of type nav2_waypoint_follower::WaitAtWaypoint
[lifecycle_manager-8] [INFO] [1739147662.957751612] [tb4.lifecycle_manager_navigation]: Configuring velocity_smoother
[velocity_smoother-7] [INFO] [1739147662.958550155] [tb4.velocity_smoother]: Configuring velocity smoother
[lifecycle_manager-8] [INFO] [1739147663.069175109] [tb4.lifecycle_manager_navigation]: Activating controller_server
[controller_server-1] [INFO] [1739147663.069904953] [tb4.controller_server]: Activating
[controller_server-1] [INFO] [1739147663.070005957] [tb4.local_costmap.local_costmap]: Activating
[controller_server-1] [INFO] [1739147663.070123811] [tb4.local_costmap.local_costmap]: Checking transform
[controller_server-1] [INFO] [1739147663.070347706] [tb4.local_costmap.local_costmap]: start
[controller_server-1] [INFO] [1739147663.272693534] [tb4.controller_server]: Creating bond (controller_server) to lifecycle manager.
[lifecycle_manager-8] [ERROR] [1739147665.292793558] [tb4.lifecycle_manager_navigation]: Server controller_server was unable to be reached after 4.00s by bond. This server may be misconfigured.
[lifecycle_manager-8] [ERROR] [1739147665.292893387] [tb4.lifecycle_manager_navigation]: Failed to bring up all requested nodes. Aborting bringup.
[planner_server-3] [INFO] [1739147665.787061830] [tb4.global_costmap.global_costmap]: StaticLayer: Resizing costmap to 172 X 402 at 0.050000 m/pix
[controller_server-1] [INFO] [1739147668.063874559] [tb4.local_costmap.local_costmap]: Message Filter dropping message: frame 'rplidar_link' at time 1739147667.334 for reason 'the timestamp on the message is earlier than all the data in the transform cache'
[controller_server-1] [INFO] [1739147672.577698654] [tb4.local_costmap.local_costmap]: Message Filter dropping message: frame 'rplidar_link' at time 1739147671.438 for reason 'the timestamp on the message is earlier than all the data in the transform cache'
[controller_server-1] [INFO] [1739147672.621853636] [tb4.local_costmap.local_costmap]: Message Filter dropping message: frame 'rplidar_link' at time 1739147671.446 for reason 'the timestamp on the message is earlier than all the data in the transform cache'
[controller_server-1] [INFO] [1739147675.335092485] [tb4.local_costmap.local_costmap]: Message Filter dropping message: frame 'rplidar_link' at time 1739147674.459 for reason 'the timestamp on the message is earlier than all the data in the transform cache'
To Reproduce
On the remote PC, open three terminals:
- Terminal 1:
ros2 launch turtlebot4_viz view_robot.launch.py namespace:=/tb4 - Terminal 2:
ros2 launch turtlebot4_navigation localization.launch.py namespace:=/tb4 map:=map_name.yaml - Terminal 3:
ros2 launch turtlebot4_navigation nav2.launch.py namespace:=/tb4
Other notes
I am running the robot in discovery_server mode with a namespace /tb4 as suggested in #468 .
I get a lot of warnings about dropped messages as the timestamp on the message is earlier than all the data in the transform cache. Following advice in #530 I have set up time sync between the Create3, Raspberry Pi, and remote PC. Sync between the Create3 and Raspberry Pi was enabled by default. I made changes to /etc/chrony/chrony.conf on the Pi and remote PC as follows:
- Pi: added
server <remote_pc_ip> iburst prefer - Remote PC: added
allow <raspberry_pi_ip>
Running sudo chronyc sources on the Pi now shows:
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^- alphyn.canonical.com 2 6 377 55 -2317us[-2317us] +/- 326ms
^- prod-ntp-4.ntp1.ps5.cano> 2 6 375 252 +3336us[+3336us] +/- 130ms
^- prod-ntp-5.ntp4.ps5.cano> 2 6 377 252 +2938us[+2938us] +/- 130ms
^- prod-ntp-3.ntp1.ps5.cano> 2 6 377 253 +1372us[+1372us] +/- 131ms
^- smtp.juneks.com.au 1 6 377 320 -189us[ +50us] +/- 3602us
^- auvps1.sameteem.com 5 6 376 253 -228us[ -228us] +/- 5530us
^- syd.clearnet.pw 2 6 377 252 -321us[ -321us] +/- 27ms
^- time.cloudflare.com 3 6 377 254 -589us[ -589us] +/- 4914us
^* <remote_pc_ip> 2 6 377 255 +453us[ +918us] +/- 6733us
And running sudo chronyc clients on the remote PC gives:
Hostname NTP Drop Int IntL Last Cmd Drop Int Last
===============================================================================
<raspberry_pi_ip> 31 0 6 - 20 0 0 - -
This indicates to me that the remote PC is the preferred source, but sync doesn’t seem perfect. Could this be the root of the issue?
I’ve also observed the nodes started by localization.launch.py and nav2.launch.py take a long time to find services eg. controller_server/get_state (as in this example) or the map_server services. Sometimes they don’t find them after 30 seconds, and I restart the command. Is this possibly related?
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 1
- Comments: 20 (5 by maintainers)
@beardyFace
So after that first attempt has been ended (I assume you are using ctrl+c in the navigation terminals), then open up your preferred version of
topto look at the processes. A lot of people usehtop, I preferbtop. In there, filter the running processes and see if there are anyrosorlifecycleprocesses remaining. If there are, then note down which ones are running, then end those processes. Most (if not all) top programs facilitate sending terminate / kill signals to the programs to end them. Then restart your navigation and see if it works the second time.@slowrunner interesting, please do keep us up to date
@EdBrayUTS
Were those 2 the only ros processes that were left after everything else was shut down?
Hmmm, better but not good. There has got to be something more happening behind the scenes. I’m going to have to think about this.
Haha, yah, no. If the parameter were being passed through directly to a node that was already receiving a parameter config file then there would be easier ways. I mean, you could modify the launch of that node to instead receive the same yaml as everything else and then put the parameters into the config (but of course specifying the node name etc in that config file) but I don’t think that is really much less effort. It does mean that there is one less layer to modify though.
@EdBrayUTS IMO, this flakey nav2 bringup is not caused by our configurations. With multiple folks seeing it with slightly different configurations, I am starting to believe that something changed in the nav2 code base that has found an edge case in the bringup.
This morning, I did a RPi reboot, and rebooted my visualization platform and then had to launch navigation three times to get a working system. Two days ago, I had to launch five times before I got a working system.
Somehow we have to characterize the failure, as it doesn’t look like any of us have caught the attention of the god of nav2. (@SteveMacenski)
Same issue i am facing… if find any solution please help me… @EdBrayUTS
Slight update from my end. I’ve isloated the changes I made and the things that made the difference seem to be the combination of:
bond_timeoutto120innavigation_launch.pyandlocalization_launch.pywithinnav2_bringupThe newer version of
bond_coreand changing thebt_navigatortimeout parameters didn’t seem to make too much difference.Similar to @beardyFace , I’m finding it typically works fine the first time when everything has just booted, but subsequent runs are less reliable.
I’ve also looked at the nodes that are running after I’ve cancelled everything (RViz, localisation, and navigation) as suggested @hilary-luo . I can see that often
/tb4/lifecycle_manager_navigationand/or/tb4/lifecycle_manager_localizationremains. Restarting the daemon seems to make these nodes disappear, and subsequent startups are more reliable.With these changes (router,
bond_timeout, and resarting daemon) I seem to get ~60% reliability when restarting. So making progress, but still not great.Finally @hilary-luo I’m aware of how one can expose parameters through launch files, but since there are quite a few nested launch files in play here I was more asking if there was a cheaty short cut to avoid having to change all of them 😉 Seems not though
@hilary-luo - It does appear that it all works fine the first time when everything has just been booted up on either end. Then there are issues on subsequent runs.
I did try restarting the daemon, but it didn’t seem to have any impact although I can test further. Is there any more specific commands that can really knock out any processes still running?
@EdBrayUTS Thanks for sharing this info about your system and your testing so far. You may already be aware of this, but with a system this complex I would suggest building these packages for release when you are testing (has performance implications). It won’t affect the Turtlebot4 code, but likely would impact Nav2 performance (
colcon build --cmake-args -DCMAKE_BUILD_TYPE=Release). That being said, given the resources available it probably will not change the outcome on your machine.In regards to the bond timeout, the easiest answer is that you can expose it as a launch argument yourself. You can look at the
autostartparameter as an example of how to set this up, just ignore the part about theparam_substitutions. So declare the Launch configuration (https://github.com/ros-navigation/navigation2/blob/e4ef813e6b5c6edc4307231e75a9cd01ebbd8c5b/nav2_bringup/launch/navigation_launch.py#L35) and create the Launch argument (https://github.com/ros-navigation/navigation2/blob/e4ef813e6b5c6edc4307231e75a9cd01ebbd8c5b/nav2_bringup/launch/navigation_launch.py#L96-L100) , pass it in as a launch parameter (https://github.com/ros-navigation/navigation2/blob/e4ef813e6b5c6edc4307231e75a9cd01ebbd8c5b/nav2_bringup/launch/navigation_launch.py#L233) and add the action with the rest of the launch option actions (https://github.com/ros-navigation/navigation2/blob/e4ef813e6b5c6edc4307231e75a9cd01ebbd8c5b/nav2_bringup/launch/navigation_launch.py#L332). A similar process can be done in the turtlebot4 launch file if you want to expose it all of the way out as an argument there.Having done that work you could also contribute that change back to Nav2 since it could be useful for others as well (something to think about).
@ everyone here A separate thought that I would like you guys to take a look at: is this issue repeatable that it always works the first time after boot and is flaky on subsequent boots? This is going in a bit of a different direction from the other troubleshooting but after you close the nav nodes etc. can you also stop the ROS2 Daemon (
ros2 daemon stop) and then observe your processes that are running to make sure that there are no residual ROS processes that are still running on your offboard computer? One significant change for Nav2 between Humble and Jazzy is the increased use of Lifecycle nodes and thus more complex lifecycle management.@hilary-luo thanks for the suggestions. I’ve run some initial tests and had some promising results I’ll share with everyone now to see if they help anyone else. I’ve not checked each change independently to see their relative importance, but what I did was:
bond_core,navigation2, andturtlebot4repos into a cleanros2_wsand built themturtlebot4/turtlrbot4_navigation/conig/nav2.yaml:bt_navigator/default_server_timeoutfrom20to120bt_navigator/wait_for_service_timeoutand set to120000(parameter is listed in their docs)navigation2/nav2_bringup/launch/navigation_launch.pyandnavigation2/nav2_bringup/launch/localization_launch.pyto increase the bond timeout. In each launch file:lifecycle_managernodebond_timeoutwith a value of120.0(parameter is listed in their docs)I just got it working twice in a row, which is pretty unprecidented. The next bringup then failed, so not perfect though… I often get issues with AMCL actually, for example:
But AMCL doesn’t seem to have any timeout parameters we can change.
I’ll do more tests when I get time to work out which the most important changes were. Is there a way to change the
bond_timeoutparameter on launch without having to edit the launch file in the Nav2 repo? They’re not exposed this as a launch argument, so I don’t think so but maybe someone knows a way.In response to @hilary-luo 's specific questions:
No, this still happens sometimes. My guess is that network traffic is clogging up
tfmessages when everything starts up. On a successful startup, I get this warning intermittently but it eventually settles down. This doesn’t seem like a major issue: enough ‘correct’ messages get throughThis is running natively on a pretty powerful offboard PC. A successful startup uses < 5% CPU on this machine.
A successful startup results in <25 % CPU usage on the Pi. Again, this doesn’t seem unreasonable.
Both the univesity network I was on originally, and the router I’m on now, are 5 GHz networks. The Create3 is not connected to any network.
@beardyFace depending on your networking configuration that is possible. If you are fully running on discovery server with the only server on the robot, then all of the discovery traffic has to make it to the robot server and back again in order for nodes on your offboard computer to discover each other. If you have a poor or congested wifi network, this could potentially be a problem, particularly if the robot is also sending stereo data across the network.
You could try and isolate out this issue by either a) Testing over a wired connection with wifi disabled on all devices (make sure you update the server IP address on your offboard pc to point at the wired IP address of the pi) b) Adding an extra discovery server locally on your offboard computer and launch all nodes on the offboard computer pointing at both servers (that way they can complete discovery locally but also can still access the robot nodes) c) get a dedicated router to test with (needs to support 5 GHz with Wifi 5 speeds minimum)
Let me know how it goes
@slowrunner just tried that myself and no luck I’m afraid. Without changing anything, all the relevant nodes already had
use_sim_timeset toFalsewhen I checked after launching.@beardyFace thanks for the link. I’ve just tried installing packages within the
navigation2,turtlebot4, andbond_corerepositories from source (commits1c68c212db01f9f75fcb8263a0fbb5dfa711bdea,e8b710f9c4976950af2b92693bc861c7837b36b5, andfb7ed7cbd5572c14f0176f987ac45ef7b45fc952respectively).The first time starting everything from a fresh boot everything worked as it should! But I’ve not been able to replicate it. The next time I launched everything I was met with:
Then on several subsequent attempts (even after a restart of the robot) it gets stuck at:
I also tried directly over ethernet, which similarly worked once then not again. It seemed to get past waiting for servers, but struggled to call them, giving errors like:
So far none of the things I’ve tried (setting up timesync, building packages from source, going over ethernet, manually changing the
bond_timeoutparameter of thelifecycle_managernode) seems to have a statistically significant effect on whether things work or not, or even which of the many possible errors show up.