turtlebot4: Inconsistent Navigation on Existing Maps

Robot Model

Turtlebot4 Standard

ROS distro

Humble

Networking Configuration

Discovery Server

OS

Ubuntu 22.04

Built from source or installed?

Installed

Package version

turtlebot4_navigation version 1.0.5-1jammy.20250114.201201

All packages on both remote PC and Raspberry Pi fully-updated at time of opening this issue.

Type of issue

Navigation (SLAM, Nav2 etc.)

Expected behaviour

The robot to be able to autonomously navigate to goals provied in RViz using a pre-existing map of the environment.

Actual behaviour

This behaviour is very inconsistent. Very rarely, the everything works as expected. However typically something crashes when launching nav2.launch.py. An example error log is shown below.

The error is not always the same, but it is typically lifecycle_manager being unable to reach either controller_server or smoother_server after 4 seconds by bond. In the below instance, it was controller_server that cannot be reached.

Error messages

[INFO] [launch]: All log files can be found below /home/ed/.ros/log/2025-02-10-11-33-39-912698-Unhygienix-24041
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [controller_server-1]: process started with pid [24042]
[INFO] [smoother_server-2]: process started with pid [24044]
[INFO] [planner_server-3]: process started with pid [24046]
[INFO] [behavior_server-4]: process started with pid [24048]
[INFO] [bt_navigator-5]: process started with pid [24050]
[INFO] [waypoint_follower-6]: process started with pid [24052]
[INFO] [velocity_smoother-7]: process started with pid [24054]
[INFO] [lifecycle_manager-8]: process started with pid [24056]
[velocity_smoother-7] [INFO] [1739147620.022728425] [tb4.velocity_smoother]: 
[velocity_smoother-7] 	velocity_smoother lifecycle node launched. 
[velocity_smoother-7] 	Waiting on external lifecycle transitions to activate
[velocity_smoother-7] 	See https://design.ros2.org/articles/node_lifecycle.html for more information.
[waypoint_follower-6] [INFO] [1739147620.026065141] [tb4.waypoint_follower]: 
[waypoint_follower-6] 	waypoint_follower lifecycle node launched. 
[waypoint_follower-6] 	Waiting on external lifecycle transitions to activate
[waypoint_follower-6] 	See https://design.ros2.org/articles/node_lifecycle.html for more information.
[waypoint_follower-6] [INFO] [1739147620.026408458] [tb4.waypoint_follower]: Creating
[lifecycle_manager-8] [INFO] [1739147620.026702640] [tb4.lifecycle_manager_navigation]: Creating
[bt_navigator-5] [INFO] [1739147620.027269513] [tb4.bt_navigator]: 
[bt_navigator-5] 	bt_navigator lifecycle node launched. 
[bt_navigator-5] 	Waiting on external lifecycle transitions to activate
[bt_navigator-5] 	See https://design.ros2.org/articles/node_lifecycle.html for more information.
[bt_navigator-5] [INFO] [1739147620.027338303] [tb4.bt_navigator]: Creating
[lifecycle_manager-8] [INFO] [1739147620.027507127] [tb4.lifecycle_manager_navigation]: Creating and initializing lifecycle service clients
[controller_server-1] [INFO] [1739147620.029482791] [tb4.controller_server]: 
[controller_server-1] 	controller_server lifecycle node launched. 
[controller_server-1] 	Waiting on external lifecycle transitions to activate
[controller_server-1] 	See https://design.ros2.org/articles/node_lifecycle.html for more information.
[behavior_server-4] [INFO] [1739147620.029882550] [tb4.behavior_server]: 
[behavior_server-4] 	behavior_server lifecycle node launched. 
[behavior_server-4] 	Waiting on external lifecycle transitions to activate
[behavior_server-4] 	See https://design.ros2.org/articles/node_lifecycle.html for more information.
[controller_server-1] [INFO] [1739147620.031244657] [tb4.controller_server]: Creating controller server
[controller_server-1] [INFO] [1739147620.033090280] [tb4.local_costmap.local_costmap]: 
[controller_server-1] 	local_costmap lifecycle node launched. 
[controller_server-1] 	Waiting on external lifecycle transitions to activate
[controller_server-1] 	See https://design.ros2.org/articles/node_lifecycle.html for more information.
[smoother_server-2] [INFO] [1739147620.033249216] [tb4.smoother_server]: 
[smoother_server-2] 	smoother_server lifecycle node launched. 
[smoother_server-2] 	Waiting on external lifecycle transitions to activate
[smoother_server-2] 	See https://design.ros2.org/articles/node_lifecycle.html for more information.
[controller_server-1] [INFO] [1739147620.033317454] [tb4.local_costmap.local_costmap]: Creating Costmap
[smoother_server-2] [INFO] [1739147620.033950263] [tb4.smoother_server]: Creating smoother server
[planner_server-3] [INFO] [1739147620.036680856] [tb4.planner_server]: 
[planner_server-3] 	planner_server lifecycle node launched. 
[planner_server-3] 	Waiting on external lifecycle transitions to activate
[planner_server-3] 	See https://design.ros2.org/articles/node_lifecycle.html for more information.
[planner_server-3] [INFO] [1739147620.037522448] [tb4.planner_server]: Creating
[planner_server-3] [INFO] [1739147620.039236625] [tb4.global_costmap.global_costmap]: 
[planner_server-3] 	global_costmap lifecycle node launched. 
[planner_server-3] 	Waiting on external lifecycle transitions to activate
[planner_server-3] 	See https://design.ros2.org/articles/node_lifecycle.html for more information.
[planner_server-3] [INFO] [1739147620.039439076] [tb4.global_costmap.global_costmap]: Creating Costmap
[lifecycle_manager-8] [INFO] [1739147622.027983201] [tb4.lifecycle_manager_navigation]: Waiting for service controller_server/get_state...
[lifecycle_manager-8] [INFO] [1739147624.028178826] [tb4.lifecycle_manager_navigation]: Waiting for service controller_server/get_state...
[lifecycle_manager-8] [INFO] [1739147626.028357785] [tb4.lifecycle_manager_navigation]: Waiting for service controller_server/get_state...
[lifecycle_manager-8] [INFO] [1739147628.028672616] [tb4.lifecycle_manager_navigation]: Waiting for service controller_server/get_state...
[lifecycle_manager-8] [INFO] [1739147630.028848347] [tb4.lifecycle_manager_navigation]: Waiting for service controller_server/get_state...
[lifecycle_manager-8] [INFO] [1739147632.029022374] [tb4.lifecycle_manager_navigation]: Waiting for service controller_server/get_state...
[lifecycle_manager-8] [INFO] [1739147634.029209576] [tb4.lifecycle_manager_navigation]: Waiting for service controller_server/get_state...
[lifecycle_manager-8] [INFO] [1739147636.029453824] [tb4.lifecycle_manager_navigation]: Waiting for service controller_server/get_state...
[lifecycle_manager-8] [INFO] [1739147638.029705882] [tb4.lifecycle_manager_navigation]: Waiting for service controller_server/get_state...
[lifecycle_manager-8] [INFO] [1739147640.718887739] [tb4.lifecycle_manager_navigation]: Waiting for service smoother_server/get_state...
[lifecycle_manager-8] [INFO] [1739147642.719141579] [tb4.lifecycle_manager_navigation]: Waiting for service smoother_server/get_state...
[lifecycle_manager-8] [INFO] [1739147644.719448955] [tb4.lifecycle_manager_navigation]: Waiting for service smoother_server/get_state...
[lifecycle_manager-8] [INFO] [1739147646.719667778] [tb4.lifecycle_manager_navigation]: Waiting for service smoother_server/get_state...
[lifecycle_manager-8] [INFO] [1739147648.719862938] [tb4.lifecycle_manager_navigation]: Waiting for service smoother_server/get_state...
[lifecycle_manager-8] [INFO] [1739147650.720143566] [tb4.lifecycle_manager_navigation]: Waiting for service smoother_server/get_state...
[lifecycle_manager-8] [INFO] [1739147652.720410725] [tb4.lifecycle_manager_navigation]: Waiting for service smoother_server/get_state...
[lifecycle_manager-8] [INFO] [1739147657.841892730] [tb4.lifecycle_manager_navigation]: Waiting for service bt_navigator/get_state...
[lifecycle_manager-8] [INFO] [1739147658.891662652] [tb4.lifecycle_manager_navigation]: Starting managed nodes bringup...
[lifecycle_manager-8] [INFO] [1739147658.891753408] [tb4.lifecycle_manager_navigation]: Configuring controller_server
[controller_server-1] [INFO] [1739147658.892280146] [tb4.controller_server]: Configuring controller interface
[controller_server-1] [INFO] [1739147658.892660785] [tb4.controller_server]: getting goal checker plugins..
[controller_server-1] [INFO] [1739147658.892881374] [tb4.controller_server]: Controller frequency set to 20.0000Hz
[controller_server-1] [INFO] [1739147658.892927591] [tb4.local_costmap.local_costmap]: Configuring
[controller_server-1] [INFO] [1739147658.966383194] [tb4.local_costmap.local_costmap]: Using plugin "static_layer"
[controller_server-1] [INFO] [1739147658.993837239] [tb4.local_costmap.local_costmap]: Subscribing to the map topic (/tb4/map) with transient local durability
[controller_server-1] [INFO] [1739147658.997600881] [tb4.local_costmap.local_costmap]: Initialized plugin "static_layer"
[controller_server-1] [INFO] [1739147658.997698649] [tb4.local_costmap.local_costmap]: Using plugin "voxel_layer"
[controller_server-1] [INFO] [1739147659.046349681] [tb4.local_costmap.local_costmap]: Subscribed to Topics: scan
[controller_server-1] [INFO] [1739147659.539914932] [tb4.local_costmap.local_costmap]: Initialized plugin "voxel_layer"
[controller_server-1] [INFO] [1739147659.540056514] [tb4.local_costmap.local_costmap]: Using plugin "inflation_layer"
[controller_server-1] [INFO] [1739147659.587299825] [tb4.local_costmap.local_costmap]: Initialized plugin "inflation_layer"
[controller_server-1] [INFO] [1739147659.697181794] [tb4.controller_server]: Created progress_checker : progress_checker of type nav2_controller::SimpleProgressChecker
[controller_server-1] [INFO] [1739147659.700322598] [tb4.controller_server]: Created goal checker : general_goal_checker of type nav2_controller::SimpleGoalChecker
[controller_server-1] [INFO] [1739147659.719785391] [tb4.controller_server]: Controller Server has general_goal_checker  goal checkers available.
[controller_server-1] [INFO] [1739147659.723887149] [tb4.controller_server]: Created controller : FollowPath of type dwb_core::DWBLocalPlanner
[controller_server-1] [INFO] [1739147659.824447919] [tb4.controller_server]: Setting transform_tolerance to 0.200000
[controller_server-1] [INFO] [1739147659.939537472] [tb4.local_costmap.local_costmap]: StaticLayer: Resizing static layer to 172 X 402 at 0.050000 m/pix
[controller_server-1] [INFO] [1739147660.030484326] [tb4.controller_server]: Using critic "RotateToGoal" (dwb_critics::RotateToGoalCritic)
[controller_server-1] [INFO] [1739147660.045523167] [tb4.controller_server]: Critic plugin initialized
[controller_server-1] [INFO] [1739147660.045910631] [tb4.controller_server]: Using critic "Oscillation" (dwb_critics::OscillationCritic)
[controller_server-1] [INFO] [1739147660.055796507] [tb4.controller_server]: Critic plugin initialized
[controller_server-1] [INFO] [1739147660.056134382] [tb4.controller_server]: Using critic "BaseObstacle" (dwb_critics::BaseObstacleCritic)
[controller_server-1] [INFO] [1739147660.056645938] [tb4.controller_server]: Critic plugin initialized
[controller_server-1] [INFO] [1739147660.056912225] [tb4.controller_server]: Using critic "GoalAlign" (dwb_critics::GoalAlignCritic)
[controller_server-1] [INFO] [1739147660.090788793] [tb4.controller_server]: Critic plugin initialized
[controller_server-1] [INFO] [1739147660.091454840] [tb4.controller_server]: Using critic "PathAlign" (dwb_critics::PathAlignCritic)
[controller_server-1] [INFO] [1739147660.093106174] [tb4.controller_server]: Critic plugin initialized
[controller_server-1] [INFO] [1739147660.093714075] [tb4.controller_server]: Using critic "PathDist" (dwb_critics::PathDistCritic)
[controller_server-1] [INFO] [1739147660.094728391] [tb4.controller_server]: Critic plugin initialized
[controller_server-1] [INFO] [1739147660.095309701] [tb4.controller_server]: Using critic "GoalDist" (dwb_critics::GoalDistCritic)
[controller_server-1] [INFO] [1739147660.096217918] [tb4.controller_server]: Critic plugin initialized
[controller_server-1] [INFO] [1739147660.096259174] [tb4.controller_server]: Controller Server has FollowPath  controllers available.
[lifecycle_manager-8] [INFO] [1739147660.193751024] [tb4.lifecycle_manager_navigation]: Configuring smoother_server
[smoother_server-2] [INFO] [1739147660.194473525] [tb4.smoother_server]: Configuring smoother server
[smoother_server-2] [INFO] [1739147660.240800951] [tb4.smoother_server]: Created smoother : simple_smoother of type nav2_smoother::SimpleSmoother
[smoother_server-2] [INFO] [1739147660.245882476] [tb4.smoother_server]: Smoother Server has simple_smoother  smoothers available.
[lifecycle_manager-8] [INFO] [1739147660.294532184] [tb4.lifecycle_manager_navigation]: Configuring planner_server
[planner_server-3] [INFO] [1739147660.294986182] [tb4.planner_server]: Configuring
[planner_server-3] [INFO] [1739147660.295066481] [tb4.global_costmap.global_costmap]: Configuring
[planner_server-3] [INFO] [1739147660.355722352] [tb4.global_costmap.global_costmap]: Using plugin "static_layer"
[planner_server-3] [INFO] [1739147660.362795783] [tb4.global_costmap.global_costmap]: Subscribing to the map topic (/tb4/map) with transient local durability
[planner_server-3] [INFO] [1739147660.375231543] [tb4.global_costmap.global_costmap]: Initialized plugin "static_layer"
[planner_server-3] [INFO] [1739147660.375448822] [tb4.global_costmap.global_costmap]: Using plugin "obstacle_layer"
[planner_server-3] [INFO] [1739147660.392342689] [tb4.global_costmap.global_costmap]: Subscribed to Topics: scan
[planner_server-3] [INFO] [1739147660.438141231] [tb4.global_costmap.global_costmap]: Initialized plugin "obstacle_layer"
[planner_server-3] [INFO] [1739147660.438265243] [tb4.global_costmap.global_costmap]: Using plugin "inflation_layer"
[planner_server-3] [INFO] [1739147660.441842522] [tb4.global_costmap.global_costmap]: Initialized plugin "inflation_layer"
[planner_server-3] [INFO] [1739147660.496202058] [tb4.planner_server]: Created global planner plugin GridBased of type nav2_navfn_planner/NavfnPlanner
[planner_server-3] [INFO] [1739147660.496302180] [tb4.planner_server]: Configuring plugin GridBased of type NavfnPlanner
[planner_server-3] [INFO] [1739147660.506536817] [tb4.planner_server]: Planner Server has GridBased  planners available.
[lifecycle_manager-8] [INFO] [1739147660.675196184] [tb4.lifecycle_manager_navigation]: Configuring behavior_server
[behavior_server-4] [INFO] [1739147660.675945848] [tb4.behavior_server]: Configuring
[behavior_server-4] [INFO] [1739147660.723754918] [tb4.behavior_server]: Creating behavior plugin spin of type nav2_behaviors/Spin
[behavior_server-4] [INFO] [1739147660.724904956] [tb4.behavior_server]: Configuring spin
[behavior_server-4] [INFO] [1739147660.824232999] [tb4.behavior_server]: Creating behavior plugin backup of type nav2_behaviors/BackUp
[behavior_server-4] [INFO] [1739147660.825919481] [tb4.behavior_server]: Configuring backup
[behavior_server-4] [INFO] [1739147660.877482940] [tb4.behavior_server]: Creating behavior plugin drive_on_heading of type nav2_behaviors/DriveOnHeading
[behavior_server-4] [INFO] [1739147660.878910310] [tb4.behavior_server]: Configuring drive_on_heading
[behavior_server-4] [INFO] [1739147660.975635213] [tb4.behavior_server]: Creating behavior plugin assisted_teleop of type nav2_behaviors/AssistedTeleop
[behavior_server-4] [INFO] [1739147660.979125526] [tb4.behavior_server]: Configuring assisted_teleop
[behavior_server-4] [INFO] [1739147661.172087832] [tb4.behavior_server]: Creating behavior plugin wait of type nav2_behaviors/Wait
[behavior_server-4] [INFO] [1739147661.173516293] [tb4.behavior_server]: Configuring wait
[lifecycle_manager-8] [INFO] [1739147661.237295687] [tb4.lifecycle_manager_navigation]: Configuring bt_navigator
[bt_navigator-5] [INFO] [1739147661.237813645] [tb4.bt_navigator]: Configuring
[lifecycle_manager-8] [INFO] [1739147662.861497223] [tb4.lifecycle_manager_navigation]: Configuring waypoint_follower
[waypoint_follower-6] [INFO] [1739147662.861955057] [tb4.waypoint_follower]: Configuring
[waypoint_follower-6] [INFO] [1739147662.954726925] [tb4.waypoint_follower]: Created waypoint_task_executor : wait_at_waypoint of type nav2_waypoint_follower::WaitAtWaypoint
[lifecycle_manager-8] [INFO] [1739147662.957751612] [tb4.lifecycle_manager_navigation]: Configuring velocity_smoother
[velocity_smoother-7] [INFO] [1739147662.958550155] [tb4.velocity_smoother]: Configuring velocity smoother
[lifecycle_manager-8] [INFO] [1739147663.069175109] [tb4.lifecycle_manager_navigation]: Activating controller_server
[controller_server-1] [INFO] [1739147663.069904953] [tb4.controller_server]: Activating
[controller_server-1] [INFO] [1739147663.070005957] [tb4.local_costmap.local_costmap]: Activating
[controller_server-1] [INFO] [1739147663.070123811] [tb4.local_costmap.local_costmap]: Checking transform
[controller_server-1] [INFO] [1739147663.070347706] [tb4.local_costmap.local_costmap]: start
[controller_server-1] [INFO] [1739147663.272693534] [tb4.controller_server]: Creating bond (controller_server) to lifecycle manager.
[lifecycle_manager-8] [ERROR] [1739147665.292793558] [tb4.lifecycle_manager_navigation]: Server controller_server was unable to be reached after 4.00s by bond. This server may be misconfigured.
[lifecycle_manager-8] [ERROR] [1739147665.292893387] [tb4.lifecycle_manager_navigation]: Failed to bring up all requested nodes. Aborting bringup.
[planner_server-3] [INFO] [1739147665.787061830] [tb4.global_costmap.global_costmap]: StaticLayer: Resizing costmap to 172 X 402 at 0.050000 m/pix
[controller_server-1] [INFO] [1739147668.063874559] [tb4.local_costmap.local_costmap]: Message Filter dropping message: frame 'rplidar_link' at time 1739147667.334 for reason 'the timestamp on the message is earlier than all the data in the transform cache'
[controller_server-1] [INFO] [1739147672.577698654] [tb4.local_costmap.local_costmap]: Message Filter dropping message: frame 'rplidar_link' at time 1739147671.438 for reason 'the timestamp on the message is earlier than all the data in the transform cache'
[controller_server-1] [INFO] [1739147672.621853636] [tb4.local_costmap.local_costmap]: Message Filter dropping message: frame 'rplidar_link' at time 1739147671.446 for reason 'the timestamp on the message is earlier than all the data in the transform cache'
[controller_server-1] [INFO] [1739147675.335092485] [tb4.local_costmap.local_costmap]: Message Filter dropping message: frame 'rplidar_link' at time 1739147674.459 for reason 'the timestamp on the message is earlier than all the data in the transform cache'

To Reproduce

On the remote PC, open three terminals:

  1. Terminal 1: ros2 launch turtlebot4_viz view_robot.launch.py namespace:=/tb4
  2. Terminal 2: ros2 launch turtlebot4_navigation localization.launch.py namespace:=/tb4 map:=map_name.yaml
  3. Terminal 3: ros2 launch turtlebot4_navigation nav2.launch.py namespace:=/tb4

Other notes

I am running the robot in discovery_server mode with a namespace /tb4 as suggested in #468 .

I get a lot of warnings about dropped messages as the timestamp on the message is earlier than all the data in the transform cache. Following advice in #530 I have set up time sync between the Create3, Raspberry Pi, and remote PC. Sync between the Create3 and Raspberry Pi was enabled by default. I made changes to /etc/chrony/chrony.conf on the Pi and remote PC as follows:

  • Pi: added server <remote_pc_ip> iburst prefer
  • Remote PC: added allow <raspberry_pi_ip>

Running sudo chronyc sources on the Pi now shows:

MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^- alphyn.canonical.com          2   6   377    55  -2317us[-2317us] +/-  326ms
^- prod-ntp-4.ntp1.ps5.cano>     2   6   375   252  +3336us[+3336us] +/-  130ms
^- prod-ntp-5.ntp4.ps5.cano>     2   6   377   252  +2938us[+2938us] +/-  130ms
^- prod-ntp-3.ntp1.ps5.cano>     2   6   377   253  +1372us[+1372us] +/-  131ms
^- smtp.juneks.com.au            1   6   377   320   -189us[  +50us] +/- 3602us
^- auvps1.sameteem.com           5   6   376   253   -228us[ -228us] +/- 5530us
^- syd.clearnet.pw               2   6   377   252   -321us[ -321us] +/-   27ms
^- time.cloudflare.com           3   6   377   254   -589us[ -589us] +/- 4914us
^* <remote_pc_ip>                2   6   377   255   +453us[ +918us] +/- 6733us

And running sudo chronyc clients on the remote PC gives:

Hostname                      NTP   Drop Int IntL Last     Cmd   Drop Int  Last
===============================================================================
<raspberry_pi_ip>                 31      0   6   -    20       0      0   -     -

This indicates to me that the remote PC is the preferred source, but sync doesn’t seem perfect. Could this be the root of the issue?

I’ve also observed the nodes started by localization.launch.py and nav2.launch.py take a long time to find services eg. controller_server/get_state (as in this example) or the map_server services. Sometimes they don’t find them after 30 seconds, and I restart the command. Is this possibly related?

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 1
  • Comments: 20 (5 by maintainers)

Most upvoted comments

@beardyFace

I did try restarting the daemon, but it didn’t seem to have any impact although I can test further. Is there any more specific commands that can really knock out any processes still running?

So after that first attempt has been ended (I assume you are using ctrl+c in the navigation terminals), then open up your preferred version of top to look at the processes. A lot of people use htop, I prefer btop. In there, filter the running processes and see if there are any ros or lifecycle processes remaining. If there are, then note down which ones are running, then end those processes. Most (if not all) top programs facilitate sending terminate / kill signals to the programs to end them. Then restart your navigation and see if it works the second time.

@slowrunner interesting, please do keep us up to date

@EdBrayUTS

I’ve also looked at the nodes that are running after I’ve cancelled everything (RViz, localisation, and navigation) as suggested @hilary-luo . I can see that often /tb4/lifecycle_manager_navigation and/or /tb4/lifecycle_manager_localization remains. Restarting the daemon seems to make these nodes disappear, and subsequent startups are more reliable.

Were those 2 the only ros processes that were left after everything else was shut down?

With these changes (router, bond_timeout, and resarting daemon) I seem to get ~60% reliability when restarting. So making progress, but still not great.

Hmmm, better but not good. There has got to be something more happening behind the scenes. I’m going to have to think about this.

Finally @hilary-luo I’m aware of how one can expose parameters through launch files, but since there are quite a few nested launch files in play here I was more asking if there was a cheaty short cut to avoid having to change all of them 😉 Seems not though

Haha, yah, no. If the parameter were being passed through directly to a node that was already receiving a parameter config file then there would be easier ways. I mean, you could modify the launch of that node to instead receive the same yaml as everything else and then put the parameters into the config (but of course specifying the node name etc in that config file) but I don’t think that is really much less effort. It does mean that there is one less layer to modify though.

@EdBrayUTS IMO, this flakey nav2 bringup is not caused by our configurations. With multiple folks seeing it with slightly different configurations, I am starting to believe that something changed in the nav2 code base that has found an edge case in the bringup.

This morning, I did a RPi reboot, and rebooted my visualization platform and then had to launch navigation three times to get a working system. Two days ago, I had to launch five times before I got a working system.

Somehow we have to characterize the failure, as it doesn’t look like any of us have caught the attention of the god of nav2. (@SteveMacenski)

Same issue i am facing… if find any solution please help me… @EdBrayUTS

Slight update from my end. I’ve isloated the changes I made and the things that made the difference seem to be the combination of:

  • Dedicated router, not university network
  • Changing bond_timeout to 120 in navigation_launch.py and localization_launch.py within nav2_bringup

The newer version of bond_core and changing the bt_navigator timeout parameters didn’t seem to make too much difference.

Similar to @beardyFace , I’m finding it typically works fine the first time when everything has just booted, but subsequent runs are less reliable.

I’ve also looked at the nodes that are running after I’ve cancelled everything (RViz, localisation, and navigation) as suggested @hilary-luo . I can see that often /tb4/lifecycle_manager_navigation and/or /tb4/lifecycle_manager_localization remains. Restarting the daemon seems to make these nodes disappear, and subsequent startups are more reliable.

With these changes (router, bond_timeout, and resarting daemon) I seem to get ~60% reliability when restarting. So making progress, but still not great.

Finally @hilary-luo I’m aware of how one can expose parameters through launch files, but since there are quite a few nested launch files in play here I was more asking if there was a cheaty short cut to avoid having to change all of them 😉 Seems not though

@hilary-luo - It does appear that it all works fine the first time when everything has just been booted up on either end. Then there are issues on subsequent runs.

I did try restarting the daemon, but it didn’t seem to have any impact although I can test further. Is there any more specific commands that can really knock out any processes still running?

@EdBrayUTS Thanks for sharing this info about your system and your testing so far. You may already be aware of this, but with a system this complex I would suggest building these packages for release when you are testing (has performance implications). It won’t affect the Turtlebot4 code, but likely would impact Nav2 performance (colcon build --cmake-args -DCMAKE_BUILD_TYPE=Release). That being said, given the resources available it probably will not change the outcome on your machine.

In regards to the bond timeout, the easiest answer is that you can expose it as a launch argument yourself. You can look at the autostart parameter as an example of how to set this up, just ignore the part about the param_substitutions. So declare the Launch configuration (https://github.com/ros-navigation/navigation2/blob/e4ef813e6b5c6edc4307231e75a9cd01ebbd8c5b/nav2_bringup/launch/navigation_launch.py#L35) and create the Launch argument (https://github.com/ros-navigation/navigation2/blob/e4ef813e6b5c6edc4307231e75a9cd01ebbd8c5b/nav2_bringup/launch/navigation_launch.py#L96-L100) , pass it in as a launch parameter (https://github.com/ros-navigation/navigation2/blob/e4ef813e6b5c6edc4307231e75a9cd01ebbd8c5b/nav2_bringup/launch/navigation_launch.py#L233) and add the action with the rest of the launch option actions (https://github.com/ros-navigation/navigation2/blob/e4ef813e6b5c6edc4307231e75a9cd01ebbd8c5b/nav2_bringup/launch/navigation_launch.py#L332). A similar process can be done in the turtlebot4 launch file if you want to expose it all of the way out as an argument there.

Having done that work you could also contribute that change back to Nav2 since it could be useful for others as well (something to think about).

@ everyone here A separate thought that I would like you guys to take a look at: is this issue repeatable that it always works the first time after boot and is flaky on subsequent boots? This is going in a bit of a different direction from the other troubleshooting but after you close the nav nodes etc. can you also stop the ROS2 Daemon (ros2 daemon stop) and then observe your processes that are running to make sure that there are no residual ROS processes that are still running on your offboard computer? One significant change for Nav2 between Humble and Jazzy is the increased use of Lifecycle nodes and thus more complex lifecycle management.

@hilary-luo thanks for the suggestions. I’ve run some initial tests and had some promising results I’ll share with everyone now to see if they help anyone else. I’ve not checked each change independently to see their relative importance, but what I did was:

  • Switched from the university WiFi network to a dedicated router that’s not even connected to the internet: just the offboard PC and Turtlebot4 Raspberry Pi
  • Cloned the bond_core, navigation2, and turtlebot4 repos into a clean ros2_ws and built them
  • Edited parameters in turtlebot4/turtlrbot4_navigation/conig/nav2.yaml:
    • Changed bt_navigator/default_server_timeout from 20 to 120
    • Added bt_navigator/wait_for_service_timeout and set to 120000 (parameter is listed in their docs)
  • Edited navigation2/nav2_bringup/launch/navigation_launch.py and navigation2/nav2_bringup/launch/localization_launch.py to increase the bond timeout. In each launch file:
    • Find the lifecycle_manager node
    • Pass another parameter to it called bond_timeout with a value of 120.0 (parameter is listed in their docs)

I just got it working twice in a row, which is pretty unprecidented. The next bringup then failed, so not perfect though… I often get issues with AMCL actually, for example:

[amcl-2] [INFO] [1741314938.685094813] [tb4.amcl]: Subscribed to map topic.
[amcl-2] [WARN] [1741314938.789616876] [tb4.amcl.rclcpp]: failed to send response to /tb4/amcl/change_state (timeout): client will not receive response, at ./src/rmw_response.cpp:154, at ./src/rcl/service.c:314

But AMCL doesn’t seem to have any timeout parameters we can change.

I’ll do more tests when I get time to work out which the most important changes were. Is there a way to change the bond_timeout parameter on launch without having to edit the launch file in the Nav2 repo? They’re not exposed this as a launch argument, so I don’t think so but maybe someone knows a way.

In response to @hilary-luo 's specific questions:

  • After you set up the time sync with the remote PC did the the timestamp on the message is earlier than all the data in the transform cache messages go away? That should fully address the time sync issue but would be independent from the timeout issue.

No, this still happens sometimes. My guess is that network traffic is clogging up tf messages when everything starts up. On a successful startup, I get this warning intermittently but it eventually settles down. This doesn’t seem like a major issue: enough ‘correct’ messages get through

  • Are you running this natively? or in a VM / container? Take a look at your resource utilization (CPU usage etc) during launch when the timeouts occur.

This is running natively on a pretty powerful offboard PC. A successful startup uses < 5% CPU on this machine.

  • Similarly can you monitor the resource usage on the pi during this time?

A successful startup results in <25 % CPU usage on the Pi. Again, this doesn’t seem unreasonable.

  • Finally, can you confirm that you have the whole system running on a 5GHz wifi network and that the create3 is disconnected from the internet

Both the univesity network I was on originally, and the router I’m on now, are 5 GHz networks. The Create3 is not connected to any network.

@beardyFace depending on your networking configuration that is possible. If you are fully running on discovery server with the only server on the robot, then all of the discovery traffic has to make it to the robot server and back again in order for nodes on your offboard computer to discover each other. If you have a poor or congested wifi network, this could potentially be a problem, particularly if the robot is also sending stereo data across the network.

You could try and isolate out this issue by either a) Testing over a wired connection with wifi disabled on all devices (make sure you update the server IP address on your offboard pc to point at the wired IP address of the pi) b) Adding an extra discovery server locally on your offboard computer and launch all nodes on the offboard computer pointing at both servers (that way they can complete discovery locally but also can still access the robot nodes) c) get a dedicated router to test with (needs to support 5 GHz with Wifi 5 speeds minimum)

Let me know how it goes

@slowrunner just tried that myself and no luck I’m afraid. Without changing anything, all the relevant nodes already had use_sim_time set to False when I checked after launching.

@beardyFace thanks for the link. I’ve just tried installing packages within the navigation2, turtlebot4, and bond_core repositories from source (commits 1c68c212db01f9f75fcb8263a0fbb5dfa711bdea, e8b710f9c4976950af2b92693bc861c7837b36b5, and fb7ed7cbd5572c14f0176f987ac45ef7b45fc952 respectively).

The first time starting everything from a fresh boot everything worked as it should! But I’ve not been able to replicate it. The next time I launched everything I was met with:

[planner_server-3] [INFO] [1739833291.487637254] [tb4.planner_server]: Planner Server has GridBased  planners available.
[lifecycle_manager-8] [ERROR] [1739833294.433804758] [tb4.lifecycle_manager_navigation]: Failed to change state for node: planner_server. Exception: planner_server/get_state service client: async_send_request failed.
[lifecycle_manager-8] [ERROR] [1739833294.433905890] [tb4.lifecycle_manager_navigation]: Failed to bring up all requested nodes. Aborting bringup.

Then on several subsequent attempts (even after a restart of the robot) it gets stuck at:

[lifecycle_manager-8] [INFO] [1739833902.813031693] [tb4.lifecycle_manager_navigation]: Waiting for service controller_server/get_state...

I also tried directly over ethernet, which similarly worked once then not again. It seemed to get past waiting for servers, but struggled to call them, giving errors like:

[smoother_server-2] [WARN] [1739835197.561420748] [tb4.smoother_server.rclcpp]: failed to send response to /tb4/smoother_server/change_state (timeout): client will not receive response, at ./src/rmw_response.cpp:154, at ./src/rcl/service.c:314

So far none of the things I’ve tried (setting up timesync, building packages from source, going over ethernet, manually changing the bond_timeout parameter of the lifecycle_manager node) seems to have a statistically significant effect on whether things work or not, or even which of the many possible errors show up.