colcon-core: Overriding package error for packages in current workspace

I’ve just experienced a slightly annoying side effect of this recently merged PR: https://github.com/colcon/colcon-core/pull/449

Consider the following workflow:

source /opt/ros/noetic/setup.bash
cd my_workspace
colcon build --merge-install
source install/setup.bash
colcon build --merge-install

I use this workflow often because I build, run something, make some changes, then want to build again. However, the second colcon build fails because the workspace is now seen as overlaying itself and it prints the package-overridden error message for every package in the local workspace - it thinks they are overriding themselves.

Is my workflow bad (i.e. running colcon build after sourcing the local workspace) or is this a bug?

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 6
  • Comments: 19 (9 by maintainers)

Most upvoted comments

I didn’t know about having to build and source/run in separate terminals. After a bit of searching, I did find it is covered in one of the very first tutorials of ROS2.

For reference, quoting ROS2 tutorials - Creating a workspace,

Before sourcing the overlay, it is very important that you open a new terminal, separate from the one where you built the workspace. Sourcing an overlay in the same terminal where you built, or likewise building where an overlay is sourced, may create complex issues.

I’ll start with making the override warning ignore the workspace being built when looking at packages from underlay’s makes sense to me. That seems like it would resolve this particular issue. It seems like there’s more being discussed than that here.

There’s a video explaining problems with overriding workspaces here that may be helpful for context.

I use this workflow often because I build, run something, make some changes, then want to build again. However, the second colcon build fails because the workspace is now seen as overlaying itself and it prints the package-overridden error message for every package in the local workspace - it thinks they are overriding themselves.

@janjachnik I think the warning isn’t needed for your specific case, but for more context the build-source-build in the same terminal is discouraged because it breaks isolation. Packages that don’t declare their dependencies can still use them, and I think it’s possible to add circular dependencies after the first build. Since you’re using a merged workspace, this is already a problem, so sourcing and building again doesn’t add more downsides than --merge-install already did. I can see why this is a valid use case, but using isolated workspaces can save you a lot of headache if you intend to release your packages.

From my understanding of the original issue (ros2/ros2#1150), it is when the underlay has used the --merge-install option that there is a problem, not the overlay, which in some cases is out of the users control

@janjachnik ros2/ros2#1150 is only about one issue that can happen when overriding packages. Any type of workspace can experience undefined behavior (failing to build downstream packages or undefined behavior at runtime) when an overridden package changes ABI or API.

Would it be an option to have this warning/error only if using with --merge-install? For me, this would solve the issue.

@destogl The warning applies to any kind of workspace too because it’s very easy to get undefined runtime behavior if the overriding package changes API or breaks ABI.

we build a workspace (ROS 2 from source) we source that workspace we build a sub-set of the packages in another workspace

@christophebedard If the ROS-2-from-source workspace is a merged workspace, then there are of course potential issues with the search order for include directories. Besides that, do any packages in the underlay depend on the ones being overridden, but are themselves not rebuilt? If so, API or ABI changes in the re-built subset can lead to undefined behavior.

I would strongly suggest that you rethink this functionality, or provide a clear workflow for people running complex workspaces for robots.

@destogl, I hear your frustration. I don’t yet understand your workflow, could you tell me more about it? I’m not sure if you’re talking about Overlaying workspaces, or overriding packages. Overlaying workspaces is fine, it’s overriding packages (same package built in two or more workspaces) that’s the problem. Removing the overridden the package from the underlay workspace is always an alternative, and usually it’s just one or two commands. It could be apt-get remove ros-distro-package-being-overridden, or deleting the install/package-being-overridden folder from an isolated underlay, or removing and rebuilding a merged underlay workspace.

Getting overlaying to work correctly – especially with modern CMake in the mix – is non-trivial, and will likely require someone spending the time to fix how Colcon extracts the information ROS packages need to link/include the correct libraries/headers.

Solving the include directory search order problem alone is non-trivial. If the overriden package changes API or breaks ABI and another package in the underlay depends on it then there’s no good way to override it safely, even in ROS 1. The only option is to build all packages above the one you want to override from source.

+1 to this issue. I have the same workflow as @janjachnik and have never experienced the “complex issues” mentioned in the tutorial. I personally source both the overlay’s and underlay’s setup.bash in bashrc. I know it’s redundant but didn’t think it was necessarily bad or could cause issues, until this warning.

Are we expected to not source setup.bash in bashrc to avoid this warning? This would be inconvenient for quickly popping up a terminal and running a node/ros tool