colcon-core: Colcon fails to build valid PEP 517 projects

As mentioned on https://github.com/colcon/colcon-ros/pull/66 , colcon fails to build valid python packages that use pyproject.toml. This includes packages that want to use a setup.cfg without a corresponding setup.py.

$  colcon build --packages-select test_project
[0.403s] ERROR:colcon.colcon_core.package_identification:ROS package '/home/dan/ros2_ws/src/test_project' with build type 'ament_python' has no 'setup.py' file
[0.405s] WARNING:colcon.colcon_core.package_selection:ignoring unknown package 'test_project' in --packages-select
                     
Summary: 0 packages finished [0.27s]

About this issue

  • Original URL
  • State: open
  • Created 7 years ago
  • Reactions: 5
  • Comments: 15 (7 by maintainers)

Most upvoted comments

I’ve just made my colcon-python-project extensions public: https://github.com/colcon/colcon-python-project/tree/devel They were previously private because we wanted to do some more testing before migrating those extensions into colcon-core to eventually become the default but we didn’t want folks to take a dependency on what is explicitly prototype code.

The long term plan is to switch colcon-core to use “standards-based” tooling exclusively, and move the existing functionality into a new extension to support down-level platforms that can’t support the new extensions (RHEL 8, Bionic, etc).

It should be noted that these extensions implement colcon as a “build frontend”, so we aren’t wrapping pip or any of the other build frontends and aren’t coupled to any of their behaviors and limitations. This should allow use of any build backend, though I’ve only tested poetry, setuptools, and flit.

One scenario that isn’t 100% supported yet is using a build backend that is built as part of the same colcon workspace itself. I’ve been toying with introducing a build backend for Ament, and while colcon is able to bootstrap that package itself, we can’t flesh out the full dependency graph without first having built the build backend package, so there is a bootstrapping problem. If you build the backend in a separate workspace, the problem goes away. I’m sure there are other limitations to my current approach, but that one is worth calling out.

re: dependency awareness - my implementation continues colcon’s pattern of being aware of only the dependencies which are part of the colcon workspace.

To use the prototype package on a plain python project, it needs only be installed. To use it on ROS packages, you need to modify the colcon-ros package to use the new Python build extension instead of the setuptools-exclusive one. I’ll make a branch with this change and post it here.

Since my internal efforts to get some testing done on these extensions have largely stalled, I’d appreciate any testing that you or anyone else can provide. Once we’ve settled on the general architecture, we can move the work to a PR against colcon-core.

I’m open to a synchronous discussion or demo on this work if it sounds helpful. In general, I’m much more optimistic than I was previously, but there are certainly still sharp edges here that we need to address, possibly as documentation and announcements.

Looks like your approach builds wheels as an intermediate step, and therefore doesn’t have a story for develop/editable mode? I believe Poetry at least does now allow direct editable installs, and I see you’ve got some seemingly-unused stuff related to editable in the hook_caller file, but I’m not fully following that as I probably don’t have a grasp on how you’re actually interfacing to the underlying build tool.

Right, this is mentioned in the TODO section of the README.md. I simply haven’t implemented the hook yet. Per the PEPs, we should fall back to a non-editable install if the backend doesn’t implement the editable hooks. I intend for this scenario to be supported.

It does sound like there might still be room here for a Poetry-specific plugin that does let Poetry do some of its dependency- and lockfile-handling magic.

I’m all for enabling new scenarios, but if we’re going to pay the performance costs associated with adherence to the new APIs, we should reap the benefits and not couple ourselves to a single backend. As great as Poetry may be, I’d like to make sure colcon-core remains uncoupled. That said, I do think that implementing “helper” extensions for some specific backends is a great mechanism to improve performance. One of my main concerns with the new APIs is that we don’t have an reliable and efficient way to retrieve metadata like the package name and version, and in a worst-case scenario, we actually need to build the package in order to get that information depending on how many of the optional hooks a backend implements. As with most things colcon, this would be extensible and best implemented outside of colcon-core.

I’d like to encourage anyone playing in this space to read the PEPs and understand where the responsibilities of the backend and frontend meet. A lot of the features you’re discussing here are frontend features and many are even listed as explicitly out-of-scope for colcon’s core packages: https://colcon.readthedocs.io/en/released/developer/design.html#explicitly-out-of-scope