colcon-core: Colcon fails to build valid PEP 517 projects
As mentioned on https://github.com/colcon/colcon-ros/pull/66 , colcon fails to build valid python packages that use pyproject.toml. This includes packages that want to use a setup.cfg without a corresponding setup.py.
$ colcon build --packages-select test_project
[0.403s] ERROR:colcon.colcon_core.package_identification:ROS package '/home/dan/ros2_ws/src/test_project' with build type 'ament_python' has no 'setup.py' file
[0.405s] WARNING:colcon.colcon_core.package_selection:ignoring unknown package 'test_project' in --packages-select
Summary: 0 packages finished [0.27s]
About this issue
- Original URL
- State: open
- Created 7 years ago
- Reactions: 5
- Comments: 15 (7 by maintainers)
I’ve just made my
colcon-python-projectextensions public: https://github.com/colcon/colcon-python-project/tree/devel They were previously private because we wanted to do some more testing before migrating those extensions intocolcon-coreto eventually become the default but we didn’t want folks to take a dependency on what is explicitly prototype code.The long term plan is to switch
colcon-coreto use “standards-based” tooling exclusively, and move the existing functionality into a new extension to support down-level platforms that can’t support the new extensions (RHEL 8, Bionic, etc).It should be noted that these extensions implement colcon as a “build frontend”, so we aren’t wrapping pip or any of the other build frontends and aren’t coupled to any of their behaviors and limitations. This should allow use of any build backend, though I’ve only tested poetry, setuptools, and flit.
One scenario that isn’t 100% supported yet is using a build backend that is built as part of the same colcon workspace itself. I’ve been toying with introducing a build backend for Ament, and while colcon is able to bootstrap that package itself, we can’t flesh out the full dependency graph without first having built the build backend package, so there is a bootstrapping problem. If you build the backend in a separate workspace, the problem goes away. I’m sure there are other limitations to my current approach, but that one is worth calling out.
re: dependency awareness - my implementation continues colcon’s pattern of being aware of only the dependencies which are part of the colcon workspace.
To use the prototype package on a plain python project, it needs only be installed. To use it on ROS packages, you need to modify the
colcon-rospackage to use the new Python build extension instead of the setuptools-exclusive one. I’ll make a branch with this change and post it here.Since my internal efforts to get some testing done on these extensions have largely stalled, I’d appreciate any testing that you or anyone else can provide. Once we’ve settled on the general architecture, we can move the work to a PR against
colcon-core.I’m open to a synchronous discussion or demo on this work if it sounds helpful. In general, I’m much more optimistic than I was previously, but there are certainly still sharp edges here that we need to address, possibly as documentation and announcements.
Right, this is mentioned in the TODO section of the README.md. I simply haven’t implemented the hook yet. Per the PEPs, we should fall back to a non-editable install if the backend doesn’t implement the editable hooks. I intend for this scenario to be supported.
I’m all for enabling new scenarios, but if we’re going to pay the performance costs associated with adherence to the new APIs, we should reap the benefits and not couple ourselves to a single backend. As great as Poetry may be, I’d like to make sure
colcon-coreremains uncoupled. That said, I do think that implementing “helper” extensions for some specific backends is a great mechanism to improve performance. One of my main concerns with the new APIs is that we don’t have an reliable and efficient way to retrieve metadata like the package name and version, and in a worst-case scenario, we actually need to build the package in order to get that information depending on how many of the optional hooks a backend implements. As with most things colcon, this would be extensible and best implemented outside ofcolcon-core.I’d like to encourage anyone playing in this space to read the PEPs and understand where the responsibilities of the backend and frontend meet. A lot of the features you’re discussing here are frontend features and many are even listed as explicitly out-of-scope for colcon’s core packages: https://colcon.readthedocs.io/en/released/developer/design.html#explicitly-out-of-scope