copier: ⚠ CRITICAL: Copier 9.1.1 broken by too loose `pyyaml-include` requirement ⚠
Describe the problem
Yesterday, pyyaml-include 2.0 was released. Unsurprisingly for a non-CalVer major release, it stabilized a breaking change: the pyyaml-include’s package namespace yamlinclude was moved to yaml_include (ref).
The latest Copier release, 9.1.1, was released prior to removing the pyyaml-include dependency in 037d7f0.
Back then, the pyproject.toml version specification for pyyaml-include was ">=1.2":
https://github.com/copier-org/copier/blob/b3ed48d7dda94e700317226018bec37ac244e529/pyproject.toml#L43
which has been welcoming Copier-incompatible pyyaml-include==2.0 with open arms in every installation since the yesterday’s pyyaml-include release.
Copier is now unusable in every environment that installed Copier upon the pyyaml-include 2.0 release (including my today’s CI environment that adopts CTT for testing my Copier template), unless one pins the legacy pyyaml-include<2 explicitly as a direct, non-transitive dependency.
Template
Doesn’t apply.
To Reproduce
pip install copiercopier
Logs
~/dev/skeleton git:(main|)❯❯❯ copier
Traceback (most recent call last):
File "/home/bswck/.local/bin/copier", line 5, in <module>
from copier.__main__ import copier_app_run
File "/home/bswck/.local/pipx/venvs/copier/lib/python3.10/site-packages/copier/__init__.py", line 6, in <module>
from .main import * # noqa: F401,F403
File "/home/bswck/.local/pipx/venvs/copier/lib/python3.10/site-packages/copier/main.py", line 46, in <module>
from .subproject import Subproject
File "/home/bswck/.local/pipx/venvs/copier/lib/python3.10/site-packages/copier/subproject.py", line 15, in <module>
from .template import Template
File "/home/bswck/.local/pipx/venvs/copier/lib/python3.10/site-packages/copier/template.py", line 20, in <module>
from yamlinclude import YamlIncludeConstructor
ModuleNotFoundError: No module named 'yamlinclude'
Expected behavior
No errors.
Screenshots/screencasts/logs
No response
Operating system
Linux
Operating system distribution and version
Ubuntu 23.04
Copier version
9.1.1
Python version
3.10
Installation method
pipx+pypi
Additional context
Workaround: Pinning the legacy pyyaml-include<2 directly.
I strongly suggest pinning all versions properly to prevent unexpected breaking changes in the future and make Copier more reliable.
The current pyproject.toml file might expose Copier to many, many more identical pitfalls.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 21
- Comments: 22 (10 by maintainers)
Commits related to this issue
- Pin `pyyaml-include` for now copier-org/copier#1568 — committed to softboiler/boilercv by blakeNaccarato 3 years ago
- Temporary workaround for copier-org/copier#1568 — committed to hexdoc-dev/hexdoc by object-Object 3 years ago
- deps: temporary fix copier breaks with pyyaml-include v2 (copier-org/copier#1568) — committed to pwwang/immunopipe by pwwang 3 years ago
- 1.3.3 (#51) * tests: init tests * deps: bump biopipen to 0.27.2 * deps: temporary fix copier breaks with pyyaml-include v2 (copier-org/copier#1568) * docs: update FAQ.md with instructions fo... — committed to pwwang/immunopipe by pwwang 3 years ago
- deps: temporary fix copier breaks with pyyaml-include v2 (copier-org/copier#1568) — committed to pwwang/biopipen by pwwang 3 years ago
- 0.27.3 (#133) * deps: temporary fix copier breaks with pyyaml-include v2 (copier-org/copier#1568) * deps: bump pipen-poplog to 0.1.1 * deps: bump pipen-poplog to 0.1.2 * 0.27.3 * choir(sc... — committed to pwwang/biopipen by pwwang 3 years ago
- Use old pyyaml-include to avoid copier bug. See <https://github.com/copier-org/copier/issues/1568>. This should be fixed in a new version of copier in the next few days. — committed to teddygroves/bibat by teddygroves 3 years ago
- build: 👷 fix copier dependency End-to-end tests were broken because of the release of pyyaml-include 2.0 and a too loose dependency in copier. This was fixed in copier 9.2.0. See https://github.com/... — committed to xaf/qolsysgw by xaf 3 years ago
I’d vote for a quick release from master if possible indeed 😃
Fixed in https://github.com/copier-org/copier/releases/tag/v9.2.0.
Hello! Sorry everybody. Since some weeks ago, gmail decided that copier issues were spam and I lost every message 😆
FWIW one advantage of not pinning versions is that we let you pin them downstream. Whatever packaging system you use, it should have a way to downgrade a package. Just do it and it will serve as a workaround for older releases.
Let me push a new release without pyyaml-include, which we don’t require in master anymore.
I agree with @henryiii, deprecation periods with warnings are the most reliable way to inform downstream users that something is about to break.
Regarding new releases, @sisp and I agree on releasing a 9.x release quickly, and I believe we have enough privileges to do that (pushing a tag), but @yajo is our captain and we’d like to hear from him first 🙂 For a 8.x release, I’m ready to consider it myself, exceptionally. Again, needs blessing from @yajo.
FWIW, pyyaml-include should have had a deprecation period where both namespaces were provided and the one that was to be removed produced a warning. Making a sudden breaking change without warning is not a good idea in the Python ecosystem. There was a beta release, but unless you pass
--prerelease, you don’t see it (not a bad idea for CI).Yeah,
pipx inject copier 'pyyaml-include<2'would ensure that Copier’s autonomous pipx venv seespyyaml-include<2, that’s the best approach for pipx IMO.I’d keep only lower bounds because upper bounds can cause unresolvable dependency trees. Major releases in SemVer-versioned projects may break dependants but often don’t. A missing upper bound like in this case can be worked around in userland by adding a constraint – see, e.g., https://github.com/copier-org/copier/issues/1225#issuecomment-1616656418.
Ideally, we should temporarily upper-bound
pyyaml-include, make a patch release to fix the incompatibility that exists now, and remove the upper bound again along with a proper fix afterwards (see, e.g., #1244 and #1229 for a similar situation with Pydantic). But like many open-source projects do we not backport fixes, and the currentmasterbranch does not depend onpyyaml-includeanymore. I thinkmasteris stable. Should we release v9.2.0 ASAP, @yajo?8.x branch is broken too. Would you consider a patch on 8.x for those who cannot upgrade to 9.x yet?
To add to https://github.com/copier-org/copier/issues/1225#issuecomment-1616656418 regarding userland workarounds, with Rye, you can do
rye install copier --extra-requirement "pyyaml-include<2"~For those of you who installed
copierusingpipx, runningpipx install copier --force --pip-args "pyyaml-include<2"seems to work.~Edit: in https://github.com/copier-org/copier/issues/1225#issuecomment-1616656418 I see the
pipx inject ...command is used to specify a particular version of a sub-dependency. Isinjectpreferred? If so, it should bepipx inject copier 'pyyaml-include<2'instead of my suggestion!Adding that we’re pinned to 0.8.30 for other reasons and this affects us as well
I’d been fully aware of troubles that come from upper bounds and slowly adapt my projects to best practices. Again, I don’t insist on bringing them back in long term. Regardless, thanks for recommending a great article though!
I’d only like to note here that I provided a temporary workaround for troubled users, not a solution for Copier. The best solution would definitely be either cutting a new release compatible with 2 or, if we’re sure that a post-2 version introduces some backward compatibility (which I doubt), specifying
!=2.0in a new Copier release.