copier: ⚠ CRITICAL: Copier 9.1.1 broken by too loose `pyyaml-include` requirement ⚠

Describe the problem

Yesterday, pyyaml-include 2.0 was released. Unsurprisingly for a non-CalVer major release, it stabilized a breaking change: the pyyaml-include’s package namespace yamlinclude was moved to yaml_include (ref).

The latest Copier release, 9.1.1, was released prior to removing the pyyaml-include dependency in 037d7f0. Back then, the pyproject.toml version specification for pyyaml-include was ">=1.2": https://github.com/copier-org/copier/blob/b3ed48d7dda94e700317226018bec37ac244e529/pyproject.toml#L43

which has been welcoming Copier-incompatible pyyaml-include==2.0 with open arms in every installation since the yesterday’s pyyaml-include release.

Copier is now unusable in every environment that installed Copier upon the pyyaml-include 2.0 release (including my today’s CI environment that adopts CTT for testing my Copier template), unless one pins the legacy pyyaml-include<2 explicitly as a direct, non-transitive dependency.

Template

Doesn’t apply.

To Reproduce

  1. pip install copier
  2. copier

Logs

~/dev/skeleton git:(main|)❯❯❯ copier
Traceback (most recent call last):
  File "/home/bswck/.local/bin/copier", line 5, in <module>
    from copier.__main__ import copier_app_run
  File "/home/bswck/.local/pipx/venvs/copier/lib/python3.10/site-packages/copier/__init__.py", line 6, in <module>
    from .main import *  # noqa: F401,F403
  File "/home/bswck/.local/pipx/venvs/copier/lib/python3.10/site-packages/copier/main.py", line 46, in <module>
    from .subproject import Subproject
  File "/home/bswck/.local/pipx/venvs/copier/lib/python3.10/site-packages/copier/subproject.py", line 15, in <module>
    from .template import Template
  File "/home/bswck/.local/pipx/venvs/copier/lib/python3.10/site-packages/copier/template.py", line 20, in <module>
    from yamlinclude import YamlIncludeConstructor
ModuleNotFoundError: No module named 'yamlinclude'

Expected behavior

No errors.

Screenshots/screencasts/logs

No response

Operating system

Linux

Operating system distribution and version

Ubuntu 23.04

Copier version

9.1.1

Python version

3.10

Installation method

pipx+pypi

Additional context

Workaround: Pinning the legacy pyyaml-include<2 directly.

I strongly suggest pinning all versions properly to prevent unexpected breaking changes in the future and make Copier more reliable. The current pyproject.toml file might expose Copier to many, many more identical pitfalls.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 21
  • Comments: 22 (10 by maintainers)

Commits related to this issue

Most upvoted comments

I’d vote for a quick release from master if possible indeed 😃

Hello! Sorry everybody. Since some weeks ago, gmail decided that copier issues were spam and I lost every message 😆

FWIW one advantage of not pinning versions is that we let you pin them downstream. Whatever packaging system you use, it should have a way to downgrade a package. Just do it and it will serve as a workaround for older releases.

Let me push a new release without pyyaml-include, which we don’t require in master anymore.

I agree with @henryiii, deprecation periods with warnings are the most reliable way to inform downstream users that something is about to break.

Regarding new releases, @sisp and I agree on releasing a 9.x release quickly, and I believe we have enough privileges to do that (pushing a tag), but @yajo is our captain and we’d like to hear from him first 🙂 For a 8.x release, I’m ready to consider it myself, exceptionally. Again, needs blessing from @yajo.

FWIW, pyyaml-include should have had a deprecation period where both namespaces were provided and the one that was to be removed produced a warning. Making a sudden breaking change without warning is not a good idea in the Python ecosystem. There was a beta release, but unless you pass --prerelease, you don’t see it (not a bad idea for CI).

Edit: in #1225 (comment) I see the pipx inject ... command is used to specify a particular version of a sub-dependency. Is inject preferred? If so, it should be pipx inject copier 'pyyaml-include<2' instead of my suggestion!

Yeah, pipx inject copier 'pyyaml-include<2' would ensure that Copier’s autonomous pipx venv sees pyyaml-include<2, that’s the best approach for pipx IMO.

I’d keep only lower bounds because upper bounds can cause unresolvable dependency trees. Major releases in SemVer-versioned projects may break dependants but often don’t. A missing upper bound like in this case can be worked around in userland by adding a constraint – see, e.g., https://github.com/copier-org/copier/issues/1225#issuecomment-1616656418.

Ideally, we should temporarily upper-bound pyyaml-include, make a patch release to fix the incompatibility that exists now, and remove the upper bound again along with a proper fix afterwards (see, e.g., #1244 and #1229 for a similar situation with Pydantic). But like many open-source projects do we not backport fixes, and the current master branch does not depend on pyyaml-include anymore. I think master is stable. Should we release v9.2.0 ASAP, @yajo?

8.x branch is broken too. Would you consider a patch on 8.x for those who cannot upgrade to 9.x yet?

To add to https://github.com/copier-org/copier/issues/1225#issuecomment-1616656418 regarding userland workarounds, with Rye, you can do rye install copier --extra-requirement "pyyaml-include<2"

~For those of you who installed copier using pipx, running pipx install copier --force --pip-args "pyyaml-include<2" seems to work.~

Edit: in https://github.com/copier-org/copier/issues/1225#issuecomment-1616656418 I see the pipx inject ... command is used to specify a particular version of a sub-dependency. Is inject preferred? If so, it should be pipx inject copier 'pyyaml-include<2' instead of my suggestion!

Adding that we’re pinned to 0.8.30 for other reasons and this affects us as well

I’d been fully aware of troubles that come from upper bounds and slowly adapt my projects to best practices. Again, I don’t insist on bringing them back in long term. Regardless, thanks for recommending a great article though!

Upper bounds were explicitly removed before IIRC, because they usually cause conflicts in the Python ecosystem. But since Copier uses Dependabot/Renovatebot, I suppose we could add back upper bounds on major versions. I’d strongly recommend adding some CI jobs to test early on unaccepted versions if we go this route again!

I’d only like to note here that I provided a temporary workaround for troubled users, not a solution for Copier. The best solution would definitely be either cutting a new release compatible with 2 or, if we’re sure that a post-2 version introduces some backward compatibility (which I doubt), specifying !=2.0 in a new Copier release.