copier: Updating a project fails when Jinja extension with breaking change is used
Describe the problem
Consider the following situation:
- A template uses v1 of a Jinja extension.
- A project is generated using this template.
- A breaking change happens in the Jinja extension which gets released as v2.
- The template adapts to v2 of the Jinja extension and gets released.
- The project is updated using the new version of the template. This may not be possible.
So what’s the problem? Copier’s update algorithm re-generates a fresh project using the old version of the template (in this case, the one that uses v1 of the Jinja extension) and also generates a fresh project using the new version of the template (in this case, the one that uses v2 of the Jinja extension). But it is only possible to install one version of a package at a time in a virtual env, either v1 or v2 of the Jinja extension. So when v1 is installed, re-generating a fresh project using the old version of the template works but generating a fresh project using the new version of the template fails. And when v2 is installed, generating a fresh project using the new version of the template works but re-generating a fresh project using the old version of the template fails.
Template
See the next section for a complete test case.
To Reproduce
Add this test case to a new file, e.g. tests/test_jinja_extension_breaking_change.py, in Copier’s test suite:
import sys
from pathlib import Path
from textwrap import dedent
import pytest
from plumbum import local
from plumbum.cmd import git
from copier.main import run_copy, run_update
from .helpers import build_file_tree
def test_update_with_jinja_extension_breaking_change(
tmp_path_factory: pytest.TempPathFactory, monkeypatch: pytest.MonkeyPatch
) -> None:
src, dst, ext = map(tmp_path_factory.mktemp, ("src", "dst", "ext"))
monkeypatch.setattr("sys.path", [str(ext)] + sys.path)
with local.cwd(ext):
Path("test_ext").mkdir()
Path("test_ext", "__init__.py").write_text(
dedent(
"""\
from jinja2.ext import Extension
def to_slug(value):
return value.replace(" ", "-")
class TestExt(Extension):
def __init__(self, environment):
super().__init__(environment)
environment.filters["to_slug"] = to_slug
"""
)
)
with local.cwd(src):
build_file_tree(
{
"copier.yml": (
"""\
_jinja_extensions:
- test_ext.TestExt
version:
type: str
"""
),
"{{ _copier_conf.answers_file }}.jinja": "{{ _copier_answers | to_yaml }}",
"version.txt.jinja": "{{ version }}",
"output.txt.jinja": "{{ ('hello world ' + version) | to_slug }}",
}
)
git("init")
git("add", ".")
git("commit", "-m1")
git("tag", "v1")
run_copy(str(src), dst, data={"version": "v1"})
assert "_commit: v1" in (dst / ".copier-answers.yml").read_text()
assert (dst / "version.txt").read_text() == "v1"
assert (dst / "output.txt").read_text() == "hello-world-v1"
with local.cwd(dst):
git("init")
git("add", ".")
git("commit", "-m1")
with local.cwd(ext / "test_ext"):
Path("__init__.py").write_text(
Path("__init__.py").read_text().replace("to_slug", "slugify")
)
with local.cwd(src):
build_file_tree(
{
"output.txt.jinja": "{{ ('hello world ' + version) | slugify }}",
}
)
git("add", ".")
git("commit", "-m2")
git("tag", "v2")
run_update(dst, data={"version": "v2"}, overwrite=True)
assert "_commit: v2" in (dst / ".copier-answers.yml").read_text()
assert (dst / "version.txt").read_text() == "v2"
assert (dst / "output.txt").read_text() == "hello-world-v2"
Then, run tests/test_jinja_extension_breaking_change.py and observe that it fails.
Logs
...
> run_update(dst, data={"version": "v2"}, overwrite=True)
tests/test_jinja_extension_breaking_change.py:87:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
copier/main.py:939: in run_update
with Worker(dst_path=Path(dst_path), **kwargs) as worker:
copier/main.py:198: in __exit__
raise value
copier/main.py:940: in run_update
worker.run_update()
copier/main.py:768: in run_update
self._apply_update()
copier/main.py:822: in _apply_update
self.run_copy()
copier/main.py:700: in run_copy
self._render_folder(src_abspath)
copier/main.py:604: in _render_folder
self._render_file(file)
copier/main.py:521: in _render_file
tpl = self.jinja_env.get_template(src_relpath)
.venv/lib/python3.10/site-packages/jinja2/environment.py:1010: in get_template
return self._load_template(name, globals)
.venv/lib/python3.10/site-packages/jinja2/environment.py:969: in _load_template
template = self.loader.load(self, name, self.make_globals(globals))
.venv/lib/python3.10/site-packages/jinja2/loaders.py:138: in load
code = environment.compile(source, name, filename)
.venv/lib/python3.10/site-packages/jinja2/environment.py:768: in compile
self.handle_exception(source=source_hint)
.venv/lib/python3.10/site-packages/jinja2/environment.py:936: in handle_exception
raise rewrite_traceback_stack(source=source)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E jinja2.exceptions.TemplateAssertionError: No filter named 'slugify'.
/tmp/copier.vcs.clone.6a_6fzf8/output.txt.jinja:1: TemplateAssertionError
Expected behavior
The update process should succeed.
Screenshots/screencasts/logs
No response
Operating system
Linux
Operating system distribution and version
Ubuntu 22.04
Copier version
ff5aa056a0f4b265aa0aad08c196c6a978e9f5c2
Python version
CPython 3.10
Installation method
poetry+git
Additional context
I’ve come to think that re-generating a fresh project using the old version of the template is bad. Instead, the fresh project using the old version of the template should be saved and retrieved during the update process. I haven’t figured out a concrete solution yet, but I wonder whether custom Git references might be useful (along the lines of https://iterative.ai/blog/experiment-refs). Perhaps custom Git refs could point to hidden commits, each of which contains a fresh project based on a specific version of a template. Then, those custom Git refs would be pushed as well and Copier’s update algorithm could retrieve the fresh project from there instead of re-generating the project. But I haven’t tried this out yet nor have I got experience with using custom Git refs. If anybody has a good idea, I’m all ears. 🙏
In fact, the problem isn’t limited to Jinja extensions, but it’s the most likely situation. Also, for example, a breaking change in the Jinja library itself might lead to this problem.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 18 (18 by maintainers)
Commits related to this issue
- docs: clarify how to use recopy to recover from broken updates Docs were a bit repetitive and unclear. Targets https://github.com/copier-org/copier/issues/1170. It could even fix it? — committed to copier-org/copier by yajo 3 years ago
- docs: clarify how to use recopy to recover from broken updates Docs were a bit repetitive and unclear. Targets https://github.com/copier-org/copier/issues/1170. It could even fix it? Co-authore... — committed to copier-org/copier by yajo 3 years ago
I think Copier could use a custom refs namespace (like DVC), e.g.
refs/copier, and therein simply use the value from the_commitfield in the answers file as the ref name. But I haven’t tried this out yet, just an idea based on my current understanding of what DVC is doing. Then, we wouldn’t need an entry in the answers file for the replay cache.Well, this problem is inherent to the design and feature set of Copier, so I think it’s time to settle wether it’s just a known issue we can live with or there’s something doable.
Possible solutions:
Already explained: document (https://github.com/copier-org/copier/pull/1218) that, if copier cannot replay for whatever reason, you should recopy and solve problems manually.
Make copier build the environment where it’s running. I think it’s obvious that we’re not gonna do that. There are many ways to install Copier, and doing this would convert it into a chicken-and-egg self-bootstrapping package manager. That’s a path I dont wanna go for obvious reasons. You install Copier and then it runs, not the other way around.
Record the path to the executable used in last play. This would need a specific environment to work with. For example, if you install Copier with pipx and I install it with nix, none of us could find the other’s executable path. And if we did, only you would be able to replay my render, not the other way around. The only environment where a path is a guaranteed reproducible environment is nix, and still you need the source of that path. Thus it doesn’t seem like the best option.
Allow the template to specify the path to a reproducible executable. For example, if my template includes an output that produces a reproducible executable like I showcased in https://github.com/orgs/copier-org/discussions/1130#discussioncomment-5890503, I can tell copier how to reproduce the URL to that executable (it should be based in the git commit, of course, which is the only reproducibility source I have) and do the replay running that. In practical terms, this would be only useful for nix or docker executions.
Store a cache of replays in local disk. That would reduce the chances to get a broken replay, and speeding things up. On CI, you can still use caches or artifacts to pass that cache across jobs.
Store that cache on some git ref in the subproject origin repo. It seems to me like it would create a lot of unwanted git garbage. Maybe you can specify a repo for caches, which could or not be the same as the origin.
To set thing clear, I don’t have any willingness to implement any of the solutions because for me, number 1 just works and is good enough. Solutions 2 and 3 are clearly no-go. Among the others, what do you prefer?