copier: bug: _templates_suffix: "" gives UnicodeDecodeError on binary files ( .ico or .png for example )

Describe the problem

I want to use _templates_suffix: "" The problem is if I use it, then even image files will be processed by copier. And it will give error:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x93 in position 14: invalid start byte

Environment

OS: Ubuntu on WSL2 on Win10
Copier version: copier 6.0.0a6.post43.dev0+2fe3072
Python version: 3.9.6
Installation method: pipx+git

Additional context

    create  .
    create  .dependabot
    create  .dependabot/config.yml
    create  .en v copy
    create  .env
    create  .gitignore
    create  LICENSE
    create  README.md
    create  Banana
    create  Banana/.prettierignore
    create  Banana/backend
    create  Banana/backend/alembic.ini
    create  Banana/backend/app
    create  Banana/backend/app/alembic
    create  Banana/backend/app/alembic/env.py
    create  Banana/backend/app/alembic/README
    create  Banana/backend/app/alembic/script.py.mako
    create  Banana/backend/app/alembic/versions
    create  Banana/backend/app/alembic/versions/91979b40eb38_create_users_table.py
    create  Banana/backend/app/alembic/__init__.py
    create  Banana/backend/app/alembic.ini
    create  Banana/backend/app/api
    create  Banana/backend/app/api/api_v1
    create  Banana/backend/app/api/api_v1/routers
    create  Banana/backend/app/api/api_v1/routers/auth.py
    create  Banana/backend/app/api/api_v1/routers/tests
    create  Banana/backend/app/api/api_v1/routers/tests/test_auth.py
    create  Banana/backend/app/api/api_v1/routers/tests/test_users.py
    create  Banana/backend/app/api/api_v1/routers/tests/__init__.py
    create  Banana/backend/app/api/api_v1/routers/users.py
    create  Banana/backend/app/api/api_v1/routers/__init__.py
    create  Banana/backend/app/api/api_v1/__init__.py
    create  Banana/backend/app/api/dependencies
    create  Banana/backend/app/api/dependencies/__init__.py
    create  Banana/backend/app/api/__init__.py
    create  Banana/backend/app/core
    create  Banana/backend/app/core/auth.py
    create  Banana/backend/app/core/celery_app.py
    create  Banana/backend/app/core/config.py
    create  Banana/backend/app/core/security.py
    create  Banana/backend/app/core/__init__.py
    create  Banana/backend/app/db
    create  Banana/backend/app/db/crud.py
    create  Banana/backend/app/db/models.py
    create  Banana/backend/app/db/schemas.py
    create  Banana/backend/app/db/session.py
    create  Banana/backend/app/db/__init__.py
    create  Banana/backend/app/initial_data.py
    create  Banana/backend/app/main.py
    create  Banana/backend/app/tasks.py
    create  Banana/backend/app/tests
    create  Banana/backend/app/tests/test_main.py
    create  Banana/backend/app/tests/test_tasks.py
    create  Banana/backend/app/tests/__init__.py
    create  Banana/backend/app/__init__.py
    create  Banana/backend/conftest.py
    create  Banana/backend/Dockerfile
    create  Banana/backend/pyproject.toml
    create  Banana/backend/requirements.txt
    create  Banana/docker-compose.yml
    create  Banana/frontend
    create  Banana/frontend/.dockerignore
    create  Banana/frontend/.eslintrc.js
    create  Banana/frontend/.prettierrc.js
    create  Banana/frontend/Dockerfile
    create  Banana/frontend/package.json
    create  Banana/frontend/public
Traceback (most recent call last):
  File "/home/orange/.local/bin/copier", line 8, in <module>
    sys.exit(CopierApp.run())
  File "/home/orange/.local/pipx/venvs/copier/lib/python3.8/site-packages/plumbum/cli/application.py", line 614, in run
    inst, retcode = subapp.run(argv, exit=False)
  File "/home/orange/.local/pipx/venvs/copier/lib/python3.8/site-packages/plumbum/cli/application.py", line 609, in run
    retcode = inst.main(*tailargs)
  File "/home/orange/.local/pipx/venvs/copier/lib/python3.8/site-packages/copier/cli.py", line 70, in _wrapper
    return method(*args, **kwargs)
  File "/home/orange/.local/pipx/venvs/copier/lib/python3.8/site-packages/copier/cli.py", line 293, in main
    self.parent._worker(
  File "/home/orange/.local/pipx/venvs/copier/lib/python3.8/site-packages/copier/main.py", line 575, in run_copy
    self._render_folder(src_abspath)
  File "/home/orange/.local/pipx/venvs/copier/lib/python3.8/site-packages/copier/main.py", line 467, in _render_folder
    self._render_folder(file)
  File "/home/orange/.local/pipx/venvs/copier/lib/python3.8/site-packages/copier/main.py", line 467, in _render_folder
    self._render_folder(file)
  File "/home/orange/.local/pipx/venvs/copier/lib/python3.8/site-packages/copier/main.py", line 467, in _render_folder
    self._render_folder(file)
  File "/home/orange/.local/pipx/venvs/copier/lib/python3.8/site-packages/copier/main.py", line 469, in _render_folder
    self._render_file(file)
  File "/home/orange/.local/pipx/venvs/copier/lib/python3.8/site-packages/copier/main.py", line 429, in _render_file
    tpl = self.jinja_env.get_template(str(src_relpath))
  File "/home/orange/.local/pipx/venvs/copier/lib/python3.8/site-packages/jinja2/environment.py", line 883, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "/home/orange/.local/pipx/venvs/copier/lib/python3.8/site-packages/jinja2/environment.py", line 857, in _load_template
    template = self.loader.load(self, name, globals)
  File "/home/orange/.local/pipx/venvs/copier/lib/python3.8/site-packages/jinja2/loaders.py", line 115, in load
    source, filename, uptodate = self.get_source(environment, name)
  File "/home/orange/.local/pipx/venvs/copier/lib/python3.8/site-packages/jinja2/loaders.py", line 184, in get_source
    contents = f.read().decode(self.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x93 in position 14: invalid start byte

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 1
Comments: 21 (13 by maintainers)

Commits related to this issue

fix: Fix unicode error on binary files If a unicode decode error is raised while trying to compile a file as a Jinja2 template, and the templates suffix is empty, we fallback to copying this file as ... — committed to pawamoy/copier by pawamoy 3 years ago
fix: Fix unicode error on binary files (#436) * fix: Fix unicode error on binary files If a unicode decode error is raised while trying to compile a file as a Jinja2 template, and the templates ... — committed to copier-org/copier by pawamoy 3 years ago

Most upvoted comments

I prefer to keep it simple, consistent and straightforward.

If you really care about what files are templated and what files not, then use a suffix. It’s more predictable and easier to understand, and you don’t have the load of going to the copier.yaml file, reading the specific configuration, and even interpreting the ignore patterns to know why/if a certain file will be a template or not.

yajo on Aug 2, 2021

You can still use raw tags, wrapping the entire contents of a file into it to avoid rendering any of it.

{% raw %}
This will not be rendered. {{ variable }}
{% if condition %}...{% endif %}
{% endraw %}

pawamoy on Aug 1, 2021

I think that the fix is quite simple:

What’s _templates_suffix?
- ""
  - Does it render?
    - Yes: render it
    - No: copy it
- ".jinja" (default for v6+) or anything else
  - Does it have the suffix?
    - No: copy it
    - Yes:
      - Does it render?
        
        Yes: render it
        
        No: raise error.

Having the suffix has its pros and cons, while not having it has other pros and cons. It’s up to the template maintainer to decide, I think that’s outside the scope of this issue. 🤷🏼‍♂️

yajo on Jul 24, 2021

The Better Jinja VSCode extension by Samuel Colvin does wonders 😉

But I agree with you. I did that mistake several times (templating a file and forgetting to add the extension). And I like being able to simply tell Copier to try and render everything. I actually opened the ticket about empty extension suffix and sent the PR to make it possible 😄 The best of both world would be to have automatic detection of templating (and thus Jinja syntax highlighting) without an additional file extension.

About performance: using an extension suffix makes Copier even faster as only templated files pass through the Jinja engine 🙂

Anyway. I’ll stop here and see what @Yajo decides.

pawamoy on Jul 23, 2021

Can’t we just catch the UnicodeDecodeError jinja raises and perform a copy in that case?

yajo on Jul 22, 2021

Ah, true, my bad! Let see how CookieCutter does that then! Oh yeah, they have a _copy_without_render configuration option. And are using from binaryornot.check import is_binary. https://pypi.org/project/binaryornot/

pawamoy on Jul 21, 2021