kedro: 0.19.1 `kedro new` failes to create new project. Raises KedroCliError: Failed to generate project when running cookiecutter

Description

Creating a new kedro project via kedro new --name my_project --verbose fails with the following error:

kedro.framework.cli.utils.KedroCliError: Failed to generate project when running cookiecutter. (full trace below)

Context

I am trying to initiate a new project in a ‘clean’ conda environment with kedro==0.19.1. I have no other environments (except base which has only defaults installed), and no other kedro installs.
Cookiecutter is 2.5.0 (installs with kedro)
I’m on Windows 10

Issue reproduces on kedro==0.19.0. Issue is absent on `kedro==0.18.14.

Steps to Reproduce

conda create --name my_env python=3.11
conda activate my_env
(my_env) $ kedro new --name my_project–verbose
select any/all/none tools (it fails on all options)
select yes/no for example pipeline (it fails on both options )

Expected Result

A base new project should be created.

Actual Result

Project setup fails in Cookiecutter afaict.

Would you like to include an example pipeline? :
 (no): yes
[01/21/24 20:56:13] ERROR    Stopping generation because post_gen_project hook script didn't exit successfully                                                          hooks.py:159
Traceback (most recent call last):
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\kedro\framework\cli\starters.py", line 864, in _create_project
    result_path = cookiecutter(template=template_path, **cookiecutter_args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\cookiecutter\main.py", line 182, in cookiecutter
    result = generate_files(
             ^^^^^^^^^^^^^^^
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\cookiecutter\generate.py", line 420, in generate_files
    run_hook_from_repo_dir(
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\cookiecutter\hooks.py", line 152, in run_hook_from_repo_dir
    run_hook(hook_name, project_dir, context)
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\cookiecutter\hooks.py", line 135, in run_hook
    run_script_with_context(script, project_dir, context)
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\cookiecutter\hooks.py", line 118, in run_script_with_context
    run_script(temp.name, cwd)
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\cookiecutter\hooks.py", line 89, in run_script
    raise FailedHookException(
cookiecutter.exceptions.FailedHookException: Hook script failed (exit status: 1)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\kedro\framework\cli\starters.py", line 268, in new
    _create_project(project_template, cookiecutter_args)
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\kedro\framework\cli\starters.py", line 866, in _create_project
    raise KedroCliError(
kedro.framework.cli.utils.KedroCliError: Failed to generate project when running cookiecutter.

Your Environment

Kedro version used (pip show kedro or kedro -V): 0.19.1, 0.19.0
Python version used (python -V): 3.11.7, 3.10.13
Operating system and version: Windows 10, build 19045.3930
conda version: 23.7.4

About this issue

Original URL
State: closed
Created 5 months ago
Reactions: 1
Comments: 23 (14 by maintainers)

Most upvoted comments

But maybe we’re onto something?

Finally, I think so!

I’ve been running kedro new ... via my Anaconda Powershell
on windows, Popen(..., shell=True) is calling %COMSPEC% -> default C:\Windows\System32\cmd.exe
I’d never normally run cmd.exe (powershell is miserable enough…)
Unrelated - I found out last night - my C:\Windows\System32\cmd.exe is totally broken. No idea why or what happened.

So. When I’ve been calling script_command = ['C:\\Users\\bcech\\anaconda3\\envs\\kedro311\\python.exe', 'C:\\Users\\bcech\\AppData\\Local\\Temp\\tmpw31svgdv.py'] manually (in my conda powershell), everything is hunky-dorey. But the subprocess.Popen(...) call is trying to go to cmd.exe, which is indeed failing. This makes sense that shell=False works fine – python is called directly and the script runs as usual.

I’ll fix that (my cmd.exe) when I’m able and confirm back here.

But, in the meantime, I’ll close this bug now. So sorry to litter up your Issue board with what is ultimately not looking like a Kedro bug at all. Thank you so much for your persistence on this.

What a wild ride.

BrianCechmanek on Jan 29, 2024

So happy we got to the bottom of this! And no need to apologize @BrianCechmanek, I know how annoying these rabbit holes can get and glad that I could help you debug this nasty issue.

Maybe as a tribute do you want to send a PR to cookiecutter to amend this clunky setting? 😄

astrojuanlu on Jan 29, 2024

I can confirm that I am facing a similar issue on windows 11, python=3.9.15, kedro==0.19.2. It looks like there is some tricky conversion on tools which sometimes generate an invalid pyproject.toml by removing quotes around the "tools": "['None']", don’t know why.

Edit: Fun fact: it works if I always press enter without writing anything on each entry.

Galileo-Galilei on Jan 28, 2024

Thanks @astrojuanlu

I’ll have time in a day or two to debug it a bit more. I’ll see if it’s more one of Conda, Windows, Cookiecutter.

BrianCechmanek on Jan 22, 2024

I’m not using python 3.12 (as it’s not supported yet). I can reproduce this issue on python 3.11 and 3.10. On Windows.

conda info -a:


     active environment : kedro311
    active env location : C:\Users\bcech\anaconda3\envs\kedro311
            shell level : 2
       user config file : C:\Users\bcech\.condarc
 populated config files : C:\Users\bcech\.condarc
          conda version : 23.7.4
    conda-build version : 3.26.1
         python version : 3.11.5.final.0
       virtual packages : __archspec=1=x86_64
                          __cuda=12.3=0
                          __win=0=0
       base environment : C:\Users\bcech\anaconda3  (writable)
      conda av data dir : C:\Users\bcech\anaconda3\etc\conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\bcech\anaconda3\pkgs
                          C:\Users\bcech\.conda\pkgs
                          C:\Users\bcech\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\bcech\anaconda3\envs
                          C:\Users\bcech\.conda\envs
                          C:\Users\bcech\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/23.7.4 requests/2.31.0 CPython/3.11.5 Windows/10 Windows/10.0.19041 aau/0.4.2 c/...
          administrator : False
             netrc file : None
           offline mode : False


# conda environments:
#
base                     C:\Users\bcech\anaconda3
kedro311              *  C:\Users\bcech\anaconda3\envs\kedro311
nanogpt                  C:\Users\bcech\anaconda3\envs\nanogpt

sys.version: 3.11.5 | packaged by Anaconda, Inc. | (m...
sys.prefix: C:\Users\bcech\anaconda3
sys.executable: C:\Users\bcech\anaconda3\python.exe
conda location: C:\Users\bcech\anaconda3\Lib\site-packages\conda
conda-build: C:\Users\bcech\anaconda3\Scripts\conda-build.exe
conda-content-trust: C:\Users\bcech\anaconda3\Scripts\conda-content-trust.exe
conda-convert: C:\Users\bcech\anaconda3\Scripts\conda-convert.exe
conda-debug: C:\Users\bcech\anaconda3\Scripts\conda-debug.exe
conda-develop: C:\Users\bcech\anaconda3\Scripts\conda-develop.exe
conda-env: C:\Users\bcech\anaconda3\Scripts\conda-env.exe
conda-index: C:\Users\bcech\anaconda3\Scripts\conda-index.exe
conda-inspect: C:\Users\bcech\anaconda3\Scripts\conda-inspect.exe
conda-metapackage: C:\Users\bcech\anaconda3\Scripts\conda-metapackage.exe
conda-pack: C:\Users\bcech\anaconda3\Scripts\conda-pack.exe
conda-render: C:\Users\bcech\anaconda3\Scripts\conda-render.exe
conda-repo: C:\Users\bcech\anaconda3\Scripts\conda-repo.exe
conda-server: C:\Users\bcech\anaconda3\Scripts\conda-server.exe
conda-skeleton: C:\Users\bcech\anaconda3\Scripts\conda-skeleton.exe
conda-token: C:\Users\bcech\anaconda3\Scripts\conda-token.exe
conda-verify: C:\Users\bcech\anaconda3\Scripts\conda-verify.exe
user site dirs: C:\Users\bcech\AppData\Roaming\Python\Python37
                C:\Users\bcech\AppData\Roaming\Python\Python38
                C:\Users\bcech\AppData\Roaming\Python\share

CIO_TEST: <not set>
CONDA_DEFAULT_ENV: kedro311
CONDA_EXE: C:\Users\bcech\anaconda3\Scripts\conda.exe
CONDA_PREFIX: C:\Users\bcech\anaconda3\envs\kedro311
CONDA_PREFIX_1: C:\Users\bcech\anaconda3
CONDA_PROMPT_MODIFIER: (kedro311)
CONDA_PYTHON_EXE: C:\Users\bcech\anaconda3\python.exe
CONDA_ROOT: C:\Users\bcech\anaconda3
CONDA_SHLVL: 2
CUDA_PATH: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.3
CURL_CA_BUNDLE: <not set>
HOMEPATH: \Users\bcech
LD_PRELOAD: <not set>
NVTOOLSEXT_PATH: C:\Program Files\NVIDIA Corporation\NvToolsExt\
PATH: C:\Users\bcech\anaconda3\envs\kedro311;C:\Users\bcech\anaconda3\envs\kedro311\Library\mingw-w64\bin;C:\Users\bcech\anaconda3\envs\kedro311\Library\usr\bin;C:\Users\bcech\anaconda3\envs\kedro311\Library\bin;C:\Users\bcech\anaconda3\envs\kedro311\Scripts;C:\Users\bcech\anaconda3\envs\kedro311\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.3\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.3\libnvvp;C:\Program Files\ffmpeg\bin;C:\Users\bcech\anaconda3\condabin;C:\Program Files (x86)\Razer Chroma SDK\bin;C:\Program Files\Razer Chroma SDK\bin;C:\Program Files (x86)\Razer\ChromaBroadcast\bin;C:\Program Files\Razer\ChromaBroadcast\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\Git\cmd;C:\Windows\system32\config\systemprofile\AppData\Local\Microsoft\WindowsApps;C:\Program Files\CMake\bin;C:\Program Files\PuTTY;C:\Program Files (x86)\Arm GNU Toolchain arm-none-eabi\11.2 2022.02\bin;C:\Users\bcech\AppData\Local\Microsoft\WindowsApps;C:\Users\bcech\AppData\Local\Programs\Microsoft VS Code\bin;C:\Program Files (x86)\Plantronics\Spokes3G;C:\Program Files\dotnet;C:\Program Files\NVIDIA Corporation\Nsight Compute 2022.2.0;C:\Users\bcech\AppData\Roaming\nvm;C:\Program Files\nodejs;C:\Users\bcech\AppData\Local\Pub\Cache\bin;C:\Program Files (x86)\Arm GNU Toolchain arm-none-eabi\11.2 2022.02\bin;C:\Users\bcech\AppData\Local\Microsoft\WindowsApps;C:\Users\bcech\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\bcech\AppData\Roaming\nvm;C:\Program Files\nodejs;C:\src\flutter\bin;C:\Users\bcech\AppData\Local\Pub\Cache\bin;.
PSMODULEPATH: C:\Users\bcech\Documents\WindowsPowerShell\Modules;C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
REQUESTS_CA_BUNDLE: <not set>
SSL_CERT_FILE: C:\Users\bcech\anaconda3\envs\kedro311\Library\ssl\cacert.pem

conda env export --from-history :

simple history was: $ conda create --name kedro311 python=3.11 and then (kedro311) $ python -m pip install kedro kedro-datasets kedro-viz numpy pandas torch ipykernel

(kedro311) PS C:\Users\bcech\Documents\Projects\sandbox> conda env export --from-history
name: kedro311
channels:
  - defaults
dependencies:
  - python=3.11
prefix: C:\Users\bcech\anaconda3\envs\kedro311

and lastly, pip freeze:

(kedro311) PS C:\Users\bcech\Documents\Projects\sandbox> python -m pip freeze
aiobotocore==2.11.0
aiofiles==23.2.1
aiohttp==3.9.1
aioitertools==0.11.0
aiosignal==1.3.1
antlr4-python3-runtime==4.9.3
anyio==3.7.1
arrow==1.3.0
asttokens==2.4.1
attrs==23.2.0
binaryornot==0.4.4
botocore==1.34.22
build==1.0.3
cachetools==5.3.2
certifi==2023.11.17
chardet==5.2.0
charset-normalizer==3.3.2
click==8.1.7
colorama==0.4.6
comm==0.2.1
cookiecutter==2.5.0
debugpy==1.8.0
decorator==5.1.1
dynaconf==3.2.4
executing==2.0.1
fastapi==0.109.0
filelock==3.13.1
frozenlist==1.4.1
fsspec==2023.12.2
gitdb==4.0.11
GitPython==3.1.41
graphql-core==3.2.3
greenlet==3.0.3
h11==0.14.0
httptools==0.6.1
idna==3.6
importlib-metadata==7.0.1
importlib-resources==6.1.1
ipykernel==6.29.0
ipython==8.20.0
jedi==0.19.1
Jinja2==3.1.3
jmespath==1.0.1
jupyter_client==8.6.0
jupyter_core==5.7.1
kedro==0.19.2
kedro-datasets==2.0.0
kedro-viz==7.0.0
lazy_loader==0.3
markdown-it-py==3.0.0
MarkupSafe==2.1.4
matplotlib-inline==0.1.6
mdurl==0.1.2
more-itertools==10.2.0
mpmath==1.3.0
multidict==6.0.4
nest-asyncio==1.6.0
networkx==3.2.1
numpy==1.26.3
omegaconf==2.3.0
orjson==3.9.12
packaging==23.2
pandas==2.2.0
parse==1.20.0
parso==0.8.3
platformdirs==4.1.0
plotly==5.18.0
pluggy==1.3.0
pre-commit-hooks==4.5.0
prompt-toolkit==3.0.43
psutil==5.9.8
pure-eval==0.2.2
pydantic==1.10.14
Pygments==2.17.2
pyproject_hooks==1.0.0
python-dateutil==2.8.2
python-dotenv==1.0.0
python-slugify==8.0.1
pytoolconfig==1.3.1
pytz==2023.3.post1
pywin32==306
PyYAML==6.0.1
pyzmq==25.1.2
requests==2.31.0
rich==13.7.0
rope==1.12.0
ruamel.yaml==0.18.5
ruamel.yaml.clib==0.2.8
s3fs==2023.12.2
secure==0.3.0
six==1.16.0
smmap==5.0.1
sniffio==1.3.0
SQLAlchemy==2.0.25
stack-data==0.6.3
starlette==0.35.1
strawberry-graphql==0.217.1
sympy==1.12
tenacity==8.2.3
text-unidecode==1.3
toml==0.10.2
toposort==1.10
torch==2.1.2
tornado==6.4
traitlets==5.14.1
types-python-dateutil==2.8.19.20240106
typing_extensions==4.9.0
tzdata==2023.4
urllib3==2.0.7
uvicorn==0.22.0
watchfiles==0.21.0
watchgod==0.8.2
wcwidth==0.2.13
websockets==12.0
wrapt==1.16.0
yarl==1.9.4
zipp==3.17.0

BrianCechmanek on Jan 22, 2024

This was fixed in 0.19.2 (confirmed by another user), I’m closing @BrianCechmanek but if an upgrade still doesn’t do the trick please leave a comment and we’ll reconsider.

astrojuanlu on Jan 22, 2024