poetry: Poetry with `virtualenvs.create false` installs packages into the wrong place on Ubuntu

Adding to the argument by saying that I’d very much like to use official tensorflow docker images and they use system python. When adding my custom packages to the default install through poetry and virtualenvs.create=false it results in the bug described here.

Using a virtual environment would defeat the purpose of using an image with everything pre-installed (including jupyter notebook).

+1 for this issue. We shouldn’t need to use a virtual environment inside a Docker image. I’m facing the issue when using an ubuntu:22.04 base image with poetry==1.3.1. Making the setting poetry config virtualenvs.create false and then installing packages via poetry install results in packages being install in the wrong location. This was not an issue in poetry==1.1.14.

Same issue for me

I’m running into this issue as well on Fedora 36

Honestly, this is such an edge case (and mostly the result of bad Debian packaging) that I’m not sure there’s much value in fixing it – it’s easily resolved if you make use of a self-compiled Python or virtual environments (there is no reason to avoid them in a container, and doing so exposes you to all sorts of sharp edges like this).

I see that the conversation has moved on quite a bit since this comment about the bug being an edge case, but I just wanted to push back on it a little bit: not only are Ubuntu and Debian a considerable portion of the total Linux space on their own, but also the official Python docker containers are built from Debian bullseye and buster (unless we want to use alpine). I would think poetry would hope to work out of the box with the official Python images.

Edit: not to say that it isn’t annoying that this issue has persisted for so long in those distros, but it really does seem like something poetry needs to be resilient to.

@neersighted Thanks for the feedback 😃

Of course there are workarounds, but it was working very seamlessly before. I think also using the term “virtual environment” is kind of misleading when one just wants to use the system Python environment. So the expecation of poetry config virtualenvs.create false would be that it’s using the actual system Python environment. Otherwise it might be even better to deprecate this option, so that it’s clear.

(Finding out the correct path of the system Python environment is also not super easy 😄)

Any update on the fix for this bug?

Yes, or use poetry run

On Mon, Sep 12, 2022, 10:30 PM manojatlas @.***> wrote:

ok in that case I have to activate virtual environment when I want to execute tests?

— Reply to this email directly, view it on GitHub https://github.com/python-poetry/poetry/issues/6459#issuecomment-1244883223, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGI56LIXAADXU5BVLHBOPB3V577OBANCNFSM6AAAAAAQILJ47U . You are receiving this because you were mentioned.Message ID: @.***>

our detection of that broke.

Is it hard to fix that? If you can give some pointers, maybe I can try?

It’s also causing issues with the Tensorflow images. And of course I don’t want a virtual environment there, since my Docker image will blow up with two TF installations. Plus, it is quite hacky to access the python executable from the created virtual environment in a script then.

The official Python images are just fine as they use a Python.org build of Python and not a patched/hacked distro-supplied Python. The edge case is mostly installing to system site-packages using a Debian-derived Python, which is definitely a thing people do, but is often ill-advised and will always be prone to sharp edges due to deviation from the standard Python ecosystem.

root@7f452a8de1d3:/# python3
Python 3.8.10 (default, Jun 22 2022, 20:18:18)
>>> import sys
>>> print(sys.path)
['', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/root/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages']
>>> import sysconfig
>>> print(sysconfig.get_paths())
{'stdlib': '/usr/lib/python3.8', 'platstdlib': '/usr/lib/python3.8', 'purelib': '/usr/lib/python3.8/site-packages', 'platlib': '/usr/lib/python3.8/site-packages', 'include': '/usr/include/python3.8', 'platinclude': '/usr/include/python3.8', 'scripts': '/usr/bin', 'data': '/usr'}

Looks like this is the old Debian patching of distutils without patching sysconfig acting up again, combined with our apparent failure to make use of our modified detection code:

https://github.com/python-poetry/poetry/blob/c4b2253793cd6b41a99e25e479e40b776cca0a0e/src/poetry/utils/env.py#L193-L222

If I had to guess, it’s because we always treat the target environment as a VirtualEnv instead of a GenericEnv and thus don’t invoke that script.

Honestly, this is such an edge case (and mostly the result of bad Debian packaging) that I’m not sure there’s much value in fixing it – it’s easily resolved if you make use of a self-compiled Python or virtual environments (there is no reason to avoid them in a container, and doing so exposes you to all sorts of sharp edges like this).

This feels like another instance of #6398 – I would highly suggest looking at the pattern here https://github.com/python-poetry/poetry/issues/6397#issuecomment-1236327500 for something that is as ergonomic, but unlikely to be broken by distro packaging decisions.

Here is a tmp solution I used to set poetry env as the default python interpreter in docker.

#base image
FROM some_python_image

RUN pip install poetry

#copy source and install requirement
COPY ../pyproject.toml poetry.lock /app/
WORKDIR /app

# Their is a bug with poetry config virtualenvs.create false https://github.com/python-poetry/poetry/issues/6459
# Below is a workaround to set poetry env as default env
RUN poetry install --no-interaction --no-ansi
RUN POETRY_ENV_PATH=$(poetry env info --path) && ln -s $POETRY_ENV_PATH /mypyenv
ENV PATH="/mypyenv/bin:$PATH"

#start command
CMD ["python", "main.py"]