pip: stderr duplication failed in console scripts

Description

I use a console_scripts entry point with a test function. If I do pip install . or use setup.py to make a binary dist then install it, under some shell like Git Bash, running the created test-script.exe. stdin, stdout and stderr should all be set to a TextIOWrapper however, stderr is set to None.

The stderr handle is apparently not correctly set.

This is probably related to https://github.com/pypa/pip/issues/10444 and the fix introduced in Distlib 0.3.4 for a similar issue.

Expected behavior

$ test-script.exe
stdin=<_io.TextIOWrapper name='<stdin>' mode='r' encoding='cp1252'>
stdout=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp1252'>
stderr=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='cp1252'>

pip version

22.0

Python version

Python 3.7

OS

Windows

How to Reproduce

# setup.py
from setuptools import setup

setup(
    name="test",
    packages=["testing"],
    entry_points={
        "console_scripts": [
            "test-script=testing:main"
        ]
    }
)

# testing/__init__.py
import sys


def main():
    print(f"stdin={sys.stdin}")
    print(f"stdout={sys.stdout}")
    print(f"stderr={sys.stderr}")

Output

# With pip 21.*
$ test-script.exe
stdin=<_io.TextIOWrapper name='<stdin>' mode='r' encoding='cp1252'>
stdout=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp1252'>
stderr=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='cp1252'>

# With pip 22.0, 22.0.1 and 22.0.2 
$ test-script.exe
stdin=<_io.TextIOWrapper name='<stdin>' mode='r' encoding='cp1252'>
stdout=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp1252'>
stderr=None

Code of Conduct

I agree to follow the PSF Code of Conduct.

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 4
Comments: 55 (26 by maintainers)

Commits related to this issue

fix: downgrade pip on Windows to 21.3.1 https://github.com/pypa/pip/issues/10875#issuecomment-1030293005 — committed to aminya/setup-cpp by aminya 2 years ago
[ci] Downgrade pip on MSVC jobs Turns out, pip 22.0 is the source of the breakage: https://github.com/mesonbuild/meson/issues/9955#issuecomment-1030843844 https://github.com/pypa/pip/issues/10875 — committed to harfbuzz/harfbuzz by khaledhosny 2 years ago
[ci] Downgrade pip on MSVC jobs Turns out, pip 22.0 is the source of the breakage: https://github.com/mesonbuild/meson/issues/9955#issuecomment-1030843844 https://github.com/pypa/pip/issues/10875 — committed to harfbuzz/harfbuzz by khaledhosny 2 years ago
[ci] Downgrade pip on MSVC jobs Turns out, pip 22.0 is the source of the breakage: https://github.com/mesonbuild/meson/issues/9955#issuecomment-1030843844 https://github.com/pypa/pip/issues/10875 — committed to harfbuzz/harfbuzz by khaledhosny 2 years ago
CI: work around broken pip version pip 22.0 contains broken Windows launcher executables which mangle sys.stderr and cause any process launched using meson.exe (as installed by that version of pip) t... — committed to mesonbuild/wrapdb by eli-schwartz 2 years ago

Most upvoted comments

So the old version of distlib broke certain GUI use cases by making error dialog boxes pop up.

The new version of distlib broke other – CLI this time – use cases by making them error out silently.

This is good context, thanks for noting this here @eli-schwartz!

I think having a visible (albeit cryptic) error that users can discover is better than a silent error that they can’t. Further, CLI entry points are a lot more common than GUI entry points and I’m pretty sure the right thing to do here is to downgrade distlib to 0.3.3 until @vsajip is able able to fix this issue in distlib.

I’ll cut a bugfix for this.

pradyunsg on Feb 25, 2022

@carlkl I understand what the problem is, and I understand what we’d need to do to fix it (revert to an older distlib, or wait for a distlib fix). What I don’t know is how many people are affected, nor do I know whether this issue is more disruptive than the issue that the distlib upgrade fixed, or whether it will be more disruptive than the cost of making a bugfix release.

The comment you linked to says it is a “severe” bug. It’s not severe for me, I’ve never seen it happen. So if you can be more precise about what circumstances cause it to occur, that would make it easier to assess the severity in a way that doesn’t just boil down to “it doesn’t happen for me” vs “it happens for a program someone else uses a lot”. You also say “Now sys.stderr is None if you start launchers created by pip 22”. My reproducer above demonstrates that this isn’t always true. So the question is how often is that the case, and under what circumstances. I can’t tell, because as yet I haven’t ever had it happen for me.

🤷 I’m not really that bothered. Ultimately someone needs to make the case that this is important enough for us to produce a bugfix release with a reverted distlib. I’m only trying to help people make that case - if they don’t, then this will likely just wait for a fix in distlib, and the next pip release.

I have an example of a very common use case affected by this bug.

Imagine a python developer using Jenkins to run his unit tests, using pytest, on a Windows host.

Now because he start using pip 22.x (he updated because pip warned him that he was not using the latest version), running pytest will not produce any output on the console, tests will not run and there is no reporting.

Now how is this user supposed to root cause this issue? There is no output at all since stderr is None.

And this is not a intermittent issue, in the affected environment (like Jenkins on Windows in this example), stderr will always be None.

@pfmoore, I hope this helps you understand the importance of getting this bug fixed.

pierreluctg on Feb 8, 2022

Thanks @vsajip for the fixes! ^>^

pradyunsg on Dec 2, 2022

@pradyunsg, what do you think about a bugfix release with patched launchers as from jeremyd2019/simple_launcher@948b4e7?

That’s not going to happen. We don’t vendor stuff in pip from arbitrary git commits. See https://pip.pypa.io/en/stable/development/vendoring-policy/ for our policy for what can be vendored into pip.

This needs a bugfix release in distlib, that we’d then pick up when we update our dependencies for a release.

I was trying to get things as ready as possible for @vsajip, by proposing a fix, and putting it out for testing to confirm that it fixes the issue and doesn’t break other things. It is still up to them to accept the fix (or coming up with some other fix), update simple_launcher, and update distlib with the updated launchers. As a side-note, I regret now not having written a commit message on that commit. I only wanted a quick build from GHA to test with, I wasn’t expecting it to be proposed as the fix 😁.

jeremyd2019 on Feb 11, 2022

One correction to what @pfmoore said: I’m fine with cutting a bugfix for this, assuming it happens soon / when I have the free time to do so.

To state more explicitly: What we’re not going to do is use a fork of simple_launchers and vendor those into a pip release.

a bugfix release of pip (e.g. 22.0.4) with an updated distlib release is out of the question?

If there’s a distlib release in the coming few weeks, I’m fine with cutting a bugfix with it.

As @pfmoore said tho, let’s wait until there’s a distlib release before we have any more discussion here. The two things that need to happen for a bugfix release of pip containing this fix, is a distlib release containing it and me having enough free time to cut a pip release. 😃

pradyunsg on Feb 11, 2022

Ask again when there’s an actual distlib release.

pfmoore on Feb 11, 2022

@pfmoore so a bugfix release of pip (e.g. 22.0.4) with an updated distlib release is out of the question? I understand that pip only vendors regular releases of packages, but what if distlib fixes this bug and makes a new release (e.g. 0.3.5)?

cbrnr on Feb 11, 2022

If I understand you correctly you want to patch this in the next regulary release?

What we’re saying (and @pradyunsg can correct me if I’ve misinterpreted anything he said) is that there won’t be a bugfix release for this. In order for a fix to appear in pip, the first thing that needs to happen is for distlib to release a new version on PyPI that contains the fix. Then, once that is done, pip will vendor that new distlib release, and that will be included in the next scheduled release of pip after that. Pip releases every 3 months, with the next release (22.1) being in April, so the earliest a fix will be in pip is then, and if the distlib release takes too long (possible as @vsajip has said he won’t be able to look at it for a while) then the fix might not appear in pip until 22.2 (July) or later.

pfmoore on Feb 11, 2022

@jeremyd2019 Many thanks for setting this up! I successfully performed some test on my local machine (no CI test from my side):

I reinstalled pip to the latest upgrade (22.0.3); after that I reinstalled meson as well as mne and mnelab (wink @cbrnr). As expected mnelab starts without problems; but meson gives an traceback:

...
\mesonbuild\mesonlib\universal.py", line 1419, in Popen_safe_legacy
    if sys.stderr.encoding:
AttributeError: 'NoneType' object has no attribute 'encoding'

After that I copied your artifacts from msvc-2019-amd64.zip (https://github.com/jeremyd2019/simple_launcher/actions/runs/1808357504) over Lib\site-packages\pip\_vendor\distlib\t64.exe andLib\site-packages\pip\_vendor\distlib\w64.exe.

Then I reinstalled meson as well as mne and mnelab again. mnelab still works and now I’m able to run meson without errors!

@pradyunsg, what do you think about a bugfix release with patched launchers as from https://github.com/jeremyd2019/simple_launcher/commit/948b4e72c6cb10eb23a0909bb887586fc34a34f3?

EDIT: my terminal is a msys2-uctr64 console (not using mintty)

carlkl on Feb 8, 2022

carlkl on Feb 7, 2022

@aminya,

For meson as an example, it is enough to run the following? or should it be specifically installed via the pip executable? python -m pip install -U pip==21.3.1 python -m pip install meson

I reinstalled meson in excact that way.

carlkl on Feb 6, 2022

Right now my solution is to install an older pip version on Windows: python -m pip install -U pip==21.3.1. After that all packages with faulty launchers should also be reinstalled. I’ve seen issues with meson as well as with tqdm.

carlkl on Feb 4, 2022

@pierreluctg I’m aware of this issue, but I probably won’t have the bandwidth to spend much (if any) time on it until the last week of March 2022, due to other commitments 😞

vsajip on Feb 4, 2022