pip: Support for web proxies is broken in pip 20.3
Environment
- pip version: 20.3
- Python version: 3.9.1rc1
- OS: Windows 10 Enterprise Version 1909 (64-bit)
I am behind a corporate firewall with an authenticating proxy that requires both a user ID and a password.
Description
pip 20.2.4 supported the proxy but pip 20.3 does not. I tried both setting the environment variables and using the --proxy command line argument, but neither worked in pip 20.3. pip 20.2.4 works fine even if no --proxy argument is used and no proxy-related environment variables are set.
Expected behavior
pip 20.3 should support the proxy in the same way that pip 20.2.4 does.
How to Reproduce
See the output below. I set up a virtual environment, and updated pip to version 20.2.4, demonstrated that there was no bug, then updated pip to version 20.3 and demonstrated the bug
Output
Paste the output of the steps above, including the commands themselves and
pip's output/traceback etc.
C:\MISC>mkdir pipbug
C:\MISC>cd pipbug
C:\MISC\pipbug>python -m venv pipbug
C:\MISC\pipbug>dir
Volume in drive C is Windows
Volume Serial Number is 0CB7-E951
Directory of C:\MISC\pipbug
11/30/2020 02:57 PM <DIR> .
11/30/2020 02:57 PM <DIR> ..
11/30/2020 02:57 PM <DIR> pipbug
0 File(s) 0 bytes
3 Dir(s) 122,653,765,632 bytes free
C:\MISC\pipbug>pipbug\Scripts\activate
(pipbug) C:\MISC\pipbug>pip list
Package Version
---------- -------
pip 20.2.3
setuptools 49.2.1
WARNING: You are using pip version 20.2.3; however, version 20.3 is available.
You should consider upgrading via the 'c:\misc\pipbug\pipbug\scripts\python.exe -m pip install --upgrade pip' command.
(pipbug) C:\MISC\pipbug>python -m pip install -U "pip<20.3"
Collecting pip<20.3
Using cached pip-20.2.4-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 20.2.3
Uninstalling pip-20.2.3:
Successfully uninstalled pip-20.2.3
Successfully installed pip-20.2.4
(pipbug) C:\MISC\pipbug>pip install -U setuptools wheel
Collecting setuptools
Using cached setuptools-50.3.2-py3-none-any.whl (785 kB)
Collecting wheel
Using cached wheel-0.35.1-py2.py3-none-any.whl (33 kB)
Installing collected packages: setuptools, wheel
Attempting uninstall: setuptools
Found existing installation: setuptools 49.2.1
Uninstalling setuptools-49.2.1:
Successfully uninstalled setuptools-49.2.1
Successfully installed setuptools-50.3.2 wheel-0.35.1
WARNING: You are using pip version 20.2.4; however, version 20.3 is available.
You should consider upgrading via the 'c:\misc\pipbug\pipbug\scripts\python.exe -m pip install --upgrade pip' command.
(pipbug) C:\MISC\pipbug>pip list
Package Version
---------- -------
pip 20.2.4
setuptools 50.3.2
wheel 0.35.1
WARNING: You are using pip version 20.2.4; however, version 20.3 is available.
You should consider upgrading via the 'c:\misc\pipbug\pipbug\scripts\python.exe -m pip install --upgrade pip' command.
(pipbug) C:\MISC\pipbug>pip list -o
Package Version Latest Type
------- ------- ------ -----
pip 20.2.4 20.3 wheel
WARNING: You are using pip version 20.2.4; however, version 20.3 is available.
You should consider upgrading via the 'c:\misc\pipbug\pipbug\scripts\python.exe -m pip install --upgrade pip' command.
(pipbug) C:\MISC\pipbug>REM Call proxies.bat to set environment variables for proxy.
(pipbug) C:\MISC\pipbug>REM Certain items (<user>, <password>, and <domain>) have been redacted.
(pipbug) C:\MISC\pipbug>proxies <password>
(pipbug) C:\MISC\pipbug>set PIP_PROXY=http://<user>:<password>@proxy-west.<domain>.org:8080/
(pipbug) C:\MISC\pipbug>set FTP_PROXY=ftp://<user>:<password>@proxy-west.<domain>.org:8080/
(pipbug) C:\MISC\pipbug>set http_proxy=http://<user>:<password>@proxy-west.<domain>.org:8080/
(pipbug) C:\MISC\pipbug>set https_PROXY=https://<user>:<password>@proxy-west.<domain>.org:8080/
(pipbug) C:\MISC\pipbug>c:\misc\pipbug\pipbug\scripts\python.exe -m pip install --upgrade pip
Collecting pip
Using cached pip-20.3-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 20.2.4
Uninstalling pip-20.2.4:
Successfully uninstalled pip-20.2.4
Successfully installed pip-20.3
(pipbug) C:\MISC\pipbug>pip list -o
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/setuptools/
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/wheel/
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/wheel/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/setuptools/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/wheel/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/setuptools/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/wheel/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/setuptools/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/wheel/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/setuptools/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
(pipbug) C:\MISC\pipbug>pip list -o --use-deprecated=legacy-resolver
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/wheel/
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/setuptools/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/wheel/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/setuptools/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/wheel/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/setuptools/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/wheel/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/setuptools/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/wheel/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/setuptools/
(pipbug) C:\MISC\pipbug>pip list -o --proxy <user>:<password>@proxy-west.<domain>.org:8080
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/wheel/
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/setuptools/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/wheel/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/setuptools/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/wheel/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/setuptools/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/wheel/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/setuptools/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/wheel/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/setuptools/
(pipbug) C:\MISC\pipbug>python -m pip install -U "pip<20.3"
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
ERROR: Could not find a version that satisfies the requirement pip<20.3
ERROR: No matching distribution found for pip<20.3
(pipbug) C:\MISC\pipbug>dir \misc\*pip*.py
Volume in drive C is Windows
Volume Serial Number is 0CB7-E951
Directory of C:\misc
11/30/2020 09:46 AM 1,910,500 get-pip.py
05/14/2016 05:43 AM 805 piplist.py
2 File(s) 1,911,305 bytes
0 Dir(s) 122,641,879,040 bytes free
(pipbug) C:\MISC\pipbug>python c:\misc\get-pip.py "pip<20.3"
Collecting pip<20.3
Using cached pip-20.2.4-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 20.3
Uninstalling pip-20.3:
Successfully uninstalled pip-20.3
Successfully installed pip-20.2.4
(pipbug) C:\MISC\pipbug>pip list -o
Package Version Latest Type
------- ------- ------ -----
pip 20.2.4 20.3 wheel
WARNING: You are using pip version 20.2.4; however, version 20.3 is available.
You should consider upgrading via the 'c:\misc\pipbug\pipbug\scripts\python.exe -m pip install --upgrade pip' command.
(pipbug) C:\MISC\pipbug>deactivate
C:\MISC\pipbug>
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 2
- Comments: 23 (12 by maintainers)
@tomerv I have found the problem. It’s actually within pip at
pi\_internal\network\session.py
in the methodPipSession().request()
line 419
I’m using pip 21.0.1 to debug.
The problem is caused because the proxy set by --proxy in the pip method is not being passed corrected to the request of urllib package.
This is the original request method of PipSession:
If you make a breakpoint there you gonna see that self.proxies is corrected setted from --proxies, however, when the call to the parent class is made the proxies get lost and urllib set it from the system, which then gets the wrong scheme (https instead of http).
An easy fix I suggest is to add the following line at the request method in the file
pi\_internal\network\session.py, line 419
:This way the proxies are carried out correctly to the request, if no proxy is set an empty dict is passed.
So now if you run
pip install package --proxy http://proxy_ip
everything works as expected.I hope this will help to remove this bug in next versions of pip asap.
The new flow for
tls_in_tls_required
is at fault - introduced in https://github.com/pypa/pip/commit/e76b1ddeaa3694ae87df577c751ba7d9d36b4771Removing these 2 lines makes everything work again:
I have no idea what’s the purpose of this, and I couldn’t find any explanation in the code or the commit message. This also explains @GreggMidon 's comment above:
EDIT: I now think that the code is fine, even if a bit unclear. Changing the proxy from “https” to “http” is the correct solution.
One way to verify this would be to run
requests.get("https://pypi.org/simple/pip/")
behind the proxy and see what happens.pip 20.3 uses the following versions:
https://github.com/pypa/pip/blob/30eeb9ceb10bf76316f091e8615a9c3d8025fff0/src/pip/_vendor/vendor.txt#L14-L18
pip 20.2.4:
https://github.com/pypa/pip/blob/57dd580f582f33310f1c53cca2af2255985425d2/src/pip/_vendor/vendor.txt#L14-L18
Here is the requested output. I have removed more than 1000 uninteresting lines.
The fix suggested by @tomerv solved my problem. The two lines are in question are lines 961 and 962 of connectionpool.py. You should apply this fix to all instances of connectionpool.py that contain these two lines. One instance will be in the pip package and another instance will be in urllib3 package if you have it installed.