pip: Pip 20.3+ break proxy connection

Environment

pip version: 20.2.3
Python version: 3.9
OS: Win10 64-bit

Description When I’m trying to update pip using py -m pip install --upgrade pip a ‘ProxySchemeUnknown’ error occurs with a plenty of other ‘File’ errors.

When I was looking for the solution to this problem, I found very similar issues, but they’re not the same: https://github.com/pypa/pip/issues/6555; https://github.com/psf/requests/issues/5297

I think the problem is with my internet provider. I am sharing the internet with my laptop through a portable hotspot in my smartphone. Also, I changed TTL on the laptop.

Expected behavior pip is updated.

How to Reproduce

Open Windows’ CommandPrompt.
Run py -m pip install --upgrade pip.
An error occurs.

Output

C:\Users\User>py -m pip install --upgrade pip
Defaulting to user installation because normal site-packages is not writeable
ERROR: Exception:
Traceback (most recent call last):
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\cli\base_command.py", line 228, in _main
    status = self.run(options, args)
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\cli\req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\commands\install.py", line 323, in run
    requirement_set = resolver.resolve(
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 183, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 388, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 331, in _get_abstract_dist_for
    skip_reason = self._check_skip_installed(req)
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 254, in _check_skip_installed
    self.finder.find_requirement(req_to_install, upgrade=True)
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\index\package_finder.py", line 898, in find_requirement
    best_candidate_result = self.find_best_candidate(
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\index\package_finder.py", line 881, in find_best_candidate
    candidates = self.find_all_candidates(project_name)
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\index\package_finder.py", line 825, in find_all_candidates
    package_links = self.process_project_url(
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\index\package_finder.py", line 790, in process_project_url
    html_page = self._link_collector.fetch_page(project_url)
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\index\collector.py", line 643, in fetch_page
    return _get_html_page(location, session=self.session)
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\index\collector.py", line 455, in _get_html_page
    resp = _get_html_response(url, session=session)
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\index\collector.py", line 152, in _get_html_response
    resp = session.get(
  File "C:\Program Files\Python39\lib\site-packages\pip\_vendor\requests\sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "C:\Program Files\Python39\lib\site-packages\pip\_internal\network\session.py", line 421, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "C:\Program Files\Python39\lib\site-packages\pip\_vendor\requests\sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Program Files\Python39\lib\site-packages\pip\_vendor\requests\sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "C:\Program Files\Python39\lib\site-packages\pip\_vendor\cachecontrol\adapter.py", line 53, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "C:\Program Files\Python39\lib\site-packages\pip\_vendor\requests\adapters.py", line 412, in send
    conn = self.get_connection(request.url, proxies)
  File "C:\Program Files\Python39\lib\site-packages\pip\_vendor\requests\adapters.py", line 309, in get_connection
    proxy_manager = self.proxy_manager_for(proxy)
  File "C:\Program Files\Python39\lib\site-packages\pip\_vendor\requests\adapters.py", line 193, in proxy_manager_for
    manager = self.proxy_manager[proxy] = proxy_from_url(
  File "C:\Program Files\Python39\lib\site-packages\pip\_vendor\urllib3\poolmanager.py", line 492, in proxy_from_url
    return ProxyManager(proxy_url=url, **kw)
  File "C:\Program Files\Python39\lib\site-packages\pip\_vendor\urllib3\poolmanager.py", line 429, in __init__
    raise ProxySchemeUnknown(proxy.scheme)
pip._vendor.urllib3.exceptions.ProxySchemeUnknown: Not supported proxy scheme None
WARNING: You are using pip version 20.2.3; however, version 20.3.1 is available.
You should consider upgrading via the 'C:\Program Files\Python39\python.exe -m pip install --upgrade pip' command.

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 1
Comments: 24 (15 by maintainers)

Most upvoted comments

Hello, urllib3 dev here.

The ProxySchemeUnknown error has been in urllib3 in its current state for 6 years and wasn’t changed between 1.25.9 and 1.26.2. Diff for these versions is here: https://github.com/urllib3/urllib3/compare/1.25.9...1.26.2, ProxySchemeUnknown is raised here: https://github.com/pypa/pip/commit/e76b1ddeaa3694ae87df577c751ba7d9d36b4771#diff-42d9ff3a3dc18438ae9ad3552070169bc4140b54b61529e8a2ac7708352e1c5dL428 and is unchanged.

My only thoughts on the ProxySchemeUnknown issue is that proxies that previously weren’t being detected and passed to urllib3 now are?
As for connection issues with a proxy: Users that have their proxy configured as proxy_url=https://... and connecting to https://pypi.org before were actually using their proxy in an HTTP configuration (urllib3 didn’t support HTTPS in HTTPS before v1.26 and would silently connect to the proxy via HTTP) so should update their proxy to be “http://…” or change their proxy to use HTTPS if that’s desired.

We added a InvalidProxyConfigurationWarning for this situation in 1.25.9 to warn users of the upcoming change to make HTTPS+HTTPS proxy requests work as expected in 1.26, we didn’t receive many hits on the linked issue so assumed this issue wasn’t widespread.

Comments like these: https://github.com/pypa/pip/issues/9190#issuecomment-739127220 https://github.com/pypa/pip/issues/9190#issuecomment-739518255 seem to indicate this is the case for some reporting users.

My recommendation to all users that know they have a proxy configured in 20.2 of HTTPS_PROXY=https://<host> should actually be HTTPS_PROXY=http://<host>. I’m interested in seeing people’s issues that don’t fit this criteria though, if any.

cc @jalopezsilva

+11

sethmlarson on Dec 9, 2020

I also had the issue ERROR: Could not install packages due to an EnvironmentError: Not supported proxy scheme None as my environment variables HTTPS_PROXY and HTTP_PROXY were configured as xxx.xxx.xxx.xxx:xxxx. I changed this to http://xxx.xxx.xxx.xxx:xxxx and everything works as normal now.

tdekelver on Dec 29, 2020

Thanks for the explanation! This is indeed the source of the issue for us, and after reading the explanation I’m convinced that simply changing “https://” to “http://” is the correct solution (and not just a workaround - as I thought before). For better usability, my recommendation would be to catch this specific mistake and give a better error message, at least in the next few releases.

tomerv on Dec 9, 2020

HI I Still have problem with the proxy setting or SSL @uranusjr

@sanjay1839 try SET HTTPS_PROXY=http://[username:pass]@server:port it’s working on my side

pip version: 21.0

weijenloi on Jan 30, 2021

same here.

Zheng ~ ❯❯❯ pip --version
pip 20.3.1 from d:\scoop\apps\python\current\lib\site-packages\pip (python 3.7)
Zheng ~ ❯❯❯ pip config list
global.index-url='https://pypi.tuna.tsinghua.edu.cn/simple'

without http_proxy and https_proxy:

Zheng ~\..\pywonderland git: master ≣ ❯❯❯ pip install -r .\requirements.txt
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError(0, 'Error'))': /simple/colour/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError(0, 'Error'))': /simple/colour/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError(0, 'Error'))': /simple/colour/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError(0, 'Error'))': /simple/colour/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError(0, 'Error'))': /simple/colour/
ERROR: Could not find a version that satisfies the requirement colour
ERROR: No matching distribution found for colour

after setting up those two environment variables:

Zheng ~\..\pywonderland git: master ≣ ❯❯❯ proxy

Name                           Value
----                           -----
http_proxy                     127.0.0.1:1080
https_proxy                    127.0.0.1:1080

Zheng ~\..\pywonderland git: master ≣ ❯❯❯ pip install -r .\requirements.txt
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
ERROR: Could not install packages due to an EnvironmentError: Not supported proxy scheme None

FDUZS on Dec 9, 2020

I dug into this a bit deeper and it seems like even requests itself is a bit inconsistent: https://github.com/psf/requests/issues/5677#issuecomment-769075427

uranusjr on Mar 4, 2021

If you make a breakpoint there you gonna see that self.proxies is corrected setted from --proxies, however, when the call to the parent class is made the proxies get lost and urllib set it from the system, which then gets the wrong scheme (https instead of http).

This makes me suspect this may be caused by an implementation change in requests.Session, and not actually intended. I’m not sure the proposed change is correct, but “proxies specified by --proxy are not applied to the underlying requests connection” should be considered a bug on its own. It should be described in its own issue report and not here though.

uranusjr on Mar 4, 2021

@pradyunsg Created two usability issues: https://github.com/urllib3/urllib3/issues/2104 and https://github.com/urllib3/urllib3/issues/2105

sethmlarson on Dec 9, 2020

Hmm, the 20.2 series was on 1.25.9, but I don’t recall anyone reporting issues on this either. Maybe our warning settings did not bubble them correct? I have to admit I’ve never understood how warnings in Python work. Either way, thanks for the recommendation! I’m changing the title and pinning this issue for better discovery. (Edit: oops, looks like we have too many important issues, and GitHub only allows 3. So we’ll have to do without pinning.)

uranusjr on Dec 9, 2020

I resolved the same problem by changing my https_proxy environement variable from <ip>:<port> to https://<ip>:<port>

Perhaps urllib got stricter

jw120 on Dec 4, 2020

I’ll defer to other people in responding to the substance of your issue, but: congratulations on filing your first GitHub issue! And thanks for the detailed report!

brainwane on Dec 3, 2020