requests: Python 3.7.6: urllib3.exceptions.ProxySchemeUnknown

Proxy setup without scheme is not working on Python 3.7.6

While it works on Python 3.7.5:

Python 3.7.5 (default, Nov 23 2019, 05:59:34) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get("http://httpbin.org/get", proxies={"http": "103.250.166.4:6666"})
<Response [200]>

Is not under Python 3.7.6

Python 3.7.6 (default, Dec 21 2019, 08:28:11) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get("http://httpbin.org/get", proxies={"http": "103.250.166.4:6666"})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 412, in send
    conn = self.get_connection(request.url, proxies)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 309, in get_connection
    proxy_manager = self.proxy_manager_for(proxy)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 199, in proxy_manager_for
    **proxy_kwargs)
  File "/usr/local/lib/python3.7/site-packages/urllib3/poolmanager.py", line 470, in proxy_from_url
    return ProxyManager(proxy_url=url, **kw)
  File "/usr/local/lib/python3.7/site-packages/urllib3/poolmanager.py", line 420, in __init__
    raise ProxySchemeUnknown(proxy.scheme)
urllib3.exceptions.ProxySchemeUnknown: Not supported proxy scheme None

It seems that some changes on Python urllib.parse.urlparse brake requests.utils.prepend_scheme_if_needed

>>> utils.prepend_scheme_if_needed('103.250.166.4:6666', 'http')
'103.250.166.4://6666'

So it looks like this utils has to update to match Python changes (I think the change that produces this is https://github.com/python/cpython/commit/5a88d50ff013a64fbdb25b877c87644a9034c969). In both cases I’m using requests 2.22.0.

Reporting here since urllib.parse.urlparse behaviour is wrong in both Python versions and I think the problem is requests.utils.prepend_scheme_if_needed problem:

Python 3.7.5

>>> from urllib import parse
>>> parse.urlparse("103.250.166.4:6666")
ParseResult(scheme='', netloc='', path='103.250.166.4:6666', params='', query='', fragment='')

Python 3.7.6

>>> from urllib import parse
>>> parse.urlparse("103.250.166.4:6666")
ParseResult(scheme='103.250.166.4', netloc='', path='6666', params='', query='', fragment='')

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 18 (4 by maintainers)

Commits related to this issue

Most upvoted comments

yes, this fixes the issue for 3.7.6 or 3.8.x either by --proxy http://<ip>:<port> or in (Windows’) %APPDATA%\pip\pip.ini, [global] section: http://<ip>:<port>

proxy={'http': 'http://91.98.102.84:1080', 'https': 'http://91.98.102.84:1080'}

you can solve the problem?? i have the same issue with python 3.8

Proxy setup without scheme is not working on Python 3.7.6

While it works on Python 3.7.5:

Python 3.7.5 (default, Nov 23 2019, 05:59:34) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get("http://httpbin.org/get", proxies={"http": "103.250.166.4:6666"})
<Response [200]>

Is not under Python 3.7.6

Python 3.7.6 (default, Dec 21 2019, 08:28:11) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get("http://httpbin.org/get", proxies={"http": "103.250.166.4:6666"})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 412, in send
    conn = self.get_connection(request.url, proxies)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 309, in get_connection
    proxy_manager = self.proxy_manager_for(proxy)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 199, in proxy_manager_for
    **proxy_kwargs)
  File "/usr/local/lib/python3.7/site-packages/urllib3/poolmanager.py", line 470, in proxy_from_url
    return ProxyManager(proxy_url=url, **kw)
  File "/usr/local/lib/python3.7/site-packages/urllib3/poolmanager.py", line 420, in __init__
    raise ProxySchemeUnknown(proxy.scheme)
urllib3.exceptions.ProxySchemeUnknown: Not supported proxy scheme None

It seems that some changes on Python urllib.parse.urlparse brake requests.utils.prepend_scheme_if_needed

>>> utils.prepend_scheme_if_needed('103.250.166.4:6666', 'http')
'103.250.166.4://6666'

So it looks like this utils has to update to match Python changes (I think the change that produces this is python/cpython@5a88d50). In both cases I’m using requests 2.22.0.

Reporting here since urllib.parse.urlparse behaviour is wrong in both Python versions and I think the problem is requests.utils.prepend_scheme_if_needed problem:

Python 3.7.5

>>> from urllib import parse
>>> parse.urlparse("103.250.166.4:6666")
ParseResult(scheme='', netloc='', path='103.250.166.4:6666', params='', query='', fragment='')

Python 3.7.6

>>> from urllib import parse
>>> parse.urlparse("103.250.166.4:6666")
ParseResult(scheme='103.250.166.4', netloc='', path='6666', params='', query='', fragment='')

you can solve the problem?? i have same issue with python 3.8

Thanks for reporting this issue and doing investigation!

You can fix this by using http://<ip>:<port> instead of just an IP.

I had to re-edit my answer.

urllib3 and requests proxy implementations are distinctly different.

From requests: Based on the requests Proxies section; https://docs.python-requests.org/en/master/user/advanced/#proxies

Notice there is no “s” in https property value.


import requests

proxies = {
  'http': 'http://10.10.1.10:3128',
  'https': 'http://10.10.1.10:1080',
               ^
}

requests.get('http://example.org', proxies=proxies)

From urllib3: Based on the the urllib3 Proxies section; https://urllib3.readthedocs.io/en/stable/advanced-usage.html?highlight=proxy#proxies

>>> import urllib3
>>> proxy = urllib3.ProxyManager('http://localhost:3128/')
>>> proxy.request('GET', 'http://google.com/')

I sove this by modify pip_vendor\requests\adapters.py def get_connection(self, url, proxies=None): “”"Returns a urllib3 connection for the given URL. This should not be called from user code, and is only exposed for use when subclassing the :class:HTTPAdapter <requests.adapters.HTTPAdapter>.

    :param url: The URL to connect to.
    :param proxies: (optional) A Requests-style dictionary of proxies used on this request.
    :rtype: urllib3.ConnectionPool
    """
    proxy = select_proxy(url, proxies)
    print(proxy)

    proxy = None #<<<<<< ADD THIS LINE
    if proxy:
        proxy = prepend_scheme_if_needed(proxy, 'http')
        proxy_url = parse_url(proxy)
        if not proxy_url.host:
            raise InvalidProxyURL("Please check proxy URL. It is malformed"
                                  " and could be missing the host.")
        proxy_manager = self.proxy_manager_for(proxy)
        conn = proxy_manager.connection_from_url(url)
    else:
        # Only scheme should be lower case
        parsed = urlparse(url)
        url = parsed.geturl()
        conn = self.poolmanager.connection_from_url(url)

    return conn