urllib3: TLS-in-TLS not working with IP proxies

Subject

I’m trying to send a GET HTTP request to an HTTPS website through a secure proxy. The proxy only allows TLS connection with basic authentication.

To achieve this, I use a ProxyManager.

As result the following exceptions is triggered:

ValueError: check_hostname requires server_hostname

Environment

>>> print("OS", platform.platform())
OS Linux-5.12.13-x86_64-Intel-R-_Core-TM-_i7-8750H_CPU_@_2.20GHz-with-glibc2.33
>>> print("Python", platform.python_version())
Python 3.9.6
>>> print("urllib3", urllib3.__version__)
urllib3 1.25.11

Steps to Reproduce

You need an HTTPS proxy with basic authentication and an HTTPS website as an example:

def get(url, proxy, username, password):

    headers  = urllib3.make_headers(proxy_basic_auth=f'{username}:{password}')
    proxy    = urllib3.ProxyManager('https://{}'.format(proxy), proxy_headers=headers)
    response = proxy.request("GET", url)            # Exception trigger here

    return (response)

Expected Behavior

Website body page as a response.

Actual Behavior

Exception triggered:

Traceback (most recent call last):
[...]
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/request.py", line 74, in request
    return self.request_encode_url(
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/request.py", line 96, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/poolmanager.py", line 532, in urlopen
    return super(ProxyManager, self).urlopen(method, url, redirect=redirect, **kw)
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 696, in urlopen
    self._prepare_proxy(conn)
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 964, in _prepare_proxy
    conn.connect()
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/connection.py", line 359, in connect
    conn = self._connect_tls_proxy(hostname, conn)
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/connection.py", line 500, in _connect_tls_proxy
    return ssl_wrap_socket(
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/util/ssl_.py", line 453, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
  File "/home/janemba/.virtualenv/test-rCp672mu-py3.9/lib64/python3.9/site-packages/urllib3/util/ssl_.py", line 495, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock)
  File "/usr/lib64/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib64/python3.9/ssl.py", line 997, in _create
    raise ValueError("check_hostname requires server_hostname")
ValueError: check_hostname requires server_hostname

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 17 (9 by maintainers)

Most upvoted comments

@Kcam9908 Did you read my reply to you directly above? In summary we don’t think the issue you’re seeing will be resolved by the PR you mentioned.

“as it is a critical fix for our product.”

Your organization might consider sponsoring our development efforts if urllib3 is critical to your business.

sethmlarson on Sep 10, 2021

Congratulations, you just found a bug! We send SNI even though we should not when the proxy is an IP. (That’s what the error message says.)

A workaround would be to use an hostname if you have one or just stop using TLS-in-TLS. (This has nothing to do with basic authentication.)

pquentin on Sep 1, 2021

The PR for relying on urllib3’s hostname matching logic for verifying proxy certificates is merged. However there’s still a bug with IPv6 addresses we’d like to fix before we ship a bugfix release.

sethmlarson on Sep 16, 2021

Could you share your configuration? There was an unfortunate situation in urllib3 where we never documented support for TLS-in-TLS proxies pre-1.26.0 but trying to use it didn’t fail with an error it instead “kinda worked” by doing HTTP on the proxy and HTTPS for the target.

Basically your “solution” may be to use a proxy with http:// and target https:// for the website you’re visiting which was the pre-1.26.0 behavior you were likely experiencing anyways.

See https://github.com/urllib3/urllib3/issues/1850 for another explanation, you would have seen this warning if you were using 1.25.9 which is the version that added the warning in preparation for us actually supporting TLS-in-TLS proxying.

sethmlarson on Sep 7, 2021

Are you connecting to your proxy using an IP address?

pquentin on Sep 1, 2021