pip: SSL certificate error with pip 21.0.1

I have found the root cause of the problem and pip can fix its end.

I understand that urllib is getting the wrong scheme for https. A workaround would be to set https_proxy= http://your_proxy. However, this solution is not ideal. Many users behind proxy don’t have full access to their system, also if you use IDEs like Pycharm this will not work for its installer. The SSL error also occurs when using Ansible to deploy to a Linux server.

Saying that pip has a way to pass proxy running: pip install package --proxy http://your_proxy, which by the way is how Ansible or Pycharm handle proxy to pip. And that is where the problem lies, after some debug I have realise that pip is not passing it to urllib request, this problem was unnoticed until now because, before urllib update, it didn’t check ssl_certificate so it didn’t matter if your scheme was either http or https.

I’m using pip 21.0.1 to debug and a venv.

To reproduce:

• Set a fake proxy using system var. i.e: set https_proxy: http://127.0.0.1:80 or export https_proxy: http://127.0.0.1:80 for Linux
• Now run pip passing a different proxy, i.e: pip install numpy --proxy http://127.0.0.2:80 or pip.main(['install', '--proxy=http://127.0.0.2:80', 'numpy']) for esier debug.

I would expect pip to use the proxy I’m passing to it and override any system proxy, but that is not the case. To see that add a breakpoint at the request method from PipSession class in pi\_internal\network\session.py. Here you can notice that PipSession has a parameter self.proxies that match the proxy I have passed to pip with --proxy. Then, this makes a call to the parent request super().request(method, url, *args, **kwargs) which is Session class from pi\_vendor\requests\session.py. One of its kwargs is proxies, add a breakpoint there and you will notice that the proxies you get are from the system, whatever you set as set https_proxy: http://127.0.0.1:80.

So basically pip is not passing through the proxies set by the user when invoking pip, and that is the root cause of all the following errors with SSL.

This is the original request method of PipSession:

def request(self, method, url, *args, **kwargs):
        # Allow setting a default timeout on a session
        kwargs.setdefault("timeout", self.timeout)
        
        # Dispatch the actual request
        return super().request(method, url, *args, **kwargs)

An easy fix I suggest is to add the following line at the request method in the file pi_internal\network\session.py:

def request(self, method, url, *args, **kwargs):
     # Allow setting a default timeout on a session
     kwargs.setdefault("timeout", self.timeout)
     kwargs.setdefault("proxies", self.proxies)  # fix problem with proxies.

    # Dispatch the actual request
    return super().request(method, url, *args, **kwargs)

This way the proxies are carried out correctly to the request, if no proxy is set an empty dict is passed.

So now if you run pip install package --proxy http://proxy_ip everything works as expected. Because I’m using the right scheme for my proxy.

I hope this will help to remove this bug in the next version of pip asap.

I downgrade pip to 20.2.3 and it works fine.

@junqfisica I’m not sure when we’ll wait for a final solution either, I’ve even gotten used to the alternative of using environment variables over the year. I’m sorry that now we have to wait for the decision of pip’s maintainers.

On Windows 10, the following two steps allowed me to install pip packages in a conda environment.

1. Set environment variables for proxy:

set HTTP_PROXY=http://<url>:<port>
set HTTPS_PROXY=https://<url>:<port>

1. Use Python 3.8 and pip 20.2.2:

conda install pip=20.2.2=py38_0

I have found the root cause of the problem and pip can fix its end.

So there are two reasons for this error:

1. urllib misparses Windows registry proxy settings. This bug is being processing.
2. pip --proxy does not work while the system proxy is set. According to #9614 , this bug is not only related to system proxies, but also other system/environment settings. The real reason may be in merge_setting(): https://github.com/pypa/pip/blob/7b7469b8454be1c0f968957e93ae23264187ee41/src/pip/_vendor/requests/sessions.py#L722-L725 https://github.com/pypa/pip/blob/7b7469b8454be1c0f968957e93ae23264187ee41/src/pip/_vendor/requests/sessions.py#L50 I am now trying to check this function.

@CrazyBoyFeng That’s correct. That’s why I have made the PR #10680 to pip, which handles pip.session.proxies the same way as pip.session.timeout. However, the PR is still waiting for checks/approvals from the maintainers. Do you have any idea of how long it will take for the PR to be processed?

@CrazyBoyFeng

Would be nice if you could do the PR patch or someone else. Currently, I’m using working hours for that…so 😃