requests: https GET request fails with "handshake failure"

Related to #1083, perhaps. Standard requests.get() for this particular site/page https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html results in:

>>> import requests
>>> requests.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/adapters.py", line 385, in send
    raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

Using request-toolbelt’s SSLAdapter to try various ssl versions, they all fail, it would seem… see following tracebacks.

TLSv1:

>>> adapter = SSLAdapter('TLSv1')
>>> s = requests.Session()
>>> s.mount('https://', adapter)
>>> s.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/adapters.py", line 385, in send
    raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

SSLv3:

>>> adapter = SSLAdapter('SSLv3')
>>> s = requests.Session()
>>> s.mount('https://', adapter)
>>> s.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/adapters.py", line 385, in send
    raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

SSLv2:

>>> adapter = SSLAdapter('SSLv2')
>>> s = requests.Session()
>>> s.mount('https://', adapter)
>>> s.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='docs.apitools.com', port=443): Max retries exceeded with url: /2014/04/24/a-small-router-for-openresty.html (Caused by <class 'socket.error'>: [Errno 54] Connection reset by peer)

Note the last one gives a Connection reset by peer error, which differs from the others, but I’m pretty sure SSLv2 isn’t supported by the server anyhow.

For fun, I tried to pass through some more appropriate headers through on the last request as well:

>>> headers = {
...     'Accept': u"text/html,application/xhtml+xml,application/xml",
...     'User-Agent': u"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36",
...     'Accept-Encoding': u"gzip,deflate",
...     'Accept-Language': u"en-US,en;q=0.8"
... }
>>> adapter = SSLAdapter('SSLv2')
>>> s = requests.Session()
>>> s.mount('https://', adapter)
>>> s.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html', headers=headers)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='docs.apitools.com', port=443): Max retries exceeded with url: /2014/04/24/a-small-router-for-openresty.html (Caused by <class 'socket.error'>: [Errno 54] Connection reset by peer)

No dice there either. Here’s what the HTTPS connection info in Chrome on Mac looks like:

screen shot 2014-04-26 at 10 35 21 am

I’m not positive, but some googling indicates it’s likely a cipher list issue, which is more urllib3, I think?

I tried to modify DEFAULT_CIPHER_LIST in pyopenssl, but started running into import errors. At this point it seemed like things were just broken, and there wasn’t really a proper way to approach fixing this yet.

Version information: OSX Mavericks Python 2.7.5 OpenSSL 0.9.8y 5 Feb 2013 - (from python -c "import ssl; print ssl.OPENSSL_VERSION") requests 2.2.1 requests-toolbelt 0.2.0 urllib3 1.8

About this issue

Original URL
State: closed
Created 10 years ago
Comments: 76 (33 by maintainers)

Commits related to this issue

Added URL/User/Passwd CLI params; added Python3 support — committed to seancummins/unisphere-rest-client by seancummins 9 years ago
Added pyasn1 for Ubuntu 14.04LTS openssl support (see: https://github.com/kennethreitz/requests/issues/2022) — committed to userify/userify-server-tools by perpetual-hydrofoil 8 years ago
avoid kennethreitz/requests/issues/2022 — committed to c5c86a/smscgateway-docker by deleted user 7 years ago

Most upvoted comments

Sadly, this is unrelated to the issue you identified, and entirely down to the crappy OpenSSL that OS X ships with by default. Version 0.9.8y has some real problems with performing SSL handshakes, and some servers don’t tolerate it well. Using Python 3 on my OS X box (therefore using a newer OpenSSL) reveals that there’s no problem.

You have two options:

Install OpenSSL from Homebrew, then install a new version of Python 2 from Homebrew which will automatically link against the Homebrew-provided OpenSSL.
Install OpenSSL from Homebrew, and then install PyOpenSSL against that new version by running env ARCHFLAGS="-arch x86_64" LDFLAGS="-L/usr/local/opt/openssl/lib" CFLAGS="-I/usr/local/opt/openssl/include" pip install PyOpenSSL.

+33

Lukasa on Apr 26, 2014

On ubuntu 14.04LTS I needed to do this:

sudo pip install ndg-httpsclient pyasn1 --upgrade

Note that in Ubuntu it’s not possible to upgrade/remove pyopenssl as it’s owned by the OS.

markstrefford on Oct 8, 2016

So I tried pip install pyopenssl ndg-httpsclient pyasn1 instead of pip install requests[security] and that worked…

lukas-gitl on Jun 15, 2016

Just want to chime in and say that I experienced this issue on OS X 10.9.5, Python 2.7.7 and OpenSSL 0.9.8zc.

I was able to fix my handshaking issue by:

Installing a newer-than-stock OpenSSL on my machine via brew install OpenSSL
Compiling and installing the cryptography package linked against the new OpenSSL (env ARCHFLAGS="-arch x86_64" LDFLAGS="-L/usr/local/opt/openssl/lib" CFLAGS="-I/usr/local/opt/openssl/include" pip install cryptography)
Installing requests with SNI support by doing pip install requests[security]

Microserf on Feb 23, 2015

OK, I may have just answered my own question. What I did boils down to:

sudo apt-get install libffi-dev
pip install pyOpenSSL ndg-httpsclient pyasn1

lsemel on Sep 25, 2015

No need to apologise, asking that question was the right thing to do: it’s bizarrely specific knowledge to know that OS X has this problem. =)

Lukasa on Apr 26, 2014

Got it to work now! I simply uninstalled pyOpenSSL: pip uninstall pyOpenSSL

MadSpindel on Apr 1, 2016

markstrefford’s solution worked for me on mac os sierra too

contmp on Dec 18, 2016

Thanks, @Microserf. I’m pretty much running the same specs (10.9.5, Python 2.7.6 installed via Homebrew but compiled with system provided OpenSSL 0.9.8zg) and this was my entire process for getting requests up and running for Django:

brew install openssl

Install requests with a bunch of SNI stuff, compiled against our new install of OpenSSL. The [security] option simply installs pyopenssl ndg-httpsclient pyasn1

env ARCHFLAGS="-arch x86_64" LDFLAGS="-L/usr/local/opt/openssl/lib" CFLAGS="-I/usr/local/opt/openssl/include" pip install requests[security] urllib3

And we’re good to go:

"""
This may or may not be needed. See:
https://urllib3.readthedocs.org/en/latest/security.html#openssl-pyopenssl
"""
# from urllib3.contrib import pyopenssl
# pyopenssl.inject_into_urllib3()

import requests
# r = requests.get(...)

brianlittmann on Aug 21, 2015

@t-8ch I haven’t installed PyOpenSSL if that’s what you’re asking?

I would have assumed (perhaps incorrectly) that pip install requests should give me everything I need to successfully call requests.get('...') on an HTTPS page. Which, of course, it works for the most part, just not for this site for some reason.

jaddison on Apr 26, 2014

@jschwinger23 Can you run pip install pyopenssl ndg-httpsclient pyasn1 as well please?

Lukasa on Jun 25, 2016

Ah, damn. That explains a lot. Thank you very much for your help!

lukas-gitl on Jun 15, 2016

Aha, I suspect your pip is too old to handle the extras.

Lukasa on Jun 15, 2016

I had the same issue on my Ubuntu 14.04 box and Python 2.7.11

It’s from SNI

What worked for me was this:

uninstall requests
uninstall urllib3
install the various crypto dependencies
install urllib3
install urllib3[secure] # just to be safe
install requests

I think there was an installation-time check on urllib3 or requests which kept things from working without the uninstall

jvanasco on May 31, 2016

@Lukasa was right. Compare:

$ openssl s_client -connect docs.apitools.com:443                              
CONNECTED(00000003)
139846853338768:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:762:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 517 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

$  openssl s_client -connect docs.apitools.com:443 -servername docs.apitools.com
... happy handshake here

t-8ch on Apr 26, 2014