scrapy: TLS handshake failure

I have this simple spider:

import scrapy


class FailingSpider(scrapy.Spider):
    name = 'Failing Spider'
    start_urls = ['https://www.skelbiu.lt/']

    def parse(self, response: scrapy.http.Response) -> None:
        pass

On debian 9 it fails with:

2017-04-25 19:01:39 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.skelbiu.lt/>
Traceback (most recent call last):
  File "/home/povilas/projects/skelbiu-scraper/pyenv/lib/python3.6/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/povilas/projects/skelbiu-scraper/pyenv/lib/python3.6/site-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/povilas/projects/skelbiu-scraper/pyenv/lib/python3.6/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'sslv3 alert handshake failure')]>]

On debian 8 it works well. And “https://www.skelbiu.lt” is the only target I can reproduce the problem.

Some more context:

$ pyenv/bin/pip freeze
asn1crypto==0.22.0
attrs==16.3.0
Automat==0.5.0
cffi==1.10.0
constantly==15.1.0
cryptography==1.8.1
cssselect==1.0.1
funcsigs==0.4
idna==2.5
incremental==16.10.1
lxml==3.7.3
mock==1.3.0
packaging==16.8
parsel==1.1.0
pbr==3.0.0
py==1.4.33
pyasn1==0.2.3
pyasn1-modules==0.0.8
pycparser==2.17
PyDispatcher==2.0.5
PyHamcrest==1.8.5
pyOpenSSL==17.0.0
pyparsing==2.2.0
pytest==2.7.2
queuelib==1.4.2
Scrapy==1.3.3
service-identity==16.0.0
six==1.10.0
Twisted==17.1.0
w3lib==1.17.0
zope.interface==4.4.0

$ dpkg --get-selections | grep libssl
libssl-dev:amd64                                install
libssl-doc                                      install
libssl1.0.2:amd64                               install
libssl1.1:amd64                                 install
libssl1.1:i386                                  install


$ apt-cache show libssl1.1
Package: libssl1.1
Source: openssl
Version: 1.1.0e-1

Any ideas what I should look for? 😃

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 2
Comments: 27 (18 by maintainers)

Most upvoted comments

Yeah, it looks like an EC thing:

using Twisted trunk and patching this line with

-_defaultCurveName = u"prime256v1"
+_defaultCurveName = u"secp384r1"

made the connection to ‘https://www.skelbiu.lt/’ work for me.

Now, I’ll have a look at how to properly configure this with Twisted Agent.

redapple on Apr 26, 2017

Right @tonal . https://www.teplodvor.ru/ does not look compatible with OpenSSL 1.1 (some weak ciphers were removed). Downgrading to cryptography<2 , which ships with OpenSSL 1.0.2 (at least for me on Ubuntu), makes it work.

redapple on Oct 9, 2017

Unfortunately, changing TLS version does not help.

povilasb on Apr 25, 2017