scrapy: SSL handshake failure

Hi,

I’m getting a handshake error for the sites listed below:

2016-12-03 00:02:19 [scrapy] ERROR: Error downloading <GET https://apnews.com>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl23_read', 'ssl handshake failure')]>]
2016-12-03 00:03:25 [scrapy] ERROR: Error downloading <GET https://techcrunch.com>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl23_read', 'ssl handshake failure')]>]
2016-12-03 00:03:53 [scrapy] ERROR: Error downloading <GET https://medium.com>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'ssl handshake failure')]>]
2016-12-03 00:05:08 [scrapy] ERROR: Error downloading <GET https://theintercept.com>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl23_read', 'ssl handshake failure')]>]
2016-12-03 00:06:32 [scrapy] ERROR: Error downloading <GET https://www.opendemocracy.net/>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'ssl handshake failure')]>]
2016-12-03 00:07:55 [scrapy] ERROR: Error downloading <GET https://www.rt.com>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl23_read', 'ssl handshake failure')]>]
2016-12-03 00:19:53 [scrapy] ERROR: Error downloading <GET https://www.thestar.com>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'ssl handshake failure')]>]
2016-12-03 00:58:42 [scrapy] ERROR: Error downloading <GET https://www.cnet.com/>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'ssl handshake failure')]>]

What’s strange is that it works if I try each one of those sites individually using scrapy shell so I might be doing something wrong.

Here’s some information about my environment:

$ scrapy version -v
Scrapy    : 1.2.1
lxml      : 3.6.4.0
libxml2   : 2.9.4
Twisted   : 16.6.0
Python    : 2.7.12 (default, Jul  1 2016, 15:12:24) - [GCC 5.4.0 20160609]
pyOpenSSL : 16.2.0 (OpenSSL 1.0.2g-fips  1 Mar 2016)
Platform  : Linux-3.13.0-52-generic-x86_64-with-Ubuntu-16.04-xenial

Any ideas where I can look to troubleshoot the problem?

About this issue

Original URL
State: closed
Created 8 years ago
Reactions: 1
Comments: 24 (10 by maintainers)

Most upvoted comments

Getting the same error: Website need to crawl: [“https://www.labor.ny.gov/”] Installed scrapy details Scrapy : 1.6.0 lxml : 4.2.5.0 libxml2 : 2.9.8 cssselect : 1.0.3 parsel : 1.5.1 w3lib : 1.20.0 Twisted : 18.7.0 Python : 3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 13:25:24) [MSC v.1900 64 bit (AMD64)] pyOpenSSL : 19.0.0 (OpenSSL 1.0.2p 14 Aug 2018) cryptography : 2.3.1 Platform : Windows-10-10.0.17763-SP0

ERROR: Retrying <GET https://www.labor.ny.gov> (failed 1 times): [<twisted.python.failure.Failure OpenSSL.SSL.Error: [(‘SSL routines’, ‘ssl23_read’, ‘ssl handshake failure’)]>]

singhalhimanshu on Mar 15, 2019