scrapy: OpenSSL unsafe legacy renegotiation disabled error

I ran into the same problem, and built a custom context factory to solve it. I hope it will be helpful for others:

contextfactory.py (within MyProject folder)

from scrapy.core.downloader.contextfactory import ScrapyClientContextFactory


class LegacyConnectContextFactory(ScrapyClientContextFactory):

    def getContext(self, hostname=None, port=None):
        ctx = self.getCertificateOptions().getContext()
        ctx.set_options(0x4)
        return ctx

within my_spider.py

class MySpider(Spider):
    name = "my_spider"

    custom_settings = {
        'DOWNLOADER_CLIENTCONTEXTFACTORY': 'MyProject.contextfactory.LegacyConnectContextFactory',
    }

@wRAR fixed by downgrading cryptography to 36.0.2 Here’s my current scrapy version --verbose result

Scrapy       : 2.6.1
lxml         : 4.8.0.0
libxml2      : 2.9.4
cssselect    : 1.1.0
parsel       : 1.6.0
w3lib        : 1.22.0
Twisted      : 22.4.0
Python       : 3.9.12 (main, Mar 26 2022, 15:44:31) - [Clang 13.1.6 (clang-1316.0.21.2)]
pyOpenSSL    : 22.0.0 (OpenSSL 1.1.1n  15 Mar 2022)
cryptography : 36.0.2
Platform     : macOS-12.3.1-arm64-arm-64bit

Now pyOpenSSL uses OpenSSL 1.1.1n Even though the problem is fixed, the flags enhancement would be great. Thank you so much @wRAR !

I want to drop this relevant link here:

https://stackoverflow.com/questions/71603314/ssl-error-unsafe-legacy-renegotiation-disabled/72374542#72374542

Here is a very useful piece of code to circumvent the issue without downgrading:

import requests
import urllib3
import SSL


class CustomHttpAdapter (requests.adapters.HTTPAdapter):
    # "Transport adapter" that allows us to use custom ssl_context.

    def __init__(self, ssl_context=None, **kwargs):
        self.ssl_context = ssl_context
        super().__init__(**kwargs)

    def init_poolmanager(self, connections, maxsize, block=False):
        self.poolmanager = urllib3.poolmanager.PoolManager(
            num_pools=connections, maxsize=maxsize,
            block=block, ssl_context=self.ssl_context)


def get_legacy_session():
    ctx = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)
    ctx.options |= 0x4  # OP_LEGACY_SERVER_CONNECT
    session = requests.session()
    session.mount('https://', CustomHttpAdapter(ctx))
    return session

with (
    get_legacy_session() as s,
    s.get("some-url") as response
):
    print(response.json())

It’s not “inside of the scrapy package”, it’s in your Python environment.

I ran into the same problem, and built a custom context factory to solve it. I hope it will be helpful for others:

contextfactory.py (within MyProject folder)

from scrapy.core.downloader.contextfactory import ScrapyClientContextFactory


class LegacyConnectContextFactory(ScrapyClientContextFactory):

    def getContext(self, hostname=None, port=None):
        ctx = self.getCertificateOptions().getContext()
        ctx.set_options(0x4)
        return ctx

within my_spider.py

class MySpider(Spider):
    name = "my_spider"

    custom_settings = {
        'DOWNLOADER_CLIENTCONTEXTFACTORY': 'MyProject.contextfactory.LegacyConnectContextFactory',
    }

thanks.

I ran into the same problem, and built a custom context factory to solve it. I hope it will be helpful for others:

contextfactory.py (within MyProject folder)

from scrapy.core.downloader.contextfactory import ScrapyClientContextFactory


class LegacyConnectContextFactory(ScrapyClientContextFactory):

    def getContext(self, hostname=None, port=None):
        ctx = self.getCertificateOptions().getContext()
        ctx.set_options(0x4)
        return ctx

this code works too.

contextfactory.py

from scrapy.core.downloader.contextfactory import ScrapyClientContextFactory
from cryptography.hazmat.bindings.openssl.binding import Binding


class LegacyConnectContextFactory(ScrapyClientContextFactory):

    def getContext(self, hostname=None, port=None):
        ctx = super(LegacyConnectContextFactory, self).getContext()
        binding = Binding()
        ctx.set_options(binding.lib.SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION)
        return ctx

Thanks @aysegulc, your suggestion works.