encounter the same problems on m1 mac

That attitude is extremely unproductive. As always, please try to remember that these are open source projects and someone has to actually do the work to fully understand the problem and implement a fix. Posts like yours are at best an irritant and quite frequently actively harmful.

pip3 uninstall pyopenssl may get around to this.

This issue seems to be blocking snowflake-python-connector and hence dbt, as well as some other packages (see mentions above), from running on M1 (arm64) Macs.

Is there anyone who could take a look at this?

Is there a bounty program for this? I want this fixed for macos (apple m1 chip), or more specifically, removed from other packages depending on pyopenssl.

End goal is to normally use python ecosystem w/ nix.

Any of these options seem reasonable or is this something that will last for a few years (like python3 transition)?

I know this is almost a year later, but I too have the issue using my M1 Silicon mac. I first encountered it using Azure’s CLI and then with home-made softwares, while it was working great on Intel platforms (both darwin and linux).

Reading the answers above, does it mean every library using openssl has to made that kind of changes ?

When @tiran mentions “use another TLS library”, is this at the OS level (like replacing OpenSSL by LibreSSL or BoringSSL) or inside the Python software ?

Resolved under newest python 3.10.2 , using conda virturl env conda create -n py310 python=3.10 my laptop is Mac m1 .

tl;dr it’s a problem in pyOpenSSL. OpenSSL is fine.

I see how my comment can be understand the wrong way. I was referring to another Python TLS library such as ssl module from Python’s standard library. OpenSSL does not require executable+writable memory. The problem only effects pyOpenSSL.

It’s an implementation artifact of pyOpenSSL’s glue code that wraps OpenSSL’s C-API and makes it available for Python. pyOpenSSL uses CFFI, which is a Python interface to libffi. libffi (library for foreign function interface) uses dynamic code creation for dynamic callbacks. Every time the code passes a Python method to OpenSSL, it has to wrap the Python method into machine code, so it looks like a C function to the C-API of OpenSSL. It writes dynamic code to an executable memory page. CFFI calls this old style callbacks.

There are ways to work around the problem, but it’s complicated and lots of work. Nobody has contributed a solution yet.

The root cause of this issue has been found and fixed (for snowflake-connector-python which also affects snowflake-snowpark-python) by conda-forge/cffi-feedstock#47 (files) for Conda users (a simple conda install cffi should update you to build number 3 of version 1.15.1 has the fix included). The solution itself is described at foss.heptapod.net/pypy/cffi/-/blob/branch/default/c/_cffi_backend.c#L64-89. This macro just needs to be enabled for cffi to build correctly on M1 macs.

@candlerb I’ve updated requests module from 2.21 to 2.26 and the problem was gone (probably because requests module no longer uses pyssl: https://docs.python-requests.org/en/latest/community/updates/

2.26.0 (2021-07-13) The requests[security] extra has been converted to a no-op install. PyOpenSSL is no longer the recommended secure option for Requests. (#5867)

-requests==2.21.0
+requests==2.26.0

Same issue with the new macbook pros M1 (no rosetta)

@WittmannF Yep. Until build 3 of cffi is available, I have been activating the conda/mamba env and:

(my_env) $ pip install --force-reinstall 'cffi>=1.15.1'

I don’t think it’s codesigning that lets it work. I did a diff of Apple’s libffi from their OSS release and upstream. There appear to be changes that they’re still working on upstreaming. For example: https://github.com/libffi/libffi/pull/621.

I’m building nixpkgs’s libffi with this patch to see if that allows PyOpenSSL to work on aarch64-darwin. Unfortunately, I will be away for most of the day and won’t be able to check the results before this evening EDT.

I have tested the latest verion of pyopenssl with Python 3.10.0 and 3.9.9 on M1 MacBook Pro, it turns out (probably) a problem of the Python Interpreter on M1. The Python interpreter 3.10 (might) change the ways to access(write) memory on M1 chips.

It works on both Python 3.10.0 and 3.9.9 (the scrapy shell). I need to install PyTorch, but PyTorch isn’t working with Python 3.10 right now… And when I was tring to build pytorch from source with M1, pytorch dragged me back to install conda before compiling ,which only supports upto python 3.9.7 right now. It’s like a dead loop…

Still having this issue on apple m1 2020, when launching API calls via asyncio [python 3.8.12 & natively installed - no rosetta here]. Hope that folks buying new MacBook Pro with M1 chip will push to fix this kind of issues 🤞

@WittmannF You are running build 2 of cffi (py310h2399d43_2) and that does not contain the patch @sfc-gh-mkeller shared. The patch is not yet merged and a new build published yet,.

@sfc-gh-mkeller Thanks for the pointer! It appears the changes have not been merged yet though. Hopefully, the change will be merged and build 3 will be available in the condo-forge channel soon.

Snowflake requires Python 3.8 (as of 09/30/2022) and Python 3.10 doesn’t fix it for Snowflake issue.

The fix for Snowflake is here: https://docs.snowflake.com/en/developer-guide/snowpark/python/setup.html#prerequisites

Reposting here:

CONDA_SUBDIR=osx-64 conda create -n snowpark python=3.8 numpy pandas -c https://repo.anaconda.com/pkgs/snowflake conda activate snowpark conda config --env --set subdir osx-64

pyOpenSSL is not compatible with systems that prevent writeable and executable memory pages. You either have to disable the security feature, use another TLS library like the builtin ssl module, or redesign+rewrite pyOpenSSL’s callback system.

The patch @sfc-gh-mkeller found is merged and build 3 is available in the condo-forge channel. The above workaround I mentioned above is no longer necessary.

Last time I checked @ffi.def_extern() was using a single, static C global to store a pointer to the callback function. The approach is incompatible with multiple subinterpreters. If that’s still the case then you need a much more complicated implementation.

Would someone with more experience and insight into this project be willing to comment on the following proposal? I would be willing to work on it, but I want to sanity check it.

I’m only going to focus on uses of the old-style CFFI ffi.callback function in SSL.py for now. In these usages, the callback functions have two properties:

1. They receive an argument that can be used to resolve to an OpenSSL SSL struct.
2. They are saved as instance members of either Connection or Context.

For each usage of ffi.callback, could we therefore write a generic version that is built and made available through the new CFFI extern Python approach, which does the following:

@ffi.def_extern()
def my_generic_callback(ssl, ...):
  # Look up Connection from ssl
  # If necessary, look up Context from Connection
  # Find the actual callback saved on the Connection or Context instance
  # Call it

Unfortunately the issue here is that these errors don’t reproduce with arbitrary Python on M1. I just tested that on my M1 Max running 3.10 from pyenv and had no error.

I have tested the latest verion of pyopenssl with Python 3.10.0 and 3.9.9 on M1 MacBook Pro, it turns out (probably) a problem of the Python Interpreter on M1. The Python interpreter 3.10 (might) change the ways to access(write) memory on M1 chips.

I also meet that question! @Alexander Liu I think you can use mini-conda to install torch where python version is 3.8, and use python which version is 3.10 to install scrapy.

Figured it out following @anribras 's advice. M1 mac, had the same issue here. As @anribras says, I upgraded Python in my conda env to 3.10.2 and now this issue is resolved.

FWIW, I’ve been following this issue for about a year after (like the OP) encountering it on Scrapy on my M1 MBP (0, 1). I’ve come back to it every few months to see if it was still crashing (and it has been).

After seeing @alex’s comment I wanted to see if I could contribute to the test suite, and lo and behold the crash is gone.

• M1 Macbook Pro
• MacOS 12.0.1
• Python 3.10.0
• Scrapy==2.5.1

My test case has always been:

$ scrapy shell 'http://httpforever.com/' # <- succeeds
$ scrapy shell 'https://n8henrie.com' # <- crashes with `Cannot allocate write+execute memory for ffi.callback()`

For me, both seem to now work without error on http or https sites, so perhaps some of the others following this thread might recheck to see if it is still an issue for them or not.

Yeah, seems like there’s a ready way to fix the code by changing to new-style “Extern Python” callbacks in CFFI. Here’s another project that ran into the same issue and fixed it that way, with a real fast turnaround:

Issue: https://github.com/FSX/misaka/issues/72 Commit: https://github.com/FSX/misaka/commit/321357b192235438aaa94061626a1fd7c3b3c67b#diff-cd1259298a2efe394f2d1bbea8b0d39e

I’m not familiar enough with the pyOpenSSL codebase to risk giving it a try, but otherwise it seems fairly straightforward - and would add some performance too, according to CFFI.