snowflake-connector-python: SNOW-507597: Not supported error on Apple M1

We’re currently working on releasing ARM builds for our Python connector

@sfc-gh-hchaturvedi I’m also facing this issue - just wondering if there is any progress on this?

@kzecchini god, that’s an unfortunate need of quotes! Thanks for pointing that out!

I’ve noticed the M1 arm vs universal issue too. I know that pip will always prefer specific wheels over universal wheels if available. I notice the arm ones are not available on pypi.

If you force building the wheels, then it will work on M1 mac: pip install 'snowflake-connector-python[pandas,secure-local-storage]' --no-binary snowflake-connector-python

Hi,

I was facing the same error (M1 mac), and this thread really helped me out. The suggestion of using --non-binary option with pip worked for me. So, Thank you so much for that @thomasaarholt.

However, during this, I discovered another issue, although after spending some time debugging it. I kinda find a solution for it, but I’m still not sure why that error was happening. I thought it would be a good idea to put it here, in case of some face the same issue as me, it might help them. So, someone smarter and well-versed, in this topic, than me (like @thomasaarholt ) might be able to explain to me why it was happening.

So, after you use the --no-binary option posted above, snowflake-connector-python will work as expected, however, if you install certain libraries after this, for example, jupyterlab, you will again start getting the error:

MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks

This is the same error I was getting while trying to run snowflake.connector.connect function when I installed snowflake-connector-python using simply conda install snowflake-connector-python and went away when I installed using the pip with --no-binary option.

Just to give the EXACT steps I followed, in case, someone wants to recreate it:

conda create -n XYZ
conda activate XYZ

conda install numpy pandas=1.35 matplotlib

conda install pycryptodomex

pip install 'snowflake-connector-python[pandas,secure-local-storage]' --no-binary snowflake-connector-python

At this point, running snowflake.connector.connect gives no error, and run as expected.

conda install jupterlab

Everything here installs perfectly but running snowflake.connector.connect gives the Memory error I mentioned above.

After debugging it for some time, I figure out the cause of it. Basically, it is not jupyterlab (or other similar packages) that is causing the issue, but one of its dependencies - CFFI (https://cffi.readthedocs.io/en/latest/).

CFFI is installed with snowflake-connector-python as one of it’s dependencies and the package works perfectly, but when you install jupyterlab conda installs cffi again (as we know conda doesn’t check for packages installed by PyPI). You can recreate the same error just by doing conda install cffi right after you install the snowflake connector with pip --no-binary option.

Work around I found, just install jupyterlab or any other package which has cffi as it’s dependency through PYPI in the conda virtual environment.

However, I’m still confused why does cffi installation from PyPI works, but from Conda doesn’t, even though conda-forge has osx-arm compatible installer for cffi. My guess is - this dependency is the same reason why conda install snowflake-connector-python doesn’t work despite it having an osx-arm compatible installer.

I hope it helps anyone who might struggle with this issue, or if someone can tell me why this issue is happening in the first place!

In my case, the solution we found is before running Pipfile install pycryptodomex==3.10.3 (the version where they fix the M1 issue).

We’re currently working on releasing ARM builds for our Python connector

Any update soon? cheers

@sfc-gh-hchaturvedi I successfully just installed and ran dbt-snowflake on an M1 Macbook Pro using latest snowflake-connector-python master branch. There the pyarrow dependency is no longer a problem (current published latest version has a dependency issue with pyarrow 5).

Could I ask the snowflake team to please publish a patch version from latest master (remember to bump the versio number 😃 )? I think that will solve everyone’s problem.

On a M1 Macbook:

# Create a python 3.9 environment (3.10 did *not* work with dbt, I think there aren't arm wheel for 3.10?)
# I did `conda create --name dbt python=3.9`
# conda activate dbt
pip install https://github.com/snowflakedb/snowflake-connector-python/archive/refs/heads/master.zip # install snowflake-connector-python master
pip install dbt-core
# check `which dbt`, ensure it is the right one, otherwise reactivate environment
# call `dbt`, this should work
pip install dbt-snowflake

dbt deps # fix new dependencies
dbt run -m your_model

Here’s the python process during dbt run, red line highlighting that this is native arm. Otherwise it would say “Intel”.

So the problem that this issue is discussing:

snowflake.connector.errors.NotSupportedError: Unknown error

means that this package’s c-extension is not available.

I can see this being a wide spread issue before pyarrow started releasing aarch64 wheels, but this should not really be an issue anymore. The flag --no-binary snowflake-connector-python should force source distribution installation and that should be able to compile the c-extension correctly.

I think Anaconda environments could have more issues, but I’m not familiar with them.

@thomasaarholt Thanks for the info! I just verified that adding the --no-binary works as well for me. I never used that before - thanks for pointing that out!

So it just seems like the solution for PyPI would be to get the ARM distribution uploaded so folks could easily install a prebuilt wheel that works on M1.

But the workaround for now is to either install from source, or to use the --no-binary flag when installing via PyPI.

Also wanted to add that installing via:

pip install git+https://github.com/snowflakedb/snowflake-connector-python.git

also works for M1 instead of the PyPI install.