dbt-core: DBT cli throw memory exception for ffi.callback() on Mac with Apple Sillicon
Describe the bug
dbt run or dbt --version throw error Cannot allocate write+execute memory for ffi.callback().
Steps To Reproduce
On Macbook pro with M1 chip
brew tap fishtown-analytics/dbt
brew install dbt
add default profile.yml with connection to snowflake
create new project dbt init
dbt run with the two sample models, or simply dbt --version
Expected behavior
Should complete successfully, instead of throwing exception. The process works fine on Intel-based Macbook pro.
Screenshots and log output
> dbt run
Running with dbt=0.19.0
Found 3 models, 8 tests, 0 snapshots, 0 analyses, 143 macros, 0 operations, 0 seed files, 0 sources, 0 exposures
Encountered an error:
Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks
System information
Which database are you using dbt with?
- postgres
- redshift
- bigquery
- snowflake
- other (specify: ____________)
The output of dbt --version:
Encountered an error:
Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks
The operating system you’re using: macOS Big Sur 11.2.2
The output of python --version:
3.8.7
Additional context
Add any other context about the problem here.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 5
- Comments: 24 (9 by maintainers)
TL;DR: if you experience the error during
dbt run, just add those lines to~/.dbt/profiles.yml(see the full doc in profiles.yml here)Details:
Here’s the stack trace (
MacBook Air (M1, 2020)runningMac OS 11.3.1). The exception is raised in user behavior tracking code. Quickfix will be disabling tracking (see above). I guess the proper way to handle this would be wrappingtracker.flush()withtry/except. Failure to track isn’t a critical error after allHey guys,
I fixed this ffi problem by reinstalling the libffi using homebrew, and install cffi wheel: https://files.pythonhosted.org/packages/3e/9b/660d6da900af1976a8b4efea713a7ce9e514bf4659eff9b17f90f00be1cf/cffi-1.15.0-cp39-cp39-macosx_11_0_arm64.whl
Hey @alyiwang, I was having similar issues and I was able to work through it using the following:
brew uninstall dbtif you already tried installingwhich dbtreturned nothing (at this point I had multiple installs of dbt that I had to get rid ofpip install virtualenvand created a virtual environment. Link for more info on those https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/pip install dbt==0.18.0and let it run it’s course. Following installation I randbt --versionwhile the environment was still active and it worked. It’s worth noting that if you go this route of installation, the virtual environment must first be activated before running any dbt commands and deactivated when complete@alyiwang This is absolutely our plan for the next minor version of dbt (v0.20.0).
dbt-corealready supports python3.9, it’s just a matter of upgrading our pin on the Snowflake connector.@tpark464 I would recommend using the latest version of dbt and the associated adapter (
brew install dbt-redshift). We are on v1.2 at the moment. The older versions of dbt in Homebrew are not actively maintained and have breaking dependency changes over the long time period@brittianwarner We didn’t include the proposed fix above in v0.20.0. You’re welcome to try turning off anonymous tracking, and see if that fixes it.
Over the past few weeks, there have been a few reports of successful installs on M1s by other means: https://github.com/dbt-labs/dbt/issues/3239#issuecomment-868996761
@vklimontovich Have you been able to confirm that the addition of that try/except results in error-free runs? (i.e. by cloning
dbt, editing, and installing from local)That’s fine as a temporary solution—if you can confirm it works, I’d welcome a PR for it. In addition, I’d like to find a way to work around the
pyOpenSSLbug, if possible.We’ve just run headfirst into this issue with a colleague using a new macbook 😂 ; any rough timelines for a v0.20.0 pre-release @jtcohen6?
Just ran into the issue as well today on my new work computer. If there is anything I can do to help out I’d be happy to chip in. The
snowflake-python-connectorcontinues to be my biggest pain in the behind with their dependency management.