python-rocksdb: Build Fails on MacOS

export CFLAGS=-stdlib=libc++
pip install python-rocksdb

this works for me on macOS 10.14.3

The following procedure worked for me on:

• MacBook Pro M1
• macOS 12.2.1 Monterey
• rocksdb 6.29.3

Install rocksdb with brew:

brew install rocksdb

Use brew to look up the path on your system of the libs required for the install of python-rocksdb:

brew info rocksdb lz4 snappy

I had all libs already installed, maybe due to the install of rocksdb. If libs are missing, I guess you need to install them.

Export (for gcc?) the paths of the libs with CPPFLAGS and LDFLAGS and install python-rocksdb with pip. For me that was (adapt paths according to your setup):

export CPPFLAGS=-I/opt/homebrew/Cellar/rocksdb/6.29.3/include
export LDFLAGS="-L/opt/homebrew/Cellar/rocksdb/6.29.3/lib -L/opt/homebrew/Cellar/snappy/1.1.9/lib -L/opt/homebrew/Cellar/lz4/1.9.3/lib"
pip install python-rocksdb

The exports fixed also the poetry install of a poetry project with faust-streaming-rocksdb (faust-streaming = {extras = ["rocksdb"], version = "^0.8.2"}) as dependency.

None of the above solutions work on macOS 12.2, I’m afraid…

Installing rocksdb with brew helped: brew install rocksdb

I’m getting rocksdb/_rocksdb.cpp:622:10: fatal error: 'rocksdb/slice.h' file not found

Above did not help.

I had the same problem and for me it seems like the problem is, that python-rocksdb only supports rocksdb 6 and not the latest one (present in Homebrew).

The following approach worked for me (use at your own risk):

1. Install old Rocksdb

brew tap-new $USER/local-rocksdb
brew extract --version=6.29.3 rocksdb $USER/local-rocksdb
brew install rocksdb@6.29.3

1. Export Buildflags

export CPPFLAGS=-I/usr/local/Cellar/rocksdb@6.29.3/6.29.3/include
export LDFLAGS="-L/usr/local/Cellar/rocksdb@6.29.3/6.29.3/lib -Lusr/local/Cellar/snappy/1.1.9/lib -L/usr/local/Cellar/lz4/1.9.4/lib"

1. Install packages

pip install python-rocksdb

@alexreg thanks for the explanation, but pardon my ignorance. When I try to poetry install your branch after installing rocksdb with Homebrew (with brew install rocksdb). I keep getting the following error. (I’m on M1 Mac pro)

.... 
    rocksdb/_rocksdb.cpp:819:10: fatal error: 'rocksdb/utilities/backupable_db.h' file not found
    #include "rocksdb/utilities/backupable_db.h"
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    3 warnings and 1 error generated.
    error: command '/usr/bin/clang' failed with exit code 1
    [end of output]
...

I’ve seen in your conversation with NightTsarina, especially in https://github.com/NightTsarina/python-rocksdb/pull/13#issuecomment-1062002636 that this may be deprecated. But I can’t find a workaround for this. Any ideas how to go about this?

Regarding my previous comment in this thread:

The following procedure worked for me on:

MacBook Pro M1 macOS 12.2.1 Monterey rocksdb 6.29.3 Install rocksdb with brew:

brew install rocksdb Use brew to look up the path on your system of the libs required for the install of python-rocksdb:

brew info rocksdb lz4 snappy I had all libs already installed, maybe due to the install of rocksdb. If libs are missing, I guess you need to install them.

Export (for gcc?) the paths of the libs with CPPFLAGS and LDFLAGS and install python-rocksdb with pip. For me that was (adapt paths according to your setup):

export CPPFLAGS=-I/opt/homebrew/Cellar/rocksdb/6.29.3/include export LDFLAGS=“-L/opt/homebrew/Cellar/rocksdb/6.29.3/lib -L/opt/homebrew/Cellar/snappy/1.1.9/lib -L/opt/homebrew/Cellar/lz4/1.9.3/lib” pip install python-rocksdb The exports fixed also the poetry install of a poetry project with faust-streaming-rocksdb (faust-streaming = {extras = [“rocksdb”], version = “^0.8.2”}) as dependency.

I Followed this same procedure just now again in order to successfully install python-rocksdb on my new computer (Apple M1 Max, 2021).

Crucial missing information from my original post: the procedure works only for rocksdb v6. If you run now brew install rocksdb, you install version 7+ and for version 7+ the procedure does not work.

For me rocksdb v6 is fine, so I used this procedure to install rocksdb v6 with brew. I also had to change the path to the lz4 lib, now /opt/homebrew/Cellar/lz4/1.9.4/lib.

Hello @alexreg, I’m experiencing the same issue as @rollue. We’re blocked because there’s no easy way to install rocksdb v6 on mac (with brew). I’ll try to find a workaround

I’m getting rocksdb/_rocksdb.cpp:622:10: fatal error: 'rocksdb/slice.h' file not found

Above did not help.

I get same problem, guess we need install rocksdb in mac and make complier link to proper headers

I managed to make it work on a MacBook Air M1 with RocksDB 6.29.3

git clone https://github.com/Homebrew/homebrew-core
cd homebrew-core
git checkout b3597b729a2
brew install Formula/rocksdb.rb
brew pin rocksdb

export CPPFLAGS="-I/opt/homebrew/Cellar/rocksdb/6.29.3/include"
export LDFLAGS="-L/opt/homebrew/Cellar/rocksdb/6.29.3/lib -L/opt/homebrew/Cellar/snappy/1.1.10/lib -L/opt/homebrew/Cellar/lz4/1.9.4/lib"

pip install python-rocksdb

For any other package and version you can check this gist.

Here is how I made it to work.

Setup:

1. brew info rocksdb lz4 snappy find out if there is any missing package and install it.
2. brew install rocksdb (this is to install latest rocksdb, v7)
3. brew install alexreg/dev/rocksdb@6
4. Find out where you installed the rocksdb in step 2 and 3, for me it’s /usr/local/Cellar/rocksdb@6 and /usr/local/Cellar/rocksdb a. export CPPFLAGS="-I/usr/local/Cellar/rocksdb@6/6.29.5/include -I/usr/local/Cellar/rocksdb/7.7.2/include" (I had to include both 6 and 7 header files because the install seems to need both). b. export LDFLAGS="-L/usr/local/Cellar/rocksdb@6/6.29.5/lib -L/usr/local/Cellar/rocksdb/7.7.2/lib -L/usr/local/Cellar/snappy/1.1.9/lib -L/usr/local/Cellar/lz4/1.9.4/lib"

Install

1. downloaded tar.gz file and unzip it https://pypi.org/project/python-rocksdb/#files
2. go to the directory and vim setup.py, change extra_compile_args “-std=c++” to “-std=c++17” because c++17 functions are used.
3. python3 setup.py install

Good luck!

@Littlecowherd I faced with the same problem and managed to solve it by moving from anaconda to simple venv or poetry. I’m not sure why it works so, but I suggest anyone to try it. It worked for me.

Hello @alexreg, I’m experiencing the same issue as @rollue.

We’re blocked because there’s no easy way to install rocksdb v6 on mac (with brew). I’ll try to find a workaround

brew install alexreg/dev/rocksdb@6

I tried to get the formula accepted in homebrew-core but sadly no luck. At least this should make it easy though.

@faberchri Thank you for sharing, although that didn’t work for me either. The only thing that worked in the end was to use the more recent fork of this library, patch that for the latest rocksdb (just removing a few definitions), and then build under brew sh. I’ll try to get a PR submitted to that project.