tensorflow: Library not loaded: @rpath/libcudart.8.0.dylib when running TF GPU on MacOS

Something related to linking seems to cause the build to fail on Mac OS 10.12.2 when building with CUDA.

Verbose build error is:

$ bazel build -c opt --config=cuda --verbose_failures //tensorflow/tools/pip_package:build_pip_package
INFO: Found 1 target...
ERROR: /Users/anton/tmp/tensorflow/tensorflow/contrib/ffmpeg/BUILD:66:1: Executing genrule //tensorflow/contrib/ffmpeg:decode_audio_op_py_pygenrule failed: bash failed: error executing command 
  (cd /private/var/tmp/_bazel_anton/d3361bfa15c75c42ab541f3d83e8eba4/execroot/tensorflow && \
  exec env - \
    PATH=/usr/local/cuda/bin:/Users/anton/.pyenv/shims:/Users/anton/.rbenv/shims:/Users/anton/.scalaenv/shims:/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Users/anton/unix/bin:/usr/local/cuda/bin \
    TMPDIR=/var/folders/4r/mq7ht1z11t72w5b5b6014zwm0000gn/T/ \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/host/bin/tensorflow/contrib/ffmpeg/gen_decode_audio_op_py_py_wrappers_cc 1 > bazel-out/local_darwin-py3-opt/genfiles/tensorflow/contrib/ffmpeg/ops/gen_decode_audio_op_py.py'): com.google.devtools.build.lib.shell.AbnormalTerminationException: Process terminated by signal 6.
dyld: Library not loaded: @rpath/libcudart.8.0.dylib
  Referenced from: /private/var/tmp/_bazel_anton/d3361bfa15c75c42ab541f3d83e8eba4/execroot/tensorflow/bazel-out/host/bin/tensorflow/contrib/ffmpeg/gen_decode_audio_op_py_py_wrappers_cc
  Reason: image not found
/bin/bash: line 1: 72493 Abort trap: 6           bazel-out/host/bin/tensorflow/contrib/ffmpeg/gen_decode_audio_op_py_py_wrappers_cc 1 > bazel-out/local_darwin-py3-opt/genfiles/tensorflow/contrib/ffmpeg/ops/gen_decode_audio_op_py.py
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 2.707s, Critical Path: 0.32s

It seems that Caffe had similar issues due to Apple dropping the LD_LIBRARY_PATH environment variable: https://github.com/BVLC/caffe/issues/3227

Environment info

Operating System: Mac OS 10.12.2

Installed version of CUDA and cuDNN:

$ ls -l /usr/local/cuda/lib/libcud*
-rwxr-xr-x  1 root  wheel  13504 Nov  3 19:39 /usr/local/cuda/lib/libcuda.dylib*
lrwxr-xr-x  1 root  wheel     45 Nov  3 19:40 /usr/local/cuda/lib/libcudadevrt.a@ -> /Developer/NVIDIA/CUDA-8.0/lib/libcudadevrt.a
lrwxr-xr-x  1 root  wheel     50 Nov  3 19:40 /usr/local/cuda/lib/libcudart.8.0.dylib@ -> /Developer/NVIDIA/CUDA-8.0/lib/libcudart.8.0.dylib
lrwxr-xr-x  1 root  wheel     46 Nov  3 19:40 /usr/local/cuda/lib/libcudart.dylib@ -> /Developer/NVIDIA/CUDA-8.0/lib/libcudart.dylib
lrwxr-xr-x  1 root  wheel     49 Nov  3 19:40 /usr/local/cuda/lib/libcudart_static.a@ -> /Developer/NVIDIA/CUDA-8.0/lib/libcudart_static.a
lrwxr-xr-x  1 root  admin     47 Jan  8 16:48 /usr/local/cuda/lib/libcudnn.5.dylib@ -> /Developer/NVIDIA/CUDA-8.0/lib/libcudnn.5.dylib
lrwxr-xr-x  1 root  admin     45 Jan  8 16:48 /usr/local/cuda/lib/libcudnn.dylib@ -> /Developer/NVIDIA/CUDA-8.0/lib/libcudnn.dylib
lrwxr-xr-x  1 root  admin     48 Jan  8 16:48 /usr/local/cuda/lib/libcudnn_static.a@ -> /Developer/NVIDIA/CUDA-8.0/lib/libcudnn_static.a

Configuration command

Please specify the location of python. [Default is /Users/anton/.pyenv/shims/python]: 
Do you wish to build TensorFlow with Google Cloud Platform support? [y/N] N
No Google Cloud Platform support will be enabled for TensorFlow
Do you wish to build TensorFlow with Hadoop File System support? [y/N] N
No Hadoop File System support will be enabled for TensorFlow
Error in sitecustomize; set PYTHONVERBOSE for traceback:
KeyError: 'PYTHONPATH'
Error in sitecustomize; set PYTHONVERBOSE for traceback:
KeyError: 'PYTHONPATH'
Error in sitecustomize; set PYTHONVERBOSE for traceback:
KeyError: 'PYTHONPATH'
Found possible Python library paths:
  /Users/anton/.pyenv/versions/anaconda3-4.1.1/lib/python3.5/site-packages
Please input the desired Python library path to use.  Default is [/Users/anton/.pyenv/versions/anaconda3-4.1.1/lib/python3.5/site-packages]

Using python library path: /Users/anton/.pyenv/versions/anaconda3-4.1.1/lib/python3.5/site-packages
Do you wish to build TensorFlow with OpenCL support? [y/N] 
No OpenCL support will be enabled for TensorFlow
Do you wish to build TensorFlow with CUDA support? [y/N] Y
CUDA support will be enabled for TensorFlow
Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]: 
Please specify the CUDA SDK version you want to use, e.g. 7.0. [Leave empty to use system default]: 
Please specify the location where CUDA  toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: 
Please specify the Cudnn version you want to use. [Leave empty to use system default]: 
Please specify the location where cuDNN  library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: 
libcudnn.dylib resolves to libcudnn.dylib
Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size.
[Default is: "3.5,5.2"]: 3.5  	 
Extracting Bazel installation...
............
INFO: Starting clean (this may take a while). Consider using --expunge_async if the clean takes more than several minutes.
.........
____Loading package: tensorflow/tools/git
____Loading package: tensorflow/contrib/opt
____Downloading http://bazel-mirror.storage.googleapis.com/github.com/google/protobuf/archive/008b5a228b37c054f46ba478ccafa5e855cb16db.tar.gz: 40,960 bytes
____Downloading http://bazel-mirror.storage.googleapis.com/github.com/google/protobuf/archive/008b5a228b37c054f46ba478ccafa5e855cb16db.tar.gz: 444,518 bytes
____Downloading http://bazel-mirror.storage.googleapis.com/github.com/google/protobuf/archive/008b5a228b37c054f46ba478ccafa5e855cb16db.tar.gz: 743,716 bytes
____Downloading http://bazel-mirror.storage.googleapis.com/github.com/google/protobuf/archive/008b5a228b37c054f46ba478ccafa5e855cb16db.tar.gz: 1,170,534 bytes
____Downloading http://bazel-mirror.storage.googleapis.com/github.com/google/protobuf/archive/008b5a228b37c054f46ba478ccafa5e855cb16db.tar.gz: 1,597,352 bytes
____Downloading http://bazel-mirror.storage.googleapis.com/github.com/google/protobuf/archive/008b5a228b37c054f46ba478ccafa5e855cb16db.tar.gz: 1,998,646 bytes
____Downloading http://bazel-mirror.storage.googleapis.com/github.com/google/protobuf/archive/008b5a228b37c054f46ba478ccafa5e855cb16db.tar.gz: 2,251,050 bytes
____Downloading http://bazel-mirror.storage.googleapis.com/github.com/google/protobuf/archive/008b5a228b37c054f46ba478ccafa5e855cb16db.tar.gz: 2,675,032 bytes
____Downloading http://bazel-mirror.storage.googleapis.com/github.com/google/protobuf/archive/008b5a228b37c054f46ba478ccafa5e855cb16db.tar.gz: 3,144,390 bytes
____Downloading http://bazel-mirror.storage.googleapis.com/github.com/google/protobuf/archive/008b5a228b37c054f46ba478ccafa5e855cb16db.tar.gz: 3,579,716 bytes
____Downloading http://bazel-mirror.storage.googleapis.com/github.com/google/protobuf/archive/008b5a228b37c054f46ba478ccafa5e855cb16db.tar.gz: 3,788,162 bytes
INFO: All external dependencies fetched successfully.
Configuration finished

Git revision

$ git rev-parse HEAD
ec7929b878926c39255254e9aea992f0bc65aa68

Bazel version

$ bazel version
Build label: 0.4.3-homebrew
Build target: bazel-out/local-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Dec 22 15:20:15 2016 (1482420015)
Build timestamp: 1482420015
Build timestamp as int: 1482420015

Update

I modified third_party/gpus/crosstool/CROSSTOOL.tpl the following lines

  cxx_flag: "-std=c++11"
  linker_flag: "-Wl,-no-as-needed"

in the toolchain section to:

  cxx_flag: "-std=c++11"
  linker_flag: "-Wl,-no-as-needed,-rpath,/usr/local/cuda/lib"

Now I get failures at:

ERROR: /Users/anton/tmp/tensorflow/tensorflow/python/BUILD:793:1: Executing genrule //tensorflow/python:control_flow_ops_pygenrule failed: bash failed: error executing command 
  (cd /private/var/tmp/_bazel_anton/d3361bfa15c75c42ab541f3d83e8eba4/execroot/tensorflow && \
  exec env - \
    PATH=/usr/local/cuda/bin:/Users/anton/.pyenv/shims:/Users/anton/.rbenv/shims:/Users/anton/.scalaenv/shims:/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Users/anton/unix/bin:/usr/local/cuda/bin \
    TMPDIR=/var/folders/4r/mq7ht1z11t72w5b5b6014zwm0000gn/T/ \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/host/bin/tensorflow/python/gen_control_flow_ops_py_wrappers_cc @tensorflow/python/ops/hidden_ops.txt 1 > bazel-out/local_darwin-py3-opt/genfiles/tensorflow/python/ops/gen_control_flow_ops.py'): com.google.devtools.build.lib.shell.AbnormalTerminationException: Process terminated by signal 6.
dyld: Library not loaded: @rpath/libcudart.8.0.dylib
  Referenced from: /private/var/tmp/_bazel_anton/d3361bfa15c75c42ab541f3d83e8eba4/execroot/tensorflow/bazel-out/host/bin/tensorflow/python/gen_control_flow_ops_py_wrappers_cc
  Reason: image not found
/bin/bash: line 1: 45635 Abort trap: 6           bazel-out/host/bin/tensorflow/python/gen_control_flow_ops_py_wrappers_cc @tensorflow/python/ops/hidden_ops.txt 1 > bazel-out/local_darwin-py3-opt/genfiles/tensorflow/python/ops/gen_control_flow_ops.py
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 1683.463s, Critical Path: 1587.37s

I got a bit further but it seems that I need to modify the link options in more files.

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 36 (26 by maintainers)

Most upvoted comments

You should see something like this if you installed MacOS GPU version


export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/mac/gpu/tensorflow_gpu-0.12.1-py3-none-any.whl
pip install --upgrade $TF_BINARY_URL

export CUDA_HOME=/usr/local/cuda
export DYLD_LIBRARY_PATH="$CUDA_HOME/lib:$CUDA_HOME:$CUDA_HOME/extras/CUPTI/lib"
export LD_LIBRARY_PATH=$DYLD_LIBRARY_PATH

(tf12rc1) (tf12rc1)bash-3.2$ python 
Python 3.5.2 |Continuum Analytics, Inc.| (default, Jul  2 2016, 17:52:12) 
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.dylib locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.dylib locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.dylib locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.dylib locally
>>>

PS, you may need to manually symlink libcuda.dylib to libcuda.1.dylib in you /usr/local/cuda directory

+35

yaroslavvb on Jan 8, 2017

BTW, I just hit this issue when building today’s head on Mac and was able to get things building with @damienmg 's suggestion

export CUDA_HOME=/usr/local/cuda
export DYLD_LIBRARY_PATH=/usr/local/cuda/lib:/usr/local/cuda/extras/CUPTI/lib
export LD_LIBRARY_PATH=$DYLD_LIBRARY_PATH
export PATH=$DYLD_LIBRARY_PATH:$PATH
export flags="--config=cuda --config=opt"
bazel build $flags --action_env PATH --action_env LD_LIBRARY_PATH --action_env DYLD_LIBRARY_PATH //tensorflow/tools/pip_package:build_pip_package &> ~/temp/bazel_output.txt

+19

yaroslavvb on Jan 25, 2017

To summarize for new users who find this:

On MacOS you need to do this:

Disable SIP (Security Integrity Protection)
Set environment variables point to your CUDA installation

ie,

export CUDA_HOME=/usr/local/cuda
export DYLD_LIBRARY_PATH=/usr/local/cuda/lib:/usr/local/cuda/extras/CUPTI/lib
export LD_LIBRARY_PATH=$DYLD_LIBRARY_PATH
export PATH=$DYLD_LIBRARY_PATH:$PATH

Furthermore, if you are running TF through bazel as in bazel test, you need to make sure those env vars are propagated, by using --action_env flags

bazel test $flags --action_env PATH --action_env LD_LIBRARY_PATH --action_env DYLD_LIBRARY_PATH //tensorflow/tools/pip_package:build_pip_package

+15

yaroslavvb on Feb 13, 2017

This solution plus symlinking libcuda.1 (cd /usr/local/cuda/lib && sudo ln -s libcuda.dylib libcuda.1.dylib) also worked for me (OSX 10.12.3, TensorFlow-GPU 1.0.0, Python 3.6.)

Note that I had to remove a line from my .bash_profile which “activated pyenv” (removed eval "$(pyenv init -)"). The fix presented here appeared to have no effect until this was done.

FYI: disabling SIP does not appear to be necessary, at least not in all cases (ie. my SIP is enabled).

Socrates:~ bRad$ python -c "import tensorflow as tf"
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.8.0.dylib locally

Compute performance has more than doubled (138%). Thanks, everyone.

U007D on Feb 24, 2017

Have you tried to use --action_env PATH --action_env LD_LIBRARY_PATH --action_env DYLD_LIBRARYPATH flags? So those environment get shipped in the action?

damienmg on Jan 16, 2017

This fixed the problem for me on mac: install_name_tool -add_rpath /usr/local/cuda/lib /usr/local/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow.so

andreasgal on Mar 4, 2017

In my case, I am not allowed to disable SIP by the admin. From Apple’s page on Dynamic Libraries, they note that the following.

The standard locations for dynamic libraries are ~/lib, /usr/local/lib, and /usr/lib.

I didn’t want to clutter /usr/local/lib or /usr/lib, so I moved my files to ~/lib.

cp -r /usr/local/cuda/lib ~/lib

and set

export DYLD_LIBRARY_PATH="${HOME}/lib:${DYLD_LIBRARY_PATH}"

and it seems to work fine.

jason-riddle on Mar 2, 2017

I have not disabled SIP on my current OSX install. I can compile and run TensorFlow from the head and release branches successfully.

http://stackoverflow.com/questions/38710339/library-not-loaded-rpath-libcudart-7-5-dylib-tensorflow-error-on-mac/41073045#41073045

http://stackoverflow.com/questions/39865212/dyld-library-not-loaded-rpath-libcudart-8-0-dylib-while-building-tensorflow?rq=1

normanheckscher on Feb 14, 2017

As mentioned in the other huge GH issue for build troubleshooting on Macs, I’ve published another slightly more up-to-date tutorial here:

https://metakermit.com/2017/compiling-tensorflow-with-gpu-support-on-a-macbook-pro/

(TensorFlow 1.3.0, CUDA 8.0 and cuDNN 6.0)

metakermit on Sep 13, 2017

Clone tensorflow including submodules git clone --recurse-submodules https://github.com/tensorflow/tensorflow

Check out release 1.1 instead of using master branch git checkout remotes/origin/r1.1

Set path variables as @yaroslavvb mentioned: export CUDA_HOME=/usr/local/cuda export DYLD_LIBRARY_PATH=/usr/local/cuda/lib:/usr/local/cuda/extras/CUPTI/lib export LD_LIBRARY_PATH=$DYLD_LIBRARY_PATH export PATH=$DYLD_LIBRARY_PATH:$PATH

For OSX Sierra 10.12.4: Use XCode 7.2

I documented the whole installation here: https://gist.github.com/to-bee/6d19b7fa0d68ee97407591de1586da62

to-bee on Apr 23, 2017