tensorflow: Tensorflow (current master) build failure due to cuda/cudnn?
I would like to try out tf.contrib.framework.sort(), which available on current master only.
Tried to build master but failed due to cuda/cudnn library cannot found by bazel. However, I verified these library are exist at /usr/local/cuda/lib64.
ERROR: /srv/yg/researches/gits/tensorflow/tensorflow/contrib/lite/toco/BUILD:330:1: Linking of rule '//tensorflow/contrib/lite/toco:toco' failed (Exit 1)
/usr/bin/ld: warning: libcublas.so.8.0, needed by bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so, not found (try using -rpath or -rpath-link)
System Environment: Ubuntu 16.04, cuda8.0, cudnn6.0, gcc-5
Build Steps:
# checkout master
$ git clone https://github.com/tensorflow/tensorflow
$ cd tensorflow
$ git checkout master
# python dependencies
$ sudo apt-get install python-numpy python-dev python-pip python-wheel
$ sudo apt-get install libcupti-dev
# gcc-5
$ bazel build --config=opt --config=cuda --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" //tensorflow/tools/pip_package:build_pip_package
Check cuda/cudnn
$ ls -alhF /usr/local/cuda/lib64
total 1.4G
drwxr-xr-x 3 root root 4.0K Nov 28 18:21 ./
drwxr-xr-x 17 root root 4.0K Nov 28 16:38 ../
-rw-r--r-- 1 root root 51M Nov 28 16:37 libcublas_device.a
lrwxrwxrwx 1 root root 16 Nov 28 16:37 libcublas.so -> libcublas.so.8.0*
lrwxrwxrwx 1 root root 19 Nov 28 16:37 libcublas.so.8.0 -> libcublas.so.8.0.61*
-rwxr-xr-x 1 root root 41M Nov 28 16:37 libcublas.so.8.0.61*
-rw-r--r-- 1 root root 47M Nov 28 16:37 libcublas_static.a
-rw-r--r-- 1 root root 543K Nov 28 16:37 libcudadevrt.a
lrwxrwxrwx 1 root root 16 Nov 28 16:37 libcudart.so -> libcudart.so.8.0*
lrwxrwxrwx 1 root root 19 Nov 28 16:37 libcudart.so.8.0 -> libcudart.so.8.0.61*
-rwxr-xr-x 1 root root 406K Nov 28 16:37 libcudart.so.8.0.61*
-rw-r--r-- 1 root root 757K Nov 28 16:37 libcudart_static.a
lrwxrwxrwx 1 root root 13 Nov 28 18:21 libcudnn.so -> libcudnn.so.6*
lrwxrwxrwx 1 root root 18 Nov 28 18:21 libcudnn.so.6 -> libcudnn.so.6.0.21*
-rwxr-xr-x 1 root root 148M Nov 28 18:19 libcudnn.so.6.0.21*
-rw-r--r-- 1 root root 138M Nov 28 18:19 libcudnn_static.a
lrwxrwxrwx 1 root root 15 Nov 28 16:37 libcufft.so -> libcufft.so.8.0*
lrwxrwxrwx 1 root root 18 Nov 28 16:37 libcufft.so.8.0 -> libcufft.so.8.0.61*
-rwxr-xr-x 1 root root 140M Nov 28 16:37 libcufft.so.8.0.61*
-rw-r--r-- 1 root root 124M Nov 28 16:37 libcufft_static.a
lrwxrwxrwx 1 root root 16 Nov 28 16:37 libcufftw.so -> libcufftw.so.8.0*
lrwxrwxrwx 1 root root 19 Nov 28 16:37 libcufftw.so.8.0 -> libcufftw.so.8.0.61*
-rwxr-xr-x 1 root root 466K Nov 28 16:37 libcufftw.so.8.0.61*
-rw-r--r-- 1 root root 42K Nov 28 16:37 libcufftw_static.a
lrwxrwxrwx 1 root root 17 Nov 28 16:37 libcuinj64.so -> libcuinj64.so.8.0*
lrwxrwxrwx 1 root root 20 Nov 28 16:37 libcuinj64.so.8.0 -> libcuinj64.so.8.0.61*
-rwxr-xr-x 1 root root 6.2M Nov 28 16:37 libcuinj64.so.8.0.61*
-rw-r--r-- 1 root root 1.6M Nov 28 16:37 libculibos.a
lrwxrwxrwx 1 root root 16 Nov 28 16:37 libcurand.so -> libcurand.so.8.0*
lrwxrwxrwx 1 root root 19 Nov 28 16:37 libcurand.so.8.0 -> libcurand.so.8.0.61*
-rwxr-xr-x 1 root root 57M Nov 28 16:37 libcurand.so.8.0.61*
-rw-r--r-- 1 root root 57M Nov 28 16:37 libcurand_static.a
lrwxrwxrwx 1 root root 18 Nov 28 16:37 libcusolver.so -> libcusolver.so.8.0*
lrwxrwxrwx 1 root root 21 Nov 28 16:37 libcusolver.so.8.0 -> libcusolver.so.8.0.61*
-rwxr-xr-x 1 root root 52M Nov 28 16:37 libcusolver.so.8.0.61*
-rw-r--r-- 1 root root 22M Nov 28 16:37 libcusolver_static.a
lrwxrwxrwx 1 root root 18 Nov 28 16:37 libcusparse.so -> libcusparse.so.8.0*
lrwxrwxrwx 1 root root 21 Nov 28 16:37 libcusparse.so.8.0 -> libcusparse.so.8.0.61*
-rwxr-xr-x 1 root root 42M Nov 28 16:37 libcusparse.so.8.0.61*
-rw-r--r-- 1 root root 50M Nov 28 16:37 libcusparse_static.a
lrwxrwxrwx 1 root root 14 Nov 28 16:37 libnppc.so -> libnppc.so.8.0*
lrwxrwxrwx 1 root root 17 Nov 28 16:37 libnppc.so.8.0 -> libnppc.so.8.0.61*
-rwxr-xr-x 1 root root 446K Nov 28 16:37 libnppc.so.8.0.61*
-rw-r--r-- 1 root root 24K Nov 28 16:37 libnppc_static.a
lrwxrwxrwx 1 root root 16 Nov 28 16:37 libnppial.so -> libnppial.so.8.0*
lrwxrwxrwx 1 root root 19 Nov 28 16:37 libnppial.so.8.0 -> libnppial.so.8.0.61*
-rwxr-xr-x 1 root root 9.7M Nov 28 16:37 libnppial.so.8.0.61*
lrwxrwxrwx 1 root root 16 Nov 28 16:37 libnppicc.so -> libnppicc.so.8.0*
lrwxrwxrwx 1 root root 19 Nov 28 16:37 libnppicc.so.8.0 -> libnppicc.so.8.0.61*
-rwxr-xr-x 1 root root 3.7M Nov 28 16:37 libnppicc.so.8.0.61*
lrwxrwxrwx 1 root root 17 Nov 28 16:37 libnppicom.so -> libnppicom.so.8.0*
lrwxrwxrwx 1 root root 20 Nov 28 16:37 libnppicom.so.8.0 -> libnppicom.so.8.0.61*
-rwxr-xr-x 1 root root 1007K Nov 28 16:37 libnppicom.so.8.0.61*
lrwxrwxrwx 1 root root 17 Nov 28 16:37 libnppidei.so -> libnppidei.so.8.0*
lrwxrwxrwx 1 root root 20 Nov 28 16:37 libnppidei.so.8.0 -> libnppidei.so.8.0.61*
-rwxr-xr-x 1 root root 6.8M Nov 28 16:37 libnppidei.so.8.0.61*
lrwxrwxrwx 1 root root 15 Nov 28 16:37 libnppif.so -> libnppif.so.8.0*
lrwxrwxrwx 1 root root 18 Nov 28 16:37 libnppif.so.8.0 -> libnppif.so.8.0.61*
-rwxr-xr-x 1 root root 46M Nov 28 16:37 libnppif.so.8.0.61*
lrwxrwxrwx 1 root root 15 Nov 28 16:37 libnppig.so -> libnppig.so.8.0*
lrwxrwxrwx 1 root root 18 Nov 28 16:37 libnppig.so.8.0 -> libnppig.so.8.0.61*
-rwxr-xr-x 1 root root 21M Nov 28 16:37 libnppig.so.8.0.61*
lrwxrwxrwx 1 root root 15 Nov 28 16:37 libnppim.so -> libnppim.so.8.0*
lrwxrwxrwx 1 root root 18 Nov 28 16:37 libnppim.so.8.0 -> libnppim.so.8.0.61*
-rwxr-xr-x 1 root root 4.2M Nov 28 16:37 libnppim.so.8.0.61*
lrwxrwxrwx 1 root root 14 Nov 28 16:37 libnppi.so -> libnppi.so.8.0*
lrwxrwxrwx 1 root root 17 Nov 28 16:37 libnppi.so.8.0 -> libnppi.so.8.0.61*
-rwxr-xr-x 1 root root 104M Nov 28 16:37 libnppi.so.8.0.61*
-rw-r--r-- 1 root root 131M Nov 28 16:37 libnppi_static.a
lrwxrwxrwx 1 root root 16 Nov 28 16:37 libnppist.so -> libnppist.so.8.0*
lrwxrwxrwx 1 root root 19 Nov 28 16:37 libnppist.so.8.0 -> libnppist.so.8.0.61*
-rwxr-xr-x 1 root root 14M Nov 28 16:37 libnppist.so.8.0.61*
lrwxrwxrwx 1 root root 16 Nov 28 16:37 libnppisu.so -> libnppisu.so.8.0*
lrwxrwxrwx 1 root root 19 Nov 28 16:37 libnppisu.so.8.0 -> libnppisu.so.8.0.61*
-rwxr-xr-x 1 root root 438K Nov 28 16:37 libnppisu.so.8.0.61*
lrwxrwxrwx 1 root root 16 Nov 28 16:37 libnppitc.so -> libnppitc.so.8.0*
lrwxrwxrwx 1 root root 19 Nov 28 16:37 libnppitc.so.8.0 -> libnppitc.so.8.0.61*
-rwxr-xr-x 1 root root 2.8M Nov 28 16:37 libnppitc.so.8.0.61*
lrwxrwxrwx 1 root root 14 Nov 28 16:37 libnpps.so -> libnpps.so.8.0*
lrwxrwxrwx 1 root root 17 Nov 28 16:37 libnpps.so.8.0 -> libnpps.so.8.0.61*
-rwxr-xr-x 1 root root 7.8M Nov 28 16:37 libnpps.so.8.0.61*
-rw-r--r-- 1 root root 11M Nov 28 16:37 libnpps_static.a
lrwxrwxrwx 1 root root 16 Nov 28 16:37 libnvblas.so -> libnvblas.so.8.0*
lrwxrwxrwx 1 root root 19 Nov 28 16:37 libnvblas.so.8.0 -> libnvblas.so.8.0.61*
-rwxr-xr-x 1 root root 487K Nov 28 16:37 libnvblas.so.8.0.61*
lrwxrwxrwx 1 root root 17 Nov 28 16:37 libnvgraph.so -> libnvgraph.so.8.0*
lrwxrwxrwx 1 root root 20 Nov 28 16:37 libnvgraph.so.8.0 -> libnvgraph.so.8.0.61*
-rwxr-xr-x 1 root root 5.0M Nov 28 16:37 libnvgraph.so.8.0.61*
-rw-r--r-- 1 root root 7.8M Nov 28 16:37 libnvgraph_static.a
lrwxrwxrwx 1 root root 24 Nov 28 16:37 libnvrtc-builtins.so -> libnvrtc-builtins.so.8.0*
lrwxrwxrwx 1 root root 27 Nov 28 16:37 libnvrtc-builtins.so.8.0 -> libnvrtc-builtins.so.8.0.61*
-rwxr-xr-x 1 root root 9.3M Nov 28 16:37 libnvrtc-builtins.so.8.0.61*
lrwxrwxrwx 1 root root 15 Nov 28 16:37 libnvrtc.so -> libnvrtc.so.8.0*
lrwxrwxrwx 1 root root 18 Nov 28 16:37 libnvrtc.so.8.0 -> libnvrtc.so.8.0.61*
-rwxr-xr-x 1 root root 18M Nov 28 16:37 libnvrtc.so.8.0.61*
lrwxrwxrwx 1 root root 18 Nov 28 16:37 libnvToolsExt.so -> libnvToolsExt.so.1*
lrwxrwxrwx 1 root root 22 Nov 28 16:37 libnvToolsExt.so.1 -> libnvToolsExt.so.1.0.0*
-rwxr-xr-x 1 root root 37K Nov 28 16:37 libnvToolsExt.so.1.0.0*
lrwxrwxrwx 1 root root 14 Nov 28 16:37 libOpenCL.so -> libOpenCL.so.1
lrwxrwxrwx 1 root root 16 Nov 28 16:37 libOpenCL.so.1 -> libOpenCL.so.1.0
lrwxrwxrwx 1 root root 18 Nov 28 16:37 libOpenCL.so.1.0 -> libOpenCL.so.1.0.0
-rw-r--r-- 1 root root 26K Nov 28 16:37 libOpenCL.so.1.0.0
drwxr-xr-x 2 root root 4.0K Nov 28 16:37 stubs/
Verbose Failure:
$ bazel build --verbose_failures --config=opt --config=cuda --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" //tensorflow/tools/pip_package:build_pip_package
WARNING: /srv/yg/researches/gits/tensorflow/tensorflow/core/BUILD:1806:1: in includes attribute of cc_library rule //tensorflow/core:framework_headers_lib: '../../external/nsync/public' resolves to 'external/nsync/public' not below the relative path of its package 'tensorflow/core'. This will be an error in the future. Since this rule was created by the macro 'cc_header_only_library', the error might have been caused by the macro implementation in /srv/yg/researches/gits/tensorflow/tensorflow/tensorflow.bzl:1100:30
WARNING: /srv/yg/researches/gits/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:exporter': No longer supported. Switch to SavedModel immediately.
WARNING: /srv/yg/researches/gits/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:gc': No longer supported. Switch to SavedModel immediately.
INFO: Analysed target //tensorflow/tools/pip_package:build_pip_package (2 packages loaded).
INFO: Found 1 target...
ERROR: /srv/yg/researches/gits/tensorflow/tensorflow/contrib/lite/toco/BUILD:330:1: Linking of rule '//tensorflow/contrib/lite/toco:toco' failed (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command
(cd /home/<usr>/.cache/bazel/_bazel_<usr>/eac5d1dca31d986f2b063251871502de/execroot/org_tensorflow && \
exec env - \
CUDA_TOOLKIT_PATH=/usr/local/cuda \
CUDNN_INSTALL_PATH=/usr/local/cuda-8.0 \
GCC_HOST_COMPILER_PATH=/usr/bin/gcc \
PWD=/proc/self/cwd \
PYTHON_BIN_PATH=/usr/bin/python \
PYTHON_LIB_PATH=/home/<usr>/anaconda2/envs/ygtf/lib/python2.7/site-packages \
TF_CUDA_CLANG=0 \
TF_CUDA_COMPUTE_CAPABILITIES=3.7,3.7 \
TF_CUDA_VERSION=8.0 \
TF_CUDNN_VERSION=6 \
TF_NEED_CUDA=1 \
TF_NEED_OPENCL_SYCL=0 \
external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -o bazel-out/local_linux-opt/bin/tensorflow/contrib/lite/toco/toco '-Wl,-rpath,$ORIGIN/../../../../_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow' '-Wl,-rpath,$ORIGIN/../../../../_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccudart___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib' -Lbazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow -Lbazel-out/local_linux-opt/bin/_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccudart___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib '-Wl,-rpath,$ORIGIN/,-rpath,$ORIGIN/..,-rpath,$ORIGIN/../..,-rpath,$ORIGIN/../../..' -pthread -Wl,-rpath,../local_config_cuda/cuda/lib64 -Wl,-rpath,../local_config_cuda/cuda/extras/CUPTI/lib64 -Wl,-no-as-needed -B/usr/bin/ -pie -Wl,-z,relro,-z,now -no-canonical-prefixes -pass-exit-codes '-Wl,--build-id=md5' '-Wl,--hash-style=gnu' -Wl,--gc-sections -Wl,@bazel-out/local_linux-opt/bin/tensorflow/contrib/lite/toco/toco-2.params)
/usr/bin/ld: warning: libcublas.so.8.0, needed by bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libcudnn.so.6, needed by bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libcufft.so.8.0, needed by bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libcurand.so.8.0, needed by bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so, not found (try using -rpath or -rpath-link)
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasStrmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZtpmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetLRNDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftPlan2d'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDrot_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDsyrk_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetConvolutionBackwardDataWorkspaceSize'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasChemv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetRNNLinLayerMatrixParams'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetConvolutionForwardAlgorithm'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCaxpy_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetConvolutionBackwardFilterWorkspaceSize'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZtrsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDgemmBatched'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasIzamax_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDspr2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDtbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftPlanMany'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasStbsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCsyr2k_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreateConvolutionDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyConvolutionDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetTensorNdDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftExecD2Z'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetFilterNdDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZhemm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSetStream_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDrotmg_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZtrmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSsbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSetPointerMode_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZher2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftMakePlan1d'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDswap_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZgeru_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZsymm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnConvolutionForward'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZher2k_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCtbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSswap_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSrotm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasChpr_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCgemmBatched'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetConvolutionNdForwardOutputDim'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCherk_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftMakePlan2d'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetRNNLinLayerBiasParams'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCgemm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZtbsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnLRNCrossChannelBackward'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCrotg_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDsymm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `curandGenerateUniform'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCtbsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnPoolingBackward'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSsymv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZaxpy_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasStpmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDspmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetConvolutionNdDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreateTensorDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyDropoutDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnRNNForwardInference'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnConvolutionBackwardData'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDdot_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZgemv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreateDropoutDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCtrmm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `curandGenerateUniformDouble'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftExecZ2Z'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDrotm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreatePoolingDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyFilterDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnBatchNormalizationForwardInference'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSnrm2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasIsamin_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCgerc_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetPoolingNdDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZtrsm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZgerc_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZswap_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSspr2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCher2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftDestroy'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyTensorDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetRNNParamsSize'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDaxpy_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasGetPointerMode_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreateLRNDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZscal_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZhemv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `curandCreateGenerator'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasChpmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyRNNDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCdotc_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetConvolutionForwardWorkspaceSize'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDscal_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZrotg_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasScopy_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZsyr2k_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDsymv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftSetAutoAllocation'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZcopy_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasChbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnAddTensor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZgemm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyPoolingDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasStpsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCtrsm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetFilterNdDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSrotmg_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnActivationForward'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDger_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetRNNWorkspaceSize'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetVersion'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZdotc_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasStbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCtpsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDsyr_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZher_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnPoolingForward'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCher2k_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftCreate'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSsyr2k_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnRNNBackwardWeights'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSrot_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSgemv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetRNNTrainingReserveSize'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDgbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `curandSetPseudoRandomGeneratorSeed'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSdot_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftExecC2R'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZgemmBatched'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCswap_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZhpmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasScnrm2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreateRNNDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDnrm2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSsyr_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDropoutGetStatesSize'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCsscal_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `curandDestroyGenerator'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnConvolutionBackwardBias'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnBatchNormalizationBackward'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasStrsm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasGemmEx'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDcopy_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetConvolutionBackwardFilterAlgorithm'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftExecC2C'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDestroy_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasIcamin_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZdscal_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZgbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCsrot_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZsyrk_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `curandGenerateNormalDouble'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSgemmEx'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDsbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSgemm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDtrsm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasStrsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreateFilterDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZhpr2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetActivationDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDzasum_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSrotg_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnTransformTensor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDgemv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftExecR2C'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDtbsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDtrmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDasum_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasStrmm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCgemv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnConvolutionBackwardFilter'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetStream'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnLRNCrossChannelForward'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDsyr2k_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyLRNDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCcopy_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCdotu_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnRNNBackwardData'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDtpsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetRNNDescriptor_v6'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasIzamin_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZhbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `curandSetStream'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreateActivationDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `curandSetGeneratorOffset'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnConvolutionBiasActivationForward'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSsyrk_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSaxpy_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSsymm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasIsamax_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDrotg_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetTensor4dDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZherk_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroy'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftPlan3d'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftSetWorkArea'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyActivationDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDspr_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftPlan1d'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDgemm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCtrsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasIdamin_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasChpr2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSscal_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSgemmBatched'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSsyr2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDznrm2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDtpmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCsymm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSspmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZtbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnBatchNormalizationForwardTraining'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCsyrk_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSger_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCgbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasIdamax_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftSetStream'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSasum_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftExecZ2D'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSgbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZtrmm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCtpmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCreate_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZhpr_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftMakePlanMany'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCher_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDtrsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZdotu_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasScasum_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCscal_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasChemm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `curandGenerateNormal'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasIcamax_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetConvolutionBackwardDataAlgorithm'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDtrmm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCtrmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCgeru_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnRNNForwardTraining'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZdrot_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetDropoutDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftMakePlan3d'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDsyr2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZtpsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreate'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSspr_v2'
collect2: error: ld returned 1 exit status
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 1.557s, Critical Path: 0.36s
FAILED: Build did NOT complete successfully
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 16 (2 by maintainers)
Adding
--action_env="LD_LIBRARY_PATH=${LD_LIBRARY_PATH}"tobazel buildfixed my issue.Solution was found here: https://stackoverflow.com/questions/47080760/tensorflow-fails-to-compile/47295278#47295278
I’ve solved it with:
After edit the file :
/usr/local/cuda/lib64Save the changes :
sudo ldconfigCF : https://stackoverflow.com/questions/13428910/how-to-set-the-environmental-variable-ld-library-path-in-linux.
had the same issue, solution from @frankgh works!
this is still broken. I had run into the same issue with building tf 1.8 from source on redhat 6.7 I am using cudnn 7 and cuda toolkit 9.2 . see my solutions below.
The solution from @frankchn resolved the initial issue, however, there is another one followed.
I checked and I am sure that libcudnn.so.7 is in LD_LIBRARY_PATH.