tensorflow: Tensorflow (current master) build failure due to cuda/cudnn?

I would like to try out tf.contrib.framework.sort(), which available on current master only.

Tried to build master but failed due to cuda/cudnn library cannot found by bazel. However, I verified these library are exist at /usr/local/cuda/lib64.

ERROR: /srv/yg/researches/gits/tensorflow/tensorflow/contrib/lite/toco/BUILD:330:1: Linking of rule '//tensorflow/contrib/lite/toco:toco' failed (Exit 1)
/usr/bin/ld: warning: libcublas.so.8.0, needed by bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so, not found (try using -rpath or -rpath-link)

System Environment: Ubuntu 16.04, cuda8.0, cudnn6.0, gcc-5

Build Steps:

# checkout master
$ git clone https://github.com/tensorflow/tensorflow 
$ cd tensorflow
$ git checkout master

# python dependencies
$ sudo apt-get install python-numpy python-dev python-pip python-wheel
$ sudo apt-get install libcupti-dev 

# gcc-5
$ bazel build --config=opt --config=cuda --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" //tensorflow/tools/pip_package:build_pip_package

Check cuda/cudnn

$ ls -alhF /usr/local/cuda/lib64

total 1.4G
drwxr-xr-x  3 root root  4.0K Nov 28 18:21 ./
drwxr-xr-x 17 root root  4.0K Nov 28 16:38 ../
-rw-r--r--  1 root root   51M Nov 28 16:37 libcublas_device.a
lrwxrwxrwx  1 root root    16 Nov 28 16:37 libcublas.so -> libcublas.so.8.0*
lrwxrwxrwx  1 root root    19 Nov 28 16:37 libcublas.so.8.0 -> libcublas.so.8.0.61*
-rwxr-xr-x  1 root root   41M Nov 28 16:37 libcublas.so.8.0.61*
-rw-r--r--  1 root root   47M Nov 28 16:37 libcublas_static.a
-rw-r--r--  1 root root  543K Nov 28 16:37 libcudadevrt.a
lrwxrwxrwx  1 root root    16 Nov 28 16:37 libcudart.so -> libcudart.so.8.0*
lrwxrwxrwx  1 root root    19 Nov 28 16:37 libcudart.so.8.0 -> libcudart.so.8.0.61*
-rwxr-xr-x  1 root root  406K Nov 28 16:37 libcudart.so.8.0.61*
-rw-r--r--  1 root root  757K Nov 28 16:37 libcudart_static.a
lrwxrwxrwx  1 root root    13 Nov 28 18:21 libcudnn.so -> libcudnn.so.6*
lrwxrwxrwx  1 root root    18 Nov 28 18:21 libcudnn.so.6 -> libcudnn.so.6.0.21*
-rwxr-xr-x  1 root root  148M Nov 28 18:19 libcudnn.so.6.0.21*
-rw-r--r--  1 root root  138M Nov 28 18:19 libcudnn_static.a
lrwxrwxrwx  1 root root    15 Nov 28 16:37 libcufft.so -> libcufft.so.8.0*
lrwxrwxrwx  1 root root    18 Nov 28 16:37 libcufft.so.8.0 -> libcufft.so.8.0.61*
-rwxr-xr-x  1 root root  140M Nov 28 16:37 libcufft.so.8.0.61*
-rw-r--r--  1 root root  124M Nov 28 16:37 libcufft_static.a
lrwxrwxrwx  1 root root    16 Nov 28 16:37 libcufftw.so -> libcufftw.so.8.0*
lrwxrwxrwx  1 root root    19 Nov 28 16:37 libcufftw.so.8.0 -> libcufftw.so.8.0.61*
-rwxr-xr-x  1 root root  466K Nov 28 16:37 libcufftw.so.8.0.61*
-rw-r--r--  1 root root   42K Nov 28 16:37 libcufftw_static.a
lrwxrwxrwx  1 root root    17 Nov 28 16:37 libcuinj64.so -> libcuinj64.so.8.0*
lrwxrwxrwx  1 root root    20 Nov 28 16:37 libcuinj64.so.8.0 -> libcuinj64.so.8.0.61*
-rwxr-xr-x  1 root root  6.2M Nov 28 16:37 libcuinj64.so.8.0.61*
-rw-r--r--  1 root root  1.6M Nov 28 16:37 libculibos.a
lrwxrwxrwx  1 root root    16 Nov 28 16:37 libcurand.so -> libcurand.so.8.0*
lrwxrwxrwx  1 root root    19 Nov 28 16:37 libcurand.so.8.0 -> libcurand.so.8.0.61*
-rwxr-xr-x  1 root root   57M Nov 28 16:37 libcurand.so.8.0.61*
-rw-r--r--  1 root root   57M Nov 28 16:37 libcurand_static.a
lrwxrwxrwx  1 root root    18 Nov 28 16:37 libcusolver.so -> libcusolver.so.8.0*
lrwxrwxrwx  1 root root    21 Nov 28 16:37 libcusolver.so.8.0 -> libcusolver.so.8.0.61*
-rwxr-xr-x  1 root root   52M Nov 28 16:37 libcusolver.so.8.0.61*
-rw-r--r--  1 root root   22M Nov 28 16:37 libcusolver_static.a
lrwxrwxrwx  1 root root    18 Nov 28 16:37 libcusparse.so -> libcusparse.so.8.0*
lrwxrwxrwx  1 root root    21 Nov 28 16:37 libcusparse.so.8.0 -> libcusparse.so.8.0.61*
-rwxr-xr-x  1 root root   42M Nov 28 16:37 libcusparse.so.8.0.61*
-rw-r--r--  1 root root   50M Nov 28 16:37 libcusparse_static.a
lrwxrwxrwx  1 root root    14 Nov 28 16:37 libnppc.so -> libnppc.so.8.0*
lrwxrwxrwx  1 root root    17 Nov 28 16:37 libnppc.so.8.0 -> libnppc.so.8.0.61*
-rwxr-xr-x  1 root root  446K Nov 28 16:37 libnppc.so.8.0.61*
-rw-r--r--  1 root root   24K Nov 28 16:37 libnppc_static.a
lrwxrwxrwx  1 root root    16 Nov 28 16:37 libnppial.so -> libnppial.so.8.0*
lrwxrwxrwx  1 root root    19 Nov 28 16:37 libnppial.so.8.0 -> libnppial.so.8.0.61*
-rwxr-xr-x  1 root root  9.7M Nov 28 16:37 libnppial.so.8.0.61*
lrwxrwxrwx  1 root root    16 Nov 28 16:37 libnppicc.so -> libnppicc.so.8.0*
lrwxrwxrwx  1 root root    19 Nov 28 16:37 libnppicc.so.8.0 -> libnppicc.so.8.0.61*
-rwxr-xr-x  1 root root  3.7M Nov 28 16:37 libnppicc.so.8.0.61*
lrwxrwxrwx  1 root root    17 Nov 28 16:37 libnppicom.so -> libnppicom.so.8.0*
lrwxrwxrwx  1 root root    20 Nov 28 16:37 libnppicom.so.8.0 -> libnppicom.so.8.0.61*
-rwxr-xr-x  1 root root 1007K Nov 28 16:37 libnppicom.so.8.0.61*
lrwxrwxrwx  1 root root    17 Nov 28 16:37 libnppidei.so -> libnppidei.so.8.0*
lrwxrwxrwx  1 root root    20 Nov 28 16:37 libnppidei.so.8.0 -> libnppidei.so.8.0.61*
-rwxr-xr-x  1 root root  6.8M Nov 28 16:37 libnppidei.so.8.0.61*
lrwxrwxrwx  1 root root    15 Nov 28 16:37 libnppif.so -> libnppif.so.8.0*
lrwxrwxrwx  1 root root    18 Nov 28 16:37 libnppif.so.8.0 -> libnppif.so.8.0.61*
-rwxr-xr-x  1 root root   46M Nov 28 16:37 libnppif.so.8.0.61*
lrwxrwxrwx  1 root root    15 Nov 28 16:37 libnppig.so -> libnppig.so.8.0*
lrwxrwxrwx  1 root root    18 Nov 28 16:37 libnppig.so.8.0 -> libnppig.so.8.0.61*
-rwxr-xr-x  1 root root   21M Nov 28 16:37 libnppig.so.8.0.61*
lrwxrwxrwx  1 root root    15 Nov 28 16:37 libnppim.so -> libnppim.so.8.0*
lrwxrwxrwx  1 root root    18 Nov 28 16:37 libnppim.so.8.0 -> libnppim.so.8.0.61*
-rwxr-xr-x  1 root root  4.2M Nov 28 16:37 libnppim.so.8.0.61*
lrwxrwxrwx  1 root root    14 Nov 28 16:37 libnppi.so -> libnppi.so.8.0*
lrwxrwxrwx  1 root root    17 Nov 28 16:37 libnppi.so.8.0 -> libnppi.so.8.0.61*
-rwxr-xr-x  1 root root  104M Nov 28 16:37 libnppi.so.8.0.61*
-rw-r--r--  1 root root  131M Nov 28 16:37 libnppi_static.a
lrwxrwxrwx  1 root root    16 Nov 28 16:37 libnppist.so -> libnppist.so.8.0*
lrwxrwxrwx  1 root root    19 Nov 28 16:37 libnppist.so.8.0 -> libnppist.so.8.0.61*
-rwxr-xr-x  1 root root   14M Nov 28 16:37 libnppist.so.8.0.61*
lrwxrwxrwx  1 root root    16 Nov 28 16:37 libnppisu.so -> libnppisu.so.8.0*
lrwxrwxrwx  1 root root    19 Nov 28 16:37 libnppisu.so.8.0 -> libnppisu.so.8.0.61*
-rwxr-xr-x  1 root root  438K Nov 28 16:37 libnppisu.so.8.0.61*
lrwxrwxrwx  1 root root    16 Nov 28 16:37 libnppitc.so -> libnppitc.so.8.0*
lrwxrwxrwx  1 root root    19 Nov 28 16:37 libnppitc.so.8.0 -> libnppitc.so.8.0.61*
-rwxr-xr-x  1 root root  2.8M Nov 28 16:37 libnppitc.so.8.0.61*
lrwxrwxrwx  1 root root    14 Nov 28 16:37 libnpps.so -> libnpps.so.8.0*
lrwxrwxrwx  1 root root    17 Nov 28 16:37 libnpps.so.8.0 -> libnpps.so.8.0.61*
-rwxr-xr-x  1 root root  7.8M Nov 28 16:37 libnpps.so.8.0.61*
-rw-r--r--  1 root root   11M Nov 28 16:37 libnpps_static.a
lrwxrwxrwx  1 root root    16 Nov 28 16:37 libnvblas.so -> libnvblas.so.8.0*
lrwxrwxrwx  1 root root    19 Nov 28 16:37 libnvblas.so.8.0 -> libnvblas.so.8.0.61*
-rwxr-xr-x  1 root root  487K Nov 28 16:37 libnvblas.so.8.0.61*
lrwxrwxrwx  1 root root    17 Nov 28 16:37 libnvgraph.so -> libnvgraph.so.8.0*
lrwxrwxrwx  1 root root    20 Nov 28 16:37 libnvgraph.so.8.0 -> libnvgraph.so.8.0.61*
-rwxr-xr-x  1 root root  5.0M Nov 28 16:37 libnvgraph.so.8.0.61*
-rw-r--r--  1 root root  7.8M Nov 28 16:37 libnvgraph_static.a
lrwxrwxrwx  1 root root    24 Nov 28 16:37 libnvrtc-builtins.so -> libnvrtc-builtins.so.8.0*
lrwxrwxrwx  1 root root    27 Nov 28 16:37 libnvrtc-builtins.so.8.0 -> libnvrtc-builtins.so.8.0.61*
-rwxr-xr-x  1 root root  9.3M Nov 28 16:37 libnvrtc-builtins.so.8.0.61*
lrwxrwxrwx  1 root root    15 Nov 28 16:37 libnvrtc.so -> libnvrtc.so.8.0*
lrwxrwxrwx  1 root root    18 Nov 28 16:37 libnvrtc.so.8.0 -> libnvrtc.so.8.0.61*
-rwxr-xr-x  1 root root   18M Nov 28 16:37 libnvrtc.so.8.0.61*
lrwxrwxrwx  1 root root    18 Nov 28 16:37 libnvToolsExt.so -> libnvToolsExt.so.1*
lrwxrwxrwx  1 root root    22 Nov 28 16:37 libnvToolsExt.so.1 -> libnvToolsExt.so.1.0.0*
-rwxr-xr-x  1 root root   37K Nov 28 16:37 libnvToolsExt.so.1.0.0*
lrwxrwxrwx  1 root root    14 Nov 28 16:37 libOpenCL.so -> libOpenCL.so.1
lrwxrwxrwx  1 root root    16 Nov 28 16:37 libOpenCL.so.1 -> libOpenCL.so.1.0
lrwxrwxrwx  1 root root    18 Nov 28 16:37 libOpenCL.so.1.0 -> libOpenCL.so.1.0.0
-rw-r--r--  1 root root   26K Nov 28 16:37 libOpenCL.so.1.0.0
drwxr-xr-x  2 root root  4.0K Nov 28 16:37 stubs/

Verbose Failure:

$ bazel build --verbose_failures  --config=opt --config=cuda --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" //tensorflow/tools/pip_package:build_pip_package
WARNING: /srv/yg/researches/gits/tensorflow/tensorflow/core/BUILD:1806:1: in includes attribute of cc_library rule //tensorflow/core:framework_headers_lib: '../../external/nsync/public' resolves to 'external/nsync/public' not below the relative path of its package 'tensorflow/core'. This will be an error in the future. Since this rule was created by the macro 'cc_header_only_library', the error might have been caused by the macro implementation in /srv/yg/researches/gits/tensorflow/tensorflow/tensorflow.bzl:1100:30
WARNING: /srv/yg/researches/gits/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:exporter': No longer supported. Switch to SavedModel immediately.
WARNING: /srv/yg/researches/gits/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:gc': No longer supported. Switch to SavedModel immediately.
INFO: Analysed target //tensorflow/tools/pip_package:build_pip_package (2 packages loaded).
INFO: Found 1 target...
ERROR: /srv/yg/researches/gits/tensorflow/tensorflow/contrib/lite/toco/BUILD:330:1: Linking of rule '//tensorflow/contrib/lite/toco:toco' failed (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command 
  (cd /home/<usr>/.cache/bazel/_bazel_<usr>/eac5d1dca31d986f2b063251871502de/execroot/org_tensorflow && \
  exec env - \
    CUDA_TOOLKIT_PATH=/usr/local/cuda \
    CUDNN_INSTALL_PATH=/usr/local/cuda-8.0 \
    GCC_HOST_COMPILER_PATH=/usr/bin/gcc \
    PWD=/proc/self/cwd \
    PYTHON_BIN_PATH=/usr/bin/python \
    PYTHON_LIB_PATH=/home/<usr>/anaconda2/envs/ygtf/lib/python2.7/site-packages \
    TF_CUDA_CLANG=0 \
    TF_CUDA_COMPUTE_CAPABILITIES=3.7,3.7 \
    TF_CUDA_VERSION=8.0 \
    TF_CUDNN_VERSION=6 \
    TF_NEED_CUDA=1 \
    TF_NEED_OPENCL_SYCL=0 \
  external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -o bazel-out/local_linux-opt/bin/tensorflow/contrib/lite/toco/toco '-Wl,-rpath,$ORIGIN/../../../../_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow' '-Wl,-rpath,$ORIGIN/../../../../_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccudart___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib' -Lbazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow -Lbazel-out/local_linux-opt/bin/_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccudart___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib '-Wl,-rpath,$ORIGIN/,-rpath,$ORIGIN/..,-rpath,$ORIGIN/../..,-rpath,$ORIGIN/../../..' -pthread -Wl,-rpath,../local_config_cuda/cuda/lib64 -Wl,-rpath,../local_config_cuda/cuda/extras/CUPTI/lib64 -Wl,-no-as-needed -B/usr/bin/ -pie -Wl,-z,relro,-z,now -no-canonical-prefixes -pass-exit-codes '-Wl,--build-id=md5' '-Wl,--hash-style=gnu' -Wl,--gc-sections -Wl,@bazel-out/local_linux-opt/bin/tensorflow/contrib/lite/toco/toco-2.params)
/usr/bin/ld: warning: libcublas.so.8.0, needed by bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libcudnn.so.6, needed by bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libcufft.so.8.0, needed by bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libcurand.so.8.0, needed by bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so, not found (try using -rpath or -rpath-link)
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasStrmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZtpmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetLRNDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftPlan2d'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDrot_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDsyrk_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetConvolutionBackwardDataWorkspaceSize'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasChemv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetRNNLinLayerMatrixParams'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetConvolutionForwardAlgorithm'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCaxpy_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetConvolutionBackwardFilterWorkspaceSize'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZtrsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDgemmBatched'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasIzamax_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDspr2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDtbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftPlanMany'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasStbsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCsyr2k_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreateConvolutionDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyConvolutionDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetTensorNdDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftExecD2Z'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetFilterNdDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZhemm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSetStream_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDrotmg_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZtrmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSsbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSetPointerMode_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZher2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftMakePlan1d'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDswap_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZgeru_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZsymm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnConvolutionForward'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZher2k_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCtbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSswap_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSrotm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasChpr_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCgemmBatched'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetConvolutionNdForwardOutputDim'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCherk_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftMakePlan2d'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetRNNLinLayerBiasParams'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCgemm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZtbsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnLRNCrossChannelBackward'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCrotg_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDsymm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `curandGenerateUniform'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCtbsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnPoolingBackward'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSsymv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZaxpy_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasStpmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDspmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetConvolutionNdDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreateTensorDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyDropoutDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnRNNForwardInference'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnConvolutionBackwardData'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDdot_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZgemv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreateDropoutDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCtrmm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `curandGenerateUniformDouble'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftExecZ2Z'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDrotm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreatePoolingDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyFilterDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnBatchNormalizationForwardInference'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSnrm2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasIsamin_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCgerc_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetPoolingNdDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZtrsm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZgerc_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZswap_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSspr2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCher2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftDestroy'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyTensorDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetRNNParamsSize'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDaxpy_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasGetPointerMode_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreateLRNDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZscal_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZhemv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `curandCreateGenerator'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasChpmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyRNNDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCdotc_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetConvolutionForwardWorkspaceSize'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDscal_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZrotg_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasScopy_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZsyr2k_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDsymv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftSetAutoAllocation'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZcopy_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasChbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnAddTensor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZgemm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyPoolingDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasStpsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCtrsm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetFilterNdDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSrotmg_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnActivationForward'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDger_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetRNNWorkspaceSize'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetVersion'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZdotc_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasStbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCtpsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDsyr_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZher_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnPoolingForward'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCher2k_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftCreate'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSsyr2k_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnRNNBackwardWeights'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSrot_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSgemv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetRNNTrainingReserveSize'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDgbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `curandSetPseudoRandomGeneratorSeed'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSdot_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftExecC2R'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZgemmBatched'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCswap_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZhpmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasScnrm2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreateRNNDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDnrm2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSsyr_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDropoutGetStatesSize'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCsscal_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `curandDestroyGenerator'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnConvolutionBackwardBias'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnBatchNormalizationBackward'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasStrsm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasGemmEx'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDcopy_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetConvolutionBackwardFilterAlgorithm'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftExecC2C'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDestroy_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasIcamin_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZdscal_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZgbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCsrot_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZsyrk_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `curandGenerateNormalDouble'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSgemmEx'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDsbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSgemm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDtrsm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasStrsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreateFilterDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZhpr2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetActivationDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDzasum_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSrotg_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnTransformTensor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDgemv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftExecR2C'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDtbsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDtrmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDasum_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasStrmm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCgemv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnConvolutionBackwardFilter'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetStream'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnLRNCrossChannelForward'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDsyr2k_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyLRNDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCcopy_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCdotu_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnRNNBackwardData'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDtpsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetRNNDescriptor_v6'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasIzamin_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZhbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `curandSetStream'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreateActivationDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `curandSetGeneratorOffset'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnConvolutionBiasActivationForward'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSsyrk_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSaxpy_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSsymm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasIsamax_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDrotg_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetTensor4dDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZherk_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroy'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftPlan3d'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftSetWorkArea'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyActivationDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDspr_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftPlan1d'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDgemm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCtrsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasIdamin_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasChpr2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSscal_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSgemmBatched'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSsyr2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDznrm2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDtpmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCsymm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSspmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZtbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnBatchNormalizationForwardTraining'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCsyrk_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSger_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCgbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasIdamax_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftSetStream'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSasum_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftExecZ2D'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSgbmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZtrmm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCtpmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCreate_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZhpr_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftMakePlanMany'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCher_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDtrsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZdotu_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasScasum_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCscal_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasChemm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `curandGenerateNormal'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasIcamax_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetConvolutionBackwardDataAlgorithm'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDtrmm_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCtrmv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasCgeru_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnRNNForwardTraining'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZdrot_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetDropoutDescriptor'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftMakePlan3d'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasDsyr2_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasZtpsv_v2'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreate'
bazel-out/local_linux-opt/bin/_solib_local/_U_S_Stensorflow_Scontrib_Slite_Stoco_Ctoco___Utensorflow/libtensorflow_framework.so: undefined reference to `cublasSspr_v2'
collect2: error: ld returned 1 exit status
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 1.557s, Critical Path: 0.36s
FAILED: Build did NOT complete successfully

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 16 (2 by maintainers)

Most upvoted comments

Adding --action_env="LD_LIBRARY_PATH=${LD_LIBRARY_PATH}" to bazel build fixed my issue.

Solution was found here: https://stackoverflow.com/questions/47080760/tensorflow-fails-to-compile/47295278#47295278

I’ve solved it with:

sudo touch /etc/ld.so.conf.d/cuda.conf
sudo nano /etc/ld.so.conf.d/cuda.conf

After edit the file : /usr/local/cuda/lib64

Save the changes : sudo ldconfig

CF : https://stackoverflow.com/questions/13428910/how-to-set-the-environmental-variable-ld-library-path-in-linux.

had the same issue, solution from @frankgh works!

this is still broken. I had run into the same issue with building tf 1.8 from source on redhat 6.7 I am using cudnn 7 and cuda toolkit 9.2 . see my solutions below.

The solution from @frankchn resolved the initial issue, however, there is another one followed.

I checked and I am sure that libcudnn.so.7 is in LD_LIBRARY_PATH.

ERROR: /data/users/zhuangzr/tools/tensorflow/tensorflow/contrib/boosted_trees/BUILD:426:1: Linking of rule '//tensorflow/contrib/boosted_trees:gen_gen_training_ops_py_py_wrappers_cc' failed (Exit 1)
/usr/bin/ld: warning: libcudnn.so.7, needed by bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so, not found (try using -rpath or -rpath-link)
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetRNNParamsSize'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnActivationForward'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetTensor4dDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnRNNBackwardWeights'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetConvolutionNdDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreateFilterDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyFilterDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroy'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyDropoutDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetDropoutDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnAddTensor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnBatchNormalizationForwardInference'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetFilterNdDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreateTensorDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetLRNDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetRNNLinLayerBiasParams'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetRNNTrainingReserveSize'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyTensorDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnRNNForwardTraining'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyRNNDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetConvolutionNdForwardOutputDim'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnLRNCrossChannelForward'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyPoolingDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetRNNWorkspaceSize'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetRNNDescriptor_v6'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetConvolutionBackwardDataAlgorithm'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnBatchNormalizationBackward'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetRNNLinLayerMatrixParams'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnConvolutionBackwardBias'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreate'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreateConvolutionDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetFilterNdDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetVersion'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetConvolutionBackwardFilterAlgorithm'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetStream'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetConvolutionBackwardDataWorkspaceSize'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetPoolingNdDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnPoolingBackward'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetConvolutionForwardAlgorithm'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetConvolutionForwardWorkspaceSize'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnLRNCrossChannelBackward'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyActivationDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnGetConvolutionBackwardFilterWorkspaceSize'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnBatchNormalizationForwardTraining'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetActivationDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnRNNForwardInference'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnConvolutionForward'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnConvolutionBiasActivationForward'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreateDropoutDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreateLRNDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnTransformTensor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnConvolutionBackwardFilter'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreatePoolingDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnRNNBackwardData'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyLRNDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDropoutGetStatesSize'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnSetTensorNdDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnConvolutionBackwardData'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreateActivationDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnDestroyConvolutionDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnCreateRNNDescriptor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sboosted_Utrees_Cgen_Ugen_Utraining_Uops_Upy_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cudnnPoolingForward'
collect2: error: ld returned 1 exit status
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 223.731s, Critical Path: 171.49s
FAILED: Build did NOT complete successfully