tensorflow: TF 1.7.0-rc1 unable to build with TensorRT support

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
  • TensorFlow installed from (source or binary): source
  • TensorFlow version (use command below): 1.7.0-rc1
  • Python version: Python 3.5
  • Bazel version (if compiling from source): 0.11.1
  • GCC/Compiler version (if compiling from source): 5.4.0-6ubuntu1~16.04.9
  • CUDA/cuDNN version: 9.0 / 7.0.5
  • GPU model and memory: GTX 1070, 8Gb
  • Exact command to reproduce: bazel build -c opt --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.1 --copt=-msse4.2 --config=cuda //tensorflow/tools/pip_package:build_pip_package --verbose_failures

Describe the problem

Try to build TF with TensorRT support using the command above and following this instruction, i.e. there is no TensorRT installed from repo, only downloaded binaries for 14.04 + echo "<install_dir>/TensorRT-3.0.4/lib" | sudo tee /etc/ld.so.conf.d/tensorrt304.conf && sudo ldconfig, as suggested. The libcudnn.so* available in TensorRT lib dir, /usr/local/lib and /usr/lib/x86_64-linux-gnu. LD_LIBRARY_PATH contains all relevant paths. While configure, I pointed to TensorRT binaries dir when it asked. /usr/local/cuda points to 9.0 version and this path was specified while configure.

Source code / logs

WARNING: /home/alexandr/.cache/bazel/_bazel_alexandr/6c93df3ac6ca06598f74e718588ae6cc/external/protobuf_archive/WORKSPACE:1: Workspace name in /home/alexandr/.cache/bazel/_bazel_alexandr/6c93df3ac6ca06598f74e718588ae6cc/external/protobuf_archive/WORKSPACE (@com_google_protobuf) does not match the name given in the repository's definition (@protobuf_archive); this will cause a build error in future versions
WARNING: /home/alexandr/distr/tensorflow_tensorrt/tensorflow/core/BUILD:1955:1: in includes attribute of cc_library rule //tensorflow/core:framework_headers_lib: '../../external/nsync/public' resolves to 'external/nsync/public' not below the relative path of its package 'tensorflow/core'. This will be an error in the future. Since this rule was created by the macro 'cc_header_only_library', the error might have been caused by the macro implementation in /home/alexandr/distr/tensorflow_tensorrt/tensorflow/tensorflow.bzl:1179:30
WARNING: /home/alexandr/distr/tensorflow_tensorrt/tensorflow/core/BUILD:1955:1: in includes attribute of cc_library rule //tensorflow/core:framework_headers_lib: '../../external/nsync/public' resolves to 'external/nsync/public' not below the relative path of its package 'tensorflow/core'. This will be an error in the future. Since this rule was created by the macro 'cc_header_only_library', the error might have been caused by the macro implementation in /home/alexandr/distr/tensorflow_tensorrt/tensorflow/tensorflow.bzl:1179:30
WARNING: /home/alexandr/.cache/bazel/_bazel_alexandr/6c93df3ac6ca06598f74e718588ae6cc/external/grpc/WORKSPACE:1: Workspace name in /home/alexandr/.cache/bazel/_bazel_alexandr/6c93df3ac6ca06598f74e718588ae6cc/external/grpc/WORKSPACE (@com_github_grpc_grpc) does not match the name given in the repository's definition (@grpc); this will cause a build error in future versions
WARNING: /home/alexandr/distr/tensorflow_tensorrt/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:exporter': No longer supported. Switch to SavedModel immediately.
WARNING: /home/alexandr/distr/tensorflow_tensorrt/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:gc': No longer supported. Switch to SavedModel immediately.
INFO: Analysed target //tensorflow/tools/pip_package:build_pip_package (0 packages loaded).
INFO: Found 1 target...
ERROR: /home/alexandr/distr/tensorflow_tensorrt/tensorflow/contrib/tensorrt/BUILD:122:1: Linking of rule '//tensorflow/contrib/tensorrt:gen_trt_engine_op_py_wrappers_cc' failed (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command 
  (cd /home/alexandr/.cache/bazel/_bazel_alexandr/6c93df3ac6ca06598f74e718588ae6cc/execroot/org_tensorflow && \
  exec env - \
    LD_LIBRARY_PATH=:/usr/local/cuda/lib64 \
    PATH=/home/alexandr/bin:/home/alexandr/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/cuda/bin \
    PWD=/proc/self/cwd \
  external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -o bazel-out/host/bin/tensorflow/contrib/tensorrt/gen_trt_engine_op_py_wrappers_cc '-Wl,-rpath,$ORIGIN/../../../_solib_local/_U_S_Stensorflow_Scontrib_Stensorrt_Cgen_Utrt_Uengine_Uop_Upy_Uwrappers_Ucc___Utensorflow' '-Wl,-rpath,$ORIGIN/../../../_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib' '-Wl,-rpath,$ORIGIN/../../../_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccublas___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib' '-Wl,-rpath,$ORIGIN/../../../_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccudnn___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib' '-Wl,-rpath,$ORIGIN/../../../_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccufft___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib' '-Wl,-rpath,$ORIGIN/../../../_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccurand___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib' '-Wl,-rpath,$ORIGIN/../../../_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccudart___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib' -Lbazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Stensorrt_Cgen_Utrt_Uengine_Uop_Upy_Uwrappers_Ucc___Utensorflow -Lbazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib -Lbazel-out/host/bin/_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccublas___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib -Lbazel-out/host/bin/_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccudnn___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib -Lbazel-out/host/bin/_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccufft___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib -Lbazel-out/host/bin/_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccurand___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib -Lbazel-out/host/bin/_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccudart___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib '-Wl,-rpath,$ORIGIN/,-rpath,$ORIGIN/..,-rpath,$ORIGIN/../..' -pthread -Wl,-rpath,../local_config_cuda/cuda/lib64 -Wl,-rpath,../local_config_cuda/cuda/extras/CUPTI/lib64 -Wl,-no-as-needed -B/usr/bin/ -pie -Wl,-z,relro,-z,now -no-canonical-prefixes -pass-exit-codes '-Wl,--build-id=md5' '-Wl,--hash-style=gnu' -Wl,--gc-sections -Wl,-S -Wl,@bazel-out/host/bin/tensorflow/contrib/tensorrt/gen_trt_engine_op_py_wrappers_cc-2.params)
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnCreate@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnCreatePoolingDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnCreateConvolutionDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDestroyConvolutionDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnGetReductionWorkspaceSize@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetPooling2dDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDestroyFilterDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetReduceTensorDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnLRNCrossChannelForward@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnCreateRNNDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDestroyRNNDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnConvolutionBackwardData@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnCreateFilterDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetTensor4dDescriptorEx@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDestroyLRNDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDestroy@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDropoutGetStatesSize@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnGetConvolutionForwardWorkspaceSize@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetActivationDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnGetRNNWorkspaceSize@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetConvolutionGroupCount@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetTensorNdDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnAddTensor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDestroyActivationDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnGetConvolutionBackwardDataAlgorithmMaxCount@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnGetFilterNdDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetLRNDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDestroyTensorDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDestroyDropoutDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnCreateTensorDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetStream@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDestroyOpTensorDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnCreateOpTensorDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnCreateDropoutDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetFilter4dDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDestroyPoolingDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnCreateActivationDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnRNNForwardInference@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnActivationForward@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnGetRNNLinLayerMatrixParams@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnGetConvolutionBackwardDataWorkspaceSize@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnReduceTensor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetRNNDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetDropoutDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSoftmaxForward@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnConvolutionForward@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnGetRNNLinLayerBiasParams@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetConvolution2dDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetFilterNdDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnGetRNNParamsSize@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetTensor4dDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetOpTensorDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDestroyReduceTensorDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnFindConvolutionBackwardDataAlgorithmEx@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnPoolingForward@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnCreateLRNDescriptor@libcudnn.so.7'
collect2: error: ld returned 1 exit status
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 0.642s, Critical Path: 0.22s
FAILED: Build did NOT complete successfully

The extended system info:

== cat /etc/issue ===============================================
Linux power-linux 4.12.0-041200-generic #201707022031 SMP Mon Jul 3 00:32:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
VERSION="16.04.4 LTS (Xenial Xerus)"
VERSION_ID="16.04"
VERSION_CODENAME=xenial

== are we in docker =============================================
No

== compiler =====================================================
c++ (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


== uname -a =====================================================
Linux power-linux 4.12.0-041200-generic #201707022031 SMP Mon Jul 3 00:32:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

== check pips ===================================================
msgpack-numpy (0.4.1)
numpy (1.14.2)
protobuf (3.5.2.post1)
tensorflow (1.6.0)
tensorflow-tensorboard (0.4.0)

== check for virtualenv =========================================
False

== tensorflow import ============================================
tf.VERSION = 1.4.1
tf.GIT_VERSION = v1.4.1-0-g438604f
tf.COMPILER_VERSION = v1.4.1-0-g438604f
Sanity check: array([1], dtype=int32)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "tensorflow/__init__.py", line 24, in <module>
    from tensorflow.python import *  # pylint: disable=redefined-builtin
  File "tensorflow/python/__init__.py", line 49, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "tensorflow/python/pywrap_tensorflow.py", line 25, in <module>
    from tensorflow.python.platform import self_check
ImportError: No module named platform

== env ==========================================================
LD_LIBRARY_PATH :/usr/local/cuda/lib64:/home/alexandr/distr/tensorrt/lib:/usr/local/lib
DYLD_LIBRARY_PATH is unset

== nvidia-smi ===================================================
Thu Mar 22 10:33:28 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 387.26                 Driver Version: 387.26                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1070    Off  | 00000000:01:00.0  On |                  N/A |
| N/A   57C    P0    41W /  N/A |    372MiB /  8112MiB |     39%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1166      G   /usr/lib/xorg/Xorg                           211MiB |
|    0      3086      G   compiz                                       150MiB |
|    0      3136      G   ...-token=69155177EA9D6BB8687BA31223F7A104     7MiB |
+-----------------------------------------------------------------------------+

== cuda libs  ===================================================
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudart.so.8.0.61
/usr/local/cuda-9.0/targets/x86_64-linux/lib/libcudart.so.9.0.176
/usr/local/cuda-9.0/targets/x86_64-linux/lib/libcudart_static.a
/usr/local/cuda-9.0/doc/man/man7/libcudart.so.7
/usr/local/cuda-9.0/doc/man/man7/libcudart.7
/usr/local/cuda-9.1/targets/x86_64-linux/lib/libcudart_static.a
/usr/local/cuda-9.1/targets/x86_64-linux/lib/libcudart.so.9.1.85
/usr/local/cuda-9.1/doc/man/man7/libcudart.so.7
/usr/local/cuda-9.1/doc/man/man7/libcudart.7

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 21 (10 by maintainers)

Most upvoted comments

I though in your internal builds, LD_LIBRARY_PATH was ignored due to sandboxing. @aaroey mentioned had to do some fixes to workaround it. Also bazel build <your other config flags> -s //tensorflow/contrib/tensorrt:gen_trt_engine_op_py_wrappers_cc might help with identifying the issue. Another point, notice that cudnn is not added to the linker command line. Check whether it is in linker scipt. if not you are missing a dependency in your repositories.