tensorflow: TF 1.7.0-rc1 unable to build with TensorRT support
System information
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
- TensorFlow installed from (source or binary): source
- TensorFlow version (use command below): 1.7.0-rc1
- Python version: Python 3.5
- Bazel version (if compiling from source): 0.11.1
- GCC/Compiler version (if compiling from source): 5.4.0-6ubuntu1~16.04.9
- CUDA/cuDNN version: 9.0 / 7.0.5
- GPU model and memory: GTX 1070, 8Gb
- Exact command to reproduce:
bazel build -c opt --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.1 --copt=-msse4.2 --config=cuda //tensorflow/tools/pip_package:build_pip_package --verbose_failures
Describe the problem
Try to build TF with TensorRT support using the command above and following this instruction, i.e. there is no TensorRT installed from repo, only downloaded binaries for 14.04 + echo "<install_dir>/TensorRT-3.0.4/lib" | sudo tee /etc/ld.so.conf.d/tensorrt304.conf && sudo ldconfig, as suggested.
The libcudnn.so* available in TensorRT lib dir, /usr/local/lib and /usr/lib/x86_64-linux-gnu. LD_LIBRARY_PATH contains all relevant paths.
While configure, I pointed to TensorRT binaries dir when it asked.
/usr/local/cuda points to 9.0 version and this path was specified while configure.
Source code / logs
WARNING: /home/alexandr/.cache/bazel/_bazel_alexandr/6c93df3ac6ca06598f74e718588ae6cc/external/protobuf_archive/WORKSPACE:1: Workspace name in /home/alexandr/.cache/bazel/_bazel_alexandr/6c93df3ac6ca06598f74e718588ae6cc/external/protobuf_archive/WORKSPACE (@com_google_protobuf) does not match the name given in the repository's definition (@protobuf_archive); this will cause a build error in future versions
WARNING: /home/alexandr/distr/tensorflow_tensorrt/tensorflow/core/BUILD:1955:1: in includes attribute of cc_library rule //tensorflow/core:framework_headers_lib: '../../external/nsync/public' resolves to 'external/nsync/public' not below the relative path of its package 'tensorflow/core'. This will be an error in the future. Since this rule was created by the macro 'cc_header_only_library', the error might have been caused by the macro implementation in /home/alexandr/distr/tensorflow_tensorrt/tensorflow/tensorflow.bzl:1179:30
WARNING: /home/alexandr/distr/tensorflow_tensorrt/tensorflow/core/BUILD:1955:1: in includes attribute of cc_library rule //tensorflow/core:framework_headers_lib: '../../external/nsync/public' resolves to 'external/nsync/public' not below the relative path of its package 'tensorflow/core'. This will be an error in the future. Since this rule was created by the macro 'cc_header_only_library', the error might have been caused by the macro implementation in /home/alexandr/distr/tensorflow_tensorrt/tensorflow/tensorflow.bzl:1179:30
WARNING: /home/alexandr/.cache/bazel/_bazel_alexandr/6c93df3ac6ca06598f74e718588ae6cc/external/grpc/WORKSPACE:1: Workspace name in /home/alexandr/.cache/bazel/_bazel_alexandr/6c93df3ac6ca06598f74e718588ae6cc/external/grpc/WORKSPACE (@com_github_grpc_grpc) does not match the name given in the repository's definition (@grpc); this will cause a build error in future versions
WARNING: /home/alexandr/distr/tensorflow_tensorrt/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:exporter': No longer supported. Switch to SavedModel immediately.
WARNING: /home/alexandr/distr/tensorflow_tensorrt/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:gc': No longer supported. Switch to SavedModel immediately.
INFO: Analysed target //tensorflow/tools/pip_package:build_pip_package (0 packages loaded).
INFO: Found 1 target...
ERROR: /home/alexandr/distr/tensorflow_tensorrt/tensorflow/contrib/tensorrt/BUILD:122:1: Linking of rule '//tensorflow/contrib/tensorrt:gen_trt_engine_op_py_wrappers_cc' failed (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command
(cd /home/alexandr/.cache/bazel/_bazel_alexandr/6c93df3ac6ca06598f74e718588ae6cc/execroot/org_tensorflow && \
exec env - \
LD_LIBRARY_PATH=:/usr/local/cuda/lib64 \
PATH=/home/alexandr/bin:/home/alexandr/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/cuda/bin \
PWD=/proc/self/cwd \
external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -o bazel-out/host/bin/tensorflow/contrib/tensorrt/gen_trt_engine_op_py_wrappers_cc '-Wl,-rpath,$ORIGIN/../../../_solib_local/_U_S_Stensorflow_Scontrib_Stensorrt_Cgen_Utrt_Uengine_Uop_Upy_Uwrappers_Ucc___Utensorflow' '-Wl,-rpath,$ORIGIN/../../../_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib' '-Wl,-rpath,$ORIGIN/../../../_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccublas___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib' '-Wl,-rpath,$ORIGIN/../../../_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccudnn___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib' '-Wl,-rpath,$ORIGIN/../../../_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccufft___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib' '-Wl,-rpath,$ORIGIN/../../../_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccurand___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib' '-Wl,-rpath,$ORIGIN/../../../_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccudart___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib' -Lbazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Stensorrt_Cgen_Utrt_Uengine_Uop_Upy_Uwrappers_Ucc___Utensorflow -Lbazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib -Lbazel-out/host/bin/_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccublas___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib -Lbazel-out/host/bin/_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccudnn___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib -Lbazel-out/host/bin/_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccufft___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib -Lbazel-out/host/bin/_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccurand___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib -Lbazel-out/host/bin/_solib_local/_U@local_Uconfig_Ucuda_S_Scuda_Ccudart___Uexternal_Slocal_Uconfig_Ucuda_Scuda_Scuda_Slib '-Wl,-rpath,$ORIGIN/,-rpath,$ORIGIN/..,-rpath,$ORIGIN/../..' -pthread -Wl,-rpath,../local_config_cuda/cuda/lib64 -Wl,-rpath,../local_config_cuda/cuda/extras/CUPTI/lib64 -Wl,-no-as-needed -B/usr/bin/ -pie -Wl,-z,relro,-z,now -no-canonical-prefixes -pass-exit-codes '-Wl,--build-id=md5' '-Wl,--hash-style=gnu' -Wl,--gc-sections -Wl,-S -Wl,@bazel-out/host/bin/tensorflow/contrib/tensorrt/gen_trt_engine_op_py_wrappers_cc-2.params)
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnCreate@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnCreatePoolingDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnCreateConvolutionDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDestroyConvolutionDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnGetReductionWorkspaceSize@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetPooling2dDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDestroyFilterDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetReduceTensorDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnLRNCrossChannelForward@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnCreateRNNDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDestroyRNNDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnConvolutionBackwardData@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnCreateFilterDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetTensor4dDescriptorEx@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDestroyLRNDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDestroy@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDropoutGetStatesSize@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnGetConvolutionForwardWorkspaceSize@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetActivationDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnGetRNNWorkspaceSize@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetConvolutionGroupCount@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetTensorNdDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnAddTensor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDestroyActivationDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnGetConvolutionBackwardDataAlgorithmMaxCount@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnGetFilterNdDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetLRNDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDestroyTensorDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDestroyDropoutDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnCreateTensorDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetStream@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDestroyOpTensorDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnCreateOpTensorDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnCreateDropoutDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetFilter4dDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDestroyPoolingDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnCreateActivationDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnRNNForwardInference@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnActivationForward@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnGetRNNLinLayerMatrixParams@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnGetConvolutionBackwardDataWorkspaceSize@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnReduceTensor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetRNNDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetDropoutDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSoftmaxForward@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnConvolutionForward@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnGetRNNLinLayerBiasParams@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetConvolution2dDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetFilterNdDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnGetRNNParamsSize@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetTensor4dDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnSetOpTensorDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnDestroyReduceTensorDescriptor@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnFindConvolutionBackwardDataAlgorithmEx@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnPoolingForward@libcudnn.so.7'
bazel-out/host/bin/_solib_local/_U@local_Uconfig_Utensorrt_S_S_Cnv_Uinfer___Uexternal_Slocal_Uconfig_Utensorrt_Stensorrt_Slib/libnvinfer.so.4: undefined reference to `cudnnCreateLRNDescriptor@libcudnn.so.7'
collect2: error: ld returned 1 exit status
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 0.642s, Critical Path: 0.22s
FAILED: Build did NOT complete successfully
The extended system info:
== cat /etc/issue ===============================================
Linux power-linux 4.12.0-041200-generic #201707022031 SMP Mon Jul 3 00:32:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
VERSION="16.04.4 LTS (Xenial Xerus)"
VERSION_ID="16.04"
VERSION_CODENAME=xenial
== are we in docker =============================================
No
== compiler =====================================================
c++ (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
== uname -a =====================================================
Linux power-linux 4.12.0-041200-generic #201707022031 SMP Mon Jul 3 00:32:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
== check pips ===================================================
msgpack-numpy (0.4.1)
numpy (1.14.2)
protobuf (3.5.2.post1)
tensorflow (1.6.0)
tensorflow-tensorboard (0.4.0)
== check for virtualenv =========================================
False
== tensorflow import ============================================
tf.VERSION = 1.4.1
tf.GIT_VERSION = v1.4.1-0-g438604f
tf.COMPILER_VERSION = v1.4.1-0-g438604f
Sanity check: array([1], dtype=int32)
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "tensorflow/__init__.py", line 24, in <module>
from tensorflow.python import * # pylint: disable=redefined-builtin
File "tensorflow/python/__init__.py", line 49, in <module>
from tensorflow.python import pywrap_tensorflow
File "tensorflow/python/pywrap_tensorflow.py", line 25, in <module>
from tensorflow.python.platform import self_check
ImportError: No module named platform
== env ==========================================================
LD_LIBRARY_PATH :/usr/local/cuda/lib64:/home/alexandr/distr/tensorrt/lib:/usr/local/lib
DYLD_LIBRARY_PATH is unset
== nvidia-smi ===================================================
Thu Mar 22 10:33:28 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 387.26 Driver Version: 387.26 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 1070 Off | 00000000:01:00.0 On | N/A |
| N/A 57C P0 41W / N/A | 372MiB / 8112MiB | 39% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1166 G /usr/lib/xorg/Xorg 211MiB |
| 0 3086 G compiz 150MiB |
| 0 3136 G ...-token=69155177EA9D6BB8687BA31223F7A104 7MiB |
+-----------------------------------------------------------------------------+
== cuda libs ===================================================
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudart.so.8.0.61
/usr/local/cuda-9.0/targets/x86_64-linux/lib/libcudart.so.9.0.176
/usr/local/cuda-9.0/targets/x86_64-linux/lib/libcudart_static.a
/usr/local/cuda-9.0/doc/man/man7/libcudart.so.7
/usr/local/cuda-9.0/doc/man/man7/libcudart.7
/usr/local/cuda-9.1/targets/x86_64-linux/lib/libcudart_static.a
/usr/local/cuda-9.1/targets/x86_64-linux/lib/libcudart.so.9.1.85
/usr/local/cuda-9.1/doc/man/man7/libcudart.so.7
/usr/local/cuda-9.1/doc/man/man7/libcudart.7
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 21 (10 by maintainers)
I though in your internal builds, LD_LIBRARY_PATH was ignored due to sandboxing. @aaroey mentioned had to do some fixes to workaround it. Also
bazel build <your other config flags> -s //tensorflow/contrib/tensorrt:gen_trt_engine_op_py_wrappers_ccmight help with identifying the issue. Another point, notice that cudnn is not added to the linker command line. Check whether it is in linker scipt. if not you are missing a dependency in your repositories.