tensorflow: Failed to build TF2.0 with TensorRT: undefined symbol: _ZN15stream_executor14StreamExecutor18EnablePeerAccessToEPS0_
System information
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 19.04
- TensorFlow installed from (source or binary): source
- TensorFlow version: 2.0
- Python version: 3.7.3
- Installed using virtualenv? pip? conda?: pip
- Bazel version (if compiling from source): 1.1.0- (@non-git)
- GCC/Compiler version (if compiling from source): gcc (Ubuntu 8.3.0-6ubuntu1) 8.3.0
- CUDA/cuDNN version: 10.2/7.6.5
- GPU model and memory:
Device 0: "GeForce GTX 980M"
CUDA Driver Version / Runtime Version 10.2 / 10.2
CUDA Capability Major/Minor version number: 5.2
Total amount of global memory: 4035 MBytes (4231331840 bytes)
(12) Multiprocessors, (128) CUDA Cores/MP: 1536 CUDA Cores
GPU Max Clock rate: 1126 MHz (1.13 GHz)
WARNING: ....../tensorflow/python/BUILD:107:1: in py_library rule //tensorflow/python:no_contrib: target '//tensorflow/python:no_contrib' depends on deprecated target '//tensorflow/python/ops/distributions:distributions': TensorFlow Distributions has migrated to TensorFlow Probability (https://github.com/tensorflow/probability). Deprecated copies remaining in tf.distributions will not receive new features, and will be removed by early 2019. You should update all usage of `tf.distributions` to `tfp.distributions`.
INFO: Analyzed target //tensorflow/tools/pip_package:build_pip_package (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: ....../tensorflow/python/keras/api/BUILD:129:1: Executing genrule //tensorflow/python/keras/api:keras_python_api_gen_compat_v2 failed (Exit 1)
Traceback (most recent call last):
File "~/.cache/bazel/_bazel_longervision/b29adcaa7335b5e3a52f33ba4b4f5496/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_2_keras_python_api_gen_compat_v2.runfiles/org_tensorflow/tensorflow/python/tools/api/generator/create_python_api.py", line 776, in <module>
main()
File "~/.cache/bazel/_bazel_longervision/b29adcaa7335b5e3a52f33ba4b4f5496/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_2_keras_python_api_gen_compat_v2.runfiles/org_tensorflow/tensorflow/python/tools/api/generator/create_python_api.py", line 772, in main
lazy_loading, args.use_relative_imports)
File "~/.cache/bazel/_bazel_longervision/b29adcaa7335b5e3a52f33ba4b4f5496/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_2_keras_python_api_gen_compat_v2.runfiles/org_tensorflow/tensorflow/python/tools/api/generator/create_python_api.py", line 629, in create_api_files
compat_api_versions, lazy_loading, use_relative_imports)
File "~/.cache/bazel/_bazel_longervision/b29adcaa7335b5e3a52f33ba4b4f5496/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_2_keras_python_api_gen_compat_v2.runfiles/org_tensorflow/tensorflow/python/tools/api/generator/create_python_api.py", line 503, in get_api_init_text
_, attr = tf_decorator.unwrap(attr)
File "~/.cache/bazel/_bazel_longervision/b29adcaa7335b5e3a52f33ba4b4f5496/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_2_keras_python_api_gen_compat_v2.runfiles/org_tensorflow/tensorflow/python/util/tf_decorator.py", line 219, in unwrap
elif _has_tf_decorator_attr(cur):
File "~/.cache/bazel/_bazel_longervision/b29adcaa7335b5e3a52f33ba4b4f5496/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_2_keras_python_api_gen_compat_v2.runfiles/org_tensorflow/tensorflow/python/util/tf_decorator.py", line 124, in _has_tf_decorator_attr
hasattr(obj, '_tf_decorator') and
File "~/.cache/bazel/_bazel_longervision/b29adcaa7335b5e3a52f33ba4b4f5496/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_2_keras_python_api_gen_compat_v2.runfiles/org_tensorflow/tensorflow/python/util/lazy_loader.py", line 62, in __getattr__
module = self._load()
File "~/.cache/bazel/_bazel_longervision/b29adcaa7335b5e3a52f33ba4b4f5496/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_2_keras_python_api_gen_compat_v2.runfiles/org_tensorflow/tensorflow/python/util/lazy_loader.py", line 45, in _load
module = importlib.import_module(self.__name__)
File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "~/.cache/bazel/_bazel_longervision/b29adcaa7335b5e3a52f33ba4b4f5496/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_2_keras_python_api_gen_compat_v2.runfiles/org_tensorflow/tensorflow/compiler/tf2tensorrt/wrap_py_utils.py", line 28, in <module>
_wrap_py_utils = swig_import_helper()
File "~/.cache/bazel/_bazel_longervision/b29adcaa7335b5e3a52f33ba4b4f5496/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_2_keras_python_api_gen_compat_v2.runfiles/org_tensorflow/tensorflow/compiler/tf2tensorrt/wrap_py_utils.py", line 24, in swig_import_helper
_mod = imp.load_module('_wrap_py_utils', fp, pathname, description)
File "/usr/lib/python3.7/imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "/usr/lib/python3.7/imp.py", line 342, in load_dynamic
return _load(spec)
File "<frozen importlib._bootstrap>", line 696, in _load
File "<frozen importlib._bootstrap>", line 670, in _load_unlocked
File "<frozen importlib._bootstrap>", line 583, in module_from_spec
File "<frozen importlib._bootstrap_external>", line 1043, in create_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: ~/.cache/bazel/_bazel_longervision/b29adcaa7335b5e3a52f33ba4b4f5496/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_2_keras_python_api_gen_compat_v2.runfiles/org_tensorflow/tensorflow/compiler/tf2tensorrt/_wrap_py_utils.so: undefined symbol: _ZN15stream_executor14StreamExecutor18EnablePeerAccessToEPS0_
Target //tensorflow/tools/pip_package:build_pip_package failed to build
ERROR: ....../tensorflow/python/tools/BUILD:98:1 Executing genrule //tensorflow/python/keras/api:keras_python_api_gen_compat_v2 failed (Exit 1)
INFO: Elapsed time: 3.987s, Critical Path: 2.78s
INFO: 0 processes.
FAILED: Build did NOT complete successfully
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 52 (23 by maintainers)
@ahtik and @dbonner It seems bazel 1.x + Python 3 is problematic when building TensorRT is enabled. Workarounds:
Can you try running with:
BAZEL_LINKOPTS=""andBAZEL_LINKLIBS=-lstdc++:-lmOh, I see. I’ll try
bazel clean --expungeand use the bazel version tensorflow recommends (1.2.1) next time I try to build.