tensorflow: ERROR: ~/....../tensorflow/python/keras/api/BUILD:130:1: Executing genrule //tensorflow/python/keras/api:keras_python_api_gen_compat_v2 failed

bazel 1.1.0

➜  tensorflow git:(master) ✗ bazel version
Build label: 1.1.0- (@non-git)
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Wed. Jan. 8 01:20:59 2020 (1578446459)
Build timestamp: 1578446459
Build timestamp as int: 1578446459

failed to build tensorflow master.

INFO: From Compiling tensorflow/lite/toco/graph_transformations/propagate_array_data_types.cc [for host]:
tensorflow/lite/toco/graph_transformations/propagate_array_data_types.cc: In member function 'virtual tensorflow::Status toco::PropagateArrayDataTypes::Run(toco::Model*, std::size_t, bool*)':
tensorflow/lite/toco/graph_transformations/propagate_array_data_types.cc:173:25: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
       for (int i = 0; i < op->outputs.size(); ++i) {
                       ~~^~~~~~~~~~~~~~~~~~~~
ERROR: ~/....../tensorflow/python/keras/api/BUILD:130:1: Executing genrule //tensorflow/python/keras/api:keras_python_api_gen_compat_v2 failed (Exit 1)
Traceback (most recent call last):
  File "~/.cache/bazel/_bazel_longervision/449dd09be7229482c168e6868d06849c/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_keras_python_api_gen_compat_v2.runfiles/org_tensorflow/tensorflow/python/tools/api/generator/create_python_api.py", line 776, in <module>
    main()
  File "~/.cache/bazel/_bazel_longervision/449dd09be7229482c168e6868d06849c/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_keras_python_api_gen_compat_v2.runfiles/org_tensorflow/tensorflow/python/tools/api/generator/create_python_api.py", line 772, in main
    lazy_loading, args.use_relative_imports)
  File "~/.cache/bazel/_bazel_longervision/449dd09be7229482c168e6868d06849c/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_keras_python_api_gen_compat_v2.runfiles/org_tensorflow/tensorflow/python/tools/api/generator/create_python_api.py", line 629, in create_api_files
    compat_api_versions, lazy_loading, use_relative_imports)
  File "~/.cache/bazel/_bazel_longervision/449dd09be7229482c168e6868d06849c/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_keras_python_api_gen_compat_v2.runfiles/org_tensorflow/tensorflow/python/tools/api/generator/create_python_api.py", line 503, in get_api_init_text
    _, attr = tf_decorator.unwrap(attr)
  File "~/.cache/bazel/_bazel_longervision/449dd09be7229482c168e6868d06849c/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_keras_python_api_gen_compat_v2.runfiles/org_tensorflow/tensorflow/python/util/tf_decorator.py", line 219, in unwrap
    elif _has_tf_decorator_attr(cur):
  File "~/.cache/bazel/_bazel_longervision/449dd09be7229482c168e6868d06849c/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_keras_python_api_gen_compat_v2.runfiles/org_tensorflow/tensorflow/python/util/tf_decorator.py", line 124, in _has_tf_decorator_attr
    hasattr(obj, '_tf_decorator') and
  File "~/.cache/bazel/_bazel_longervision/449dd09be7229482c168e6868d06849c/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_keras_python_api_gen_compat_v2.runfiles/org_tensorflow/tensorflow/python/util/lazy_loader.py", line 62, in __getattr__
    module = self._load()
  File "~/.cache/bazel/_bazel_longervision/449dd09be7229482c168e6868d06849c/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_keras_python_api_gen_compat_v2.runfiles/org_tensorflow/tensorflow/python/util/lazy_loader.py", line 45, in _load
    module = importlib.import_module(self.__name__)
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "~/.cache/bazel/_bazel_longervision/449dd09be7229482c168e6868d06849c/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_keras_python_api_gen_compat_v2.runfiles/org_tensorflow/tensorflow/compiler/tf2tensorrt/wrap_py_utils.py", line 28, in <module>
    _wrap_py_utils = swig_import_helper()
  File "~/.cache/bazel/_bazel_longervision/449dd09be7229482c168e6868d06849c/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_keras_python_api_gen_compat_v2.runfiles/org_tensorflow/tensorflow/compiler/tf2tensorrt/wrap_py_utils.py", line 24, in swig_import_helper
    _mod = imp.load_module('_wrap_py_utils', fp, pathname, description)
  File "/usr/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/usr/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
  File "<frozen importlib._bootstrap>", line 684, in _load
  File "<frozen importlib._bootstrap>", line 658, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 922, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: ~/.cache/bazel/_bazel_longervision/449dd09be7229482c168e6868d06849c/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/python/keras/api/create_tensorflow.python_api_keras_python_api_gen_compat_v2.runfiles/org_tensorflow/tensorflow/compiler/tf2tensorrt/_wrap_py_utils.so: undefined symbol: _ZN15stream_executor14StreamExecutor18EnablePeerAccessToEPS0_
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: ~/....../tensorflow/python/tools/BUILD:81:1 Executing genrule //tensorflow/python/keras/api:keras_python_api_gen_compat_v2 failed (Exit 1)
INFO: Elapsed time: 10740.244s, Critical Path: 307.06s
INFO: 15793 processes: 15793 local.
FAILED: Build did NOT complete successfully

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 16 (1 by maintainers)

Most upvoted comments

Ubuntu 18.04, TF 2.1, CUDA 10.2, TensorRT 7, cudnn 7, bazel 0.27.2 and with 0.29.1 too.

[39,613 / 39,674] 15 actions running
[39,620 / 39,675] 16 actions, 15 running
    Compiling tensorflow/core/kernels/conv_ops_fused_float.cc; 203s local
[39,629 / 39,677] 15 actions, 14 running
    Compiling tensorflow/core/kernels/conv_ops_fused_float.cc; 203s local
ERROR: /home/jonas/tensorflow/tensorflow/BUILD:881:1: Executing genrule //tensorflow:tf_python_api_gen_v2 failed (Exit 1)
[39,634 / 39,678] 15 actions running
    Compiling tensorflow/core/kernels/conv_ops_fused_float.cc; 203s local
2020-02-08 14:50:30.456621: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
Traceback (most recent call last):
  File "/home/jonas/.cache/bazel/_bazel_jonas/94e1c862effc6da040a35101d0b11916/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api_2_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/python/tools/api/generator/create_python_api.py", line 27, in <module>
    from tensorflow.python.tools.api.generator import doc_srcs
  File "/home/jonas/.cache/bazel/_bazel_jonas/94e1c862effc6da040a35101d0b11916/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api_2_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/python/__init__.py", line 85, in <module>
    from tensorflow.python.ops.standard_ops import *
  File "/home/jonas/.cache/bazel/_bazel_jonas/94e1c862effc6da040a35101d0b11916/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api_2_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/python/ops/standard_ops.py", line 117, in <module>
    from tensorflow.python.compiler.tensorrt import trt_convert as trt
  File "/home/jonas/.cache/bazel/_bazel_jonas/94e1c862effc6da040a35101d0b11916/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api_2_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/python/compiler/tensorrt/__init__.py", line 22, in <module>
    from tensorflow.python.compiler.tensorrt import trt_convert as trt
  File "/home/jonas/.cache/bazel/_bazel_jonas/94e1c862effc6da040a35101d0b11916/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api_2_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/python/compiler/tensorrt/trt_convert.py", line 28, in <module>
    from tensorflow.compiler.tf2tensorrt import wrap_py_utils
  File "/home/jonas/.cache/bazel/_bazel_jonas/94e1c862effc6da040a35101d0b11916/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api_2_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/compiler/tf2tensorrt/wrap_py_utils.py", line 28, in <module>
    _wrap_py_utils = swig_import_helper()
  File "/home/jonas/.cache/bazel/_bazel_jonas/94e1c862effc6da040a35101d0b11916/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api_2_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/compiler/tf2tensorrt/wrap_py_utils.py", line 24, in swig_import_helper
    _mod = imp.load_module('_wrap_py_utils', fp, pathname, description)
  File "/usr/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/usr/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: /home/jonas/.cache/bazel/_bazel_jonas/94e1c862effc6da040a35101d0b11916/execroot/org_tensorflow/bazel-out/host/bin/tensorflow/create_tensorflow.python_api_2_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/compiler/tf2tensorrt/_wrap_py_utils.so: undefined symbol: _ZN15stream_executor14StreamExecutor18EnablePeerAccessToEPS0_
----------------
Note: The failure of target //tensorflow:create_tensorflow.python_api_2_tf_python_api_gen_v2 (with exit code 1) may have been caused by the fact that it is a Python 2 program that was built in the host configuration, which uses Python 3. You can change the host configuration (for the entire build) to instead use Python 2 by setting --host_force_python=PY2.

If this error started occurring in Bazel 0.27 and later, it may be because the Python toolchain now enforces that targets analyzed as PY2 and PY3 run under a Python 2 and Python 3 interpreter, respectively. See https://github.com/bazelbuild/bazel/issues/7899 for more information.
----------------
[39,634 / 39,678] 15 actions running
    Compiling tensorflow/core/kernels/conv_ops_fused_float.cc; 203s local
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /home/jonas/tensorflow/tensorflow/python/tools/BUILD:140:1 Executing genrule //tensorflow:tf_python_api_gen_v2 failed (Exit 1)
INFO: Elapsed time: 9782.993s, Critical Path: 386.77s
INFO: 22453 processes: 22453 local.

jggithub2017 on Feb 9, 2020

Please provide details about what platform you are using (operating system, architecture). Also include your TensorFlow version.

Provide the exact sequence of commands / steps that you executed before running into the problem. Thanks!

I simply made several docker images for building tensorflow. I have uploaded it to the docker hub and the link is here.

The images with compilation problems are as follows(You can click on the link to see the dockerfile that made the image):

The above images already contain the source code of Tensorflow 2.1.0, and the submission signature is e5bf8de410005de06a7ff5393fafdf832ef1d4ad.

The differences between the above images are the python and cuda versions, as the tag names indicate. Each image uses ubuntu18.04. Specific environment can refer to the introduction on this page. For example, the environment of the image(vistart/cuda:10.0-cudnn7-tensorrt7-devel-ubuntu18.04) is:

cuda 10.2.89
cudnn 7.6.5.32
nccl 2.5.6
tensorrt 7.0.0.11

Compilation parameters(.tf_configure.bazelrc)

python2.7 | python3.6

Steps for usage:

1. Create the container for building tensorflow:

docker run -itd --rm --runtime=nvidia --name build_tensorflow_2_py27_cuda10.0 -v /home/jenkins/workspace:/mnt  -w /root/tensorflow -u root

2. Build the tensorflow

docker exec build_tensorflow_2_py27_cuda10.0 bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package

vistart on Jan 13, 2020