tensorflow: TF 2.13.0-rc0 fails to compile on Ubuntu 22.04

Click to expand!

Issue Type

Build/Install

Have you reproduced the bug with TF nightly?

Source

source

Tensorflow Version

2.13.0-rc0

Custom Code

OS Platform and Distribution

Linux Ubuntu 22.04 / MacOS 13.3

Mobile device

No response

Python version

3.10.6

Bazel version

5.3.0

GCC/Compiler version

gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0

CUDA/cuDNN version

11.8

GPU model and memory

Quadro RTX 6000

Current Behaviour?

When compiling TF 2.13.0-rc0 from source with default bazel parameters (see attached log) compilation fails.

Standalone code to reproduce the issue

Use version of TF 2.13.0-rc0. Follow the default bazel ./configure parameters. The issues happens regardless of compilation with or without CUDA. 

Please input the desired Python library path to use.  Default is [/usr/lib/python3/dist-packages]

Do you wish to build TensorFlow with ROCm support? [y/N]: 
No ROCm support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: 
No CUDA support will be enabled for TensorFlow.

Do you wish to download a fresh release of clang? (Experimental) [y/N]: 
Clang will not be downloaded.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -Wno-sign-compare]: -Wno-sign-compare 


Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: 
Not configuring the WORKSPACE for Android builds.

Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
	--config=mkl         	# Build with MKL support.
	--config=mkl_aarch64 	# Build with oneDNN and Compute Library for the Arm Architecture (ACL).
	--config=monolithic  	# Config for mostly static monolithic build.
	--config=numa        	# Build with NUMA support.
	--config=dynamic_kernels	# (Experimental) Build kernels into separate shared objects.
	--config=v1          	# Build with TensorFlow 1 API instead of TF 2 API.
Preconfigured Bazel build configs to DISABLE default on features:
	--config=nogcp       	# Disable GCP support.
	--config=nonccl      	# Disable NVIDIA NCCL support.
Configuration finished

Relevant log output

INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=100
INFO: Reading rc options for 'build' from /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc:
  'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility
INFO: Reading rc options for 'build' from /home/nicola/Software/tensorflow/gpu/tensorflow/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/usr/bin/python3 --action_env PYTHON_LIB_PATH=/usr/lib/python3/dist-packages --python_path=/usr/bin/python3
INFO: Reading rc options for 'build' from /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc:
  'build' options: --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/ir,tensorflow/compiler/mlir/tfrt/tests/analysis,tensorflow/compiler/mlir/tfrt/tests/jit,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_tfrt,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_jitrt,tensorflow/compiler/mlir/tfrt/tests/tf_to_corert,tensorflow/compiler/mlir/tfrt/tests/tf_to_tfrt_data,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/tfrt/eager,tensorflow/core/tfrt/eager/backends/cpu,tensorflow/core/tfrt/eager/backends/gpu,tensorflow/core/tfrt/eager/core_runtime,tensorflow/core/tfrt/eager/cpp_tests/core_runtime,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/graph_executor,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils,tensorflow/core/tfrt/utils/debug
INFO: Found applicable config definition build:short_logs in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:opt in file /home/nicola/Software/tensorflow/gpu/tensorflow/.tf_configure.bazelrc: --copt=-Wno-sign-compare --host_copt=-Wno-sign-compare
INFO: Found applicable config definition build:linux in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --define=build_with_onednn_v2=true --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/tensorflow/runtime/archive/7d879c8b161085a4374ea481b93a52adb19c0529.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-project/archive/dc275fd03254d67d29cc70a5a0569acf24d2280d.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/google/benchmark/archive/f7547e29ccaed7b64ef4f7495ecfff1c9f6f3d03.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://mirror.bazel.build/github.com/bazelbuild/rules_cc/archive/081771d4a0e9d7d3aa0eed2ef389fa4700dfb23e.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/google/flatbuffers/archive/v23.1.21.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/abseil/abseil-cpp/archive/b971ac5250ea8de900eae9f95e06548d14cd95fe.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/glennrp/libpng/archive/v1.6.39.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/google/XNNPACK/archive/b9d4073a6913891ce9cbd8965c8d506075d2a45a.zip failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/openxla/stablehlo/archive/43d81c6883ade82052920bd367c61f9e52f09954.zip failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/google/boringssl/archive/c00d7ca810e93780bd0c8ee4eea28f4f2ea4bcdc.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/pybind/pybind11_abseil/archive/2c4932ed6f6204f1656e245838f4f5eae69d2e29.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
ERROR: /home/nicola/.cache/bazel/_bazel_nicola/c53ed0be17816f9e0970b1ba234e403c/external/local_config_python/BUILD:78:8: in outs attribute of genrule rule @local_config_python//:python_include: Genrules without outputs don't make sense
ERROR: /home/nicola/.cache/bazel/_bazel_nicola/c53ed0be17816f9e0970b1ba234e403c/external/local_config_python/BUILD:78:8: Analysis of target '@local_config_python//:python_include' failed
INFO: Repository cython instantiated at:
  /home/nicola/Software/tensorflow/gpu/tensorflow/WORKSPACE:15:14: in <toplevel>
  /home/nicola/Software/tensorflow/gpu/tensorflow/tensorflow/workspace2.bzl:972:21: in workspace
  /home/nicola/Software/tensorflow/gpu/tensorflow/tensorflow/workspace2.bzl:706:20: in _tf_repositories
  /home/nicola/Software/tensorflow/gpu/tensorflow/third_party/repo.bzl:136:21: in tf_http_archive
Repository rule _tf_http_archive defined at:
  /home/nicola/Software/tensorflow/gpu/tensorflow/third_party/repo.bzl:89:35: in <toplevel>
ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' failed; build aborted: 
INFO: Elapsed time: 87.858s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (389 packages loaded, 6975 targets configured)
    Fetching https://storage.googleapis.com/.../github.com/cython/cython/archive/3.0.0a11.tar.gz

About this issue

Original URL
State: closed
Created a year ago
Reactions: 2
Comments: 38 (26 by maintainers)

Most upvoted comments

As described above there is no change needed in TF, just making sure that Cython is in the correct version:

Compilation fails when Cython is newer than 0.29.28 is installed (currently the latest is 0.29.32). Cython needs to be v0.29.28, which is also that provided by Ubuntu via apt.

I am not sure what I should push as PR. Rather, this should be listed within the set of requirements for compilation…

feranick on Jul 14, 2023

[UPDATE 2 - MacOS] It turns out that for MacOS the issue is with realpath, as listed in issue #60179. While that issue is closed, the complete lack of documentation on how to fix it (which involves third-party libraries installed) in the documentation is still an issue. So for MacOS the issue is a duplicate of #60179, so I removed the MacOS from the issue title. It would be great to update the documentation to reflect the needs of realpath from Coreutils (via brew or macports) to successfully complete the build.

feranick on Jun 1, 2023

On MacOS, the error is related but slightly different:

ERROR: /Users/feranick/Desktop/tensorflow/tensorflow/BUILD:1134:21: declared output ‘tensorflow/libtensorflow_framework.2.dylib’ was not created by genrule. This is probably because the genrule actually didn’t create this output, or because the output was a directory and the genrule was run remotely (note that only the contents of declared file outputs are copied from genrules run remotely) ERROR: /Users/feranick/Desktop/tensorflow/tensorflow/BUILD:1134:21: Executing genrule //tensorflow:libtensorflow_framework.2.dylib_sym failed: not all outputs were created or valid realpath: illegal option – - usage: realpath [-q] [path …] Target //tensorflow/tools/pip_package:build_pip_package failed to build

feranick on May 16, 2023