tensorflow: Build failed `not all outputs were created or valid` on `darwin/amd64`

Issue Type

Build/Install

Have you reproduced the bug with TF nightly?

Source

source

Tensorflow Version

faf4a8eb61d6344997e1860de30e6eae434b4411

Custom Code

OS Platform and Distribution

MacOS 13.3 (22E252) Apple M1 Pro

Mobile device

N/A

Python version

3.11

Bazel version

bazel 5.3.0

GCC/Compiler version

Apple clang version 14.0.0 (clang-1400.0.29.202)

CUDA/cuDNN version

N/A

GPU model and memory

16GB RAM, no NVIDIA GPU

Current Behaviour?

Build logs

$ bazel build --verbose_failures //tensorflow/tools/pip_package:build_pip_package
Starting local Bazel server and connecting to it...
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=156
INFO: Reading rc options for 'build' from /Users/ckpn/Mine/projects/python/bert_tst/tensorflow/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /Users/ckpn/Mine/projects/python/bert_tst/tensorflow/.bazelrc:
  'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility
INFO: Reading rc options for 'build' from /Users/ckpn/Mine/projects/python/bert_tst/tensorflow/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/opt/homebrew/opt/python@3.11/bin/python3.11 --action_env PYTHON_LIB_PATH=/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages --python_path=/opt/homebrew/opt/python@3.11/bin/python3.11
INFO: Reading rc options for 'build' from /Users/ckpn/Mine/projects/python/bert_tst/tensorflow/.bazelrc:
  'build' options: --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/ir,tensorflow/compiler/mlir/tfrt/tests/analysis,tensorflow/compiler/mlir/tfrt/tests/jit,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_tfrt,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_jitrt,tensorflow/compiler/mlir/tfrt/tests/tf_to_corert,tensorflow/compiler/mlir/tfrt/tests/tf_to_tfrt_data,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/tfrt/eager,tensorflow/core/tfrt/eager/backends/cpu,tensorflow/core/tfrt/eager/backends/gpu,tensorflow/core/tfrt/eager/core_runtime,tensorflow/core/tfrt/eager/cpp_tests/core_runtime,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/graph_executor,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils
INFO: Found applicable config definition build:short_logs in file /Users/ckpn/Mine/projects/python/bert_tst/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /Users/ckpn/Mine/projects/python/bert_tst/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:macos in file /Users/ckpn/Mine/projects/python/bert_tst/tensorflow/.bazelrc: --apple_platform_type=macos --copt=-DGRPC_BAZEL_BUILD --copt=-w --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-project/archive/6330447c2509c3669d64ae753c8030be1a38dc72.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found                                                                                 
INFO: Analyzed target //tensorflow/tools/pip_package:build_pip_package (606 packages loaded, 44253 targets configured).                                     
INFO: Found 1 target...                                                                                                                                     
ERROR: /Users/ckpn/Mine/projects/python/bert_tst/tensorflow/tensorflow/BUILD:1128:21: declared output 'tensorflow/libtensorflow_framework.2.dylib' was not created by genrule. This is probably because the genrule actually didn't create this output, or because the output was a directory and the genrule was run remotely (note that only the contents of declared file outputs are copied from genrules run remotely)                                                         
ERROR: /Users/ckpn/Mine/projects/python/bert_tst/tensorflow/tensorflow/BUILD:1128:21: Executing genrule //tensorflow:libtensorflow_framework.2.dylib_sym failed: not all outputs were created or valid                                                                                                                  
realpath: illegal option -- -                                                                                                                               
usage: realpath [-q] [path ...]                                                                                                                             
Target //tensorflow/tools/pip_package:build_pip_package failed to build                                                                                     
INFO: Elapsed time: 589.942s, Critical Path: 78.69s                                                                                                         
INFO: 4653 processes: 875 internal, 3778 local.                                                                                                             
FAILED: Build did NOT complete successfully

Expect no error

Standalone code to reproduce the issue

Prepare a computer with Mac M1 chip
Do git clone https://github.com/tensorflow/tensorflow
Install dependencies such as clang and bazel
Do bazel build --verbose_failures //tensorflow/tools/pip_package:build_pip_package

Relevant log output

See above

About this issue

Original URL
State: closed
Created a year ago
Reactions: 1
Comments: 16 (4 by maintainers)

Most upvoted comments

A recent commit (https://github.com/tensorflow/tensorflow/commit/51a8992bb9daba2364ac4bb931af1010d554f0fa) used non-standard GNU realpath --relative. Quick workaround is to use GNU realpath (in coreutils). On my mac, I used macport to install realpath to /opt/local/libexec/gnubin, then add /opt/local/libexec/gnubin before /usr/bin when running bazel.

freedomtan on Mar 31, 2023

Solved environment:

Commit: 55939be39409283c4b35a62049fe548a2de3ce44

XCode: 14.3 clang

Apple clang version 14.0.3 (clang-1403.0.22.14.1)

bazel 5.3.0

zyxkad on Apr 5, 2023

Hi @zyxkad ,

I have replicated the reported error.

suryanarayanay-macbookpro:tensorflow suryanarayanay$ bazel build --verbose_failures //tensorflow/tools/pip_package:build_pip_package
Starting local Bazel server and connecting to it...
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=214
INFO: Reading rc options for 'build' from /Users/suryanarayanay/tensorflow/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /Users/suryanarayanay/tensorflow/.bazelrc:
  'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/ir,tensorflow/compiler/mlir/tfrt/tests/analysis,tensorflow/compiler/mlir/tfrt/tests/jit,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_tfrt,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_jitrt,tensorflow/compiler/mlir/tfrt/tests/tf_to_corert,tensorflow/compiler/mlir/tfrt/tests/tf_to_tfrt_data,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/tfrt/eager,tensorflow/core/tfrt/eager/backends/cpu,tensorflow/core/tfrt/eager/backends/gpu,tensorflow/core/tfrt/eager/core_runtime,tensorflow/core/tfrt/eager/cpp_tests/core_runtime,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/graph_executor,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils
INFO: Found applicable config definition build:short_logs in file /Users/suryanarayanay/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /Users/suryanarayanay/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:macos in file /Users/suryanarayanay/tensorflow/.bazelrc: --apple_platform_type=macos --copt=-DGRPC_BAZEL_BUILD --copt=-w --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/tensorflow/runtime/archive/0aaa6e679847a4eeb407136e7b0bcef93ec652e6.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-project/archive/6330447c2509c3669d64ae753c8030be1a38dc72.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/google/benchmark/archive/f7547e29ccaed7b64ef4f7495ecfff1c9f6f3d03.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/google/flatbuffers/archive/v23.1.20.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/glennrp/libpng/archive/v1.6.39.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/google/boringssl/archive/c00d7ca810e93780bd0c8ee4eea28f4f2ea4bcdc.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/openxla/stablehlo/archive/458bdd95771e9861e6488868e315a1b0340058ba.zip failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/google/XNNPACK/archive/7adae8e6ded8fff33d92212ca1028d2419cd34d4.zip failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/pybind/pybind11_abseil/archive/2c4932ed6f6204f1656e245838f4f5eae69d2e29.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
INFO: Analyzed target //tensorflow/tools/pip_package:build_pip_package (605 packages loaded, 43895 targets configured).
INFO: Found 1 target...
INFO: SystemSuspensionEvent: Computer put to sleep
INFO: SystemSuspensionEvent: Computer woken up
ERROR: /Users/suryanarayanay/tensorflow/tensorflow/BUILD:1128:21: declared output 'tensorflow/libtensorflow_framework.2.dylib' was not created by genrule. This is probably because the genrule actually didn't create this output, or because the output was a directory and the genrule was run remotely (note that only the contents of declared file outputs are copied from genrules run remotely)
ERROR: /Users/suryanarayanay/tensorflow/tensorflow/BUILD:1128:21: Executing genrule //tensorflow:libtensorflow_framework.2.dylib_sym [for host] failed: not all outputs were created or valid
realpath: illegal option -- -
usage: realpath [-q] [path ...]
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 1936.248s, Critical Path: 31.46s
INFO: 6866 processes: 2275 internal, 4591 local.
FAILED: Build did NOT complete successfully
suryanarayanay-macbookpro:tensorflow suryanarayanay$

SuryanarayanaY on Apr 2, 2023