tensorflow: Error build: target 'cuda/bin/nvcc' not declared in package

Above error when building from source with GPU on Ubuntu 18.04.

To reproduce:

 git clone https://github.com/tensorflow/tensorflow.git
cd tensorflow
./configure # all defaults except for CUDA: y
bazel build --copt=-march=native --config=nonccl --config=cuda //tensorflow/tools/pip_package:build_pip_package

OS: Ubuntu 18.04 Python 3.9.6: Anaconda managed environment GPU: NVIDIA 1080 Ti CUDA: V11.2.152 bazel 3.7.2

Full stacktrace:

bazel build --copt=-march=native --config=nonccl --config=cuda //tensorflow/tools/pip_package:build_pip_package --verbose_failures --explain=file.txt --verbose_failures 2>&1 | tee bazel.log
WARNING: The following configs were expanded more than once: [cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'build' from /home/sergey/tensorflow/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /home/sergey/tensorflow/.bazelrc:
  'build' options: --define framework_shared_object=true --java_toolchain=@tf_toolchains//toolchains/java:tf_java_toolchain --host_java_toolchain=@tf_toolchains//toolchains/java:tf_java_toolchain --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true
INFO: Reading rc options for 'build' from /home/sergey/tensorflow/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/home/sergey/anaconda3/bin/python3 --action_env PYTHON_LIB_PATH=/home/sergey/anaconda3/lib/python3.9/site-packages --python_path=/home/sergey/anaconda3/bin/python3 --action_env CUDA_TOOLKIT_PATH=/usr/local/cuda-11.2 --action_env TF_CUDA_COMPUTE_CAPABILITIES=6.1 --action_env LD_LIBRARY_PATH=/home/sergey/hadoop/lib/native/:/usr/lib/x86_64-linux-gnu/:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64 --action_env GCC_HOST_COMPILER_PATH=/usr/bin/x86_64-linux-gnu-gcc-7 --config=cuda
INFO: Found applicable config definition build:short_logs in file /home/sergey/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /home/sergey/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:cuda in file /home/sergey/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:nonccl in file /home/sergey/tensorflow/.bazelrc: --define=no_nccl_support=true
INFO: Found applicable config definition build:cuda in file /home/sergey/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:linux in file /home/sergey/tensorflow/.bazelrc: --copt=-w --host_copt=-w --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++14 --host_cxxopt=-std=c++14 --config=dynamic_kernels --distinct_host_configuration=false
INFO: Found applicable config definition build:dynamic_kernels in file /home/sergey/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
DEBUG: /home/sergey/.cache/bazel/_bazel_sergey/3b767bf3a32053879a4e137b3de41895/external/tf_runtime/third_party/cuda/dependencies.bzl:51:10: The following command will download NVIDIA proprietary software. By using the software you agree to comply with the terms of the license agreement that accompanies the software. If you do not agree to the terms of the license agreement, do not use the software.
DEBUG: /home/sergey/.cache/bazel/_bazel_sergey/3b767bf3a32053879a4e137b3de41895/external/bazel_tools/tools/cpp/lib_cc_configure.bzl:118:10:
Auto-Configuration Warning: 'TMP' environment variable is not set, using 'C:\Windows\Temp' as default
Loading:  (1 packages loaded)
Loading: 1 packages loaded
Analyzing: target //tensorflow/tools/pip_package:build_pip_package (2 packages loaded, 0 targets configured)
Analyzing: target //tensorflow/tools/pip_package:build_pip_package (57 packages loaded, 17 targets configured)
Analyzing: target //tensorflow/tools/pip_package:build_pip_package (227 packages loaded, 3960 targets configured)
Analyzing: target //tensorflow/tools/pip_package:build_pip_package (246 packages loaded, 3960 targets configured)
Analyzing: target //tensorflow/tools/pip_package:build_pip_package (248 packages loaded, 3960 targets configured)
Analyzing: target //tensorflow/tools/pip_package:build_pip_package (249 packages loaded, 3960 targets configured)
DEBUG: Rule 'io_bazel_rules_docker' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1556410077 -0400"
DEBUG: Repository io_bazel_rules_docker instantiated at:
  /home/sergey/tensorflow/WORKSPACE:23:14: in <toplevel>
  /home/sergey/tensorflow/tensorflow/workspace0.bzl:108:34: in workspace
  /home/sergey/.cache/bazel/_bazel_sergey/3b767bf3a32053879a4e137b3de41895/external/bazel_toolchains/repositories/repositories.bzl:37:23: in repositories
Repository rule git_repository defined at:
  /home/sergey/.cache/bazel/_bazel_sergey/3b767bf3a32053879a4e137b3de41895/external/bazel_tools/tools/build_defs/repo/git.bzl:199:33: in <toplevel>
Analyzing: target //tensorflow/tools/pip_package:build_pip_package (332 packages loaded, 5329 targets configured)
Analyzing: target //tensorflow/tools/pip_package:build_pip_package (427 packages loaded, 13775 targets configured)
ERROR: /home/sergey/.cache/bazel/_bazel_sergey/3b767bf3a32053879a4e137b3de41895/external/rules_cuda/cuda/BUILD:128:20: every rule of type cuda_toolchain_info implicitly depends upon the target '@local_cuda//:cuda/bin/nvcc', but this target could not be found because of: no such target '@local_cuda//:cuda/bin/nvcc': target 'cuda/bin/nvcc' not declared in package '' (did you mean 'cuda/bin/h5cc'?) defined by /home/sergey/.cache/bazel/_bazel_sergey/3b767bf3a32053879a4e137b3de41895/external/local_cuda/BUILD
INFO: Repository eigen_archive instantiated at:
  /home/sergey/tensorflow/WORKSPACE:15:14: in <toplevel>
  /home/sergey/tensorflow/tensorflow/workspace2.bzl:1091:28: in workspace
  /home/sergey/tensorflow/tensorflow/workspace2.bzl:61:11: in _initialize_third_party
  /home/sergey/tensorflow/third_party/eigen3/workspace.bzl:12:20: in repo
  /home/sergey/tensorflow/third_party/repo.bzl:113:21: in tf_http_archive
Repository rule _tf_http_archive defined at:
  /home/sergey/tensorflow/third_party/repo.bzl:66:35: in <toplevel>
INFO: Repository org_sqlite instantiated at:
  /home/sergey/tensorflow/WORKSPACE:15:14: in <toplevel>
  /home/sergey/tensorflow/tensorflow/workspace2.bzl:1098:21: in workspace
  /home/sergey/tensorflow/tensorflow/workspace2.bzl:347:20: in _tf_repositories
  /home/sergey/tensorflow/third_party/repo.bzl:113:21: in tf_http_archive
Repository rule _tf_http_archive defined at:
  /home/sergey/tensorflow/third_party/repo.bzl:66:35: in <toplevel>
INFO: Repository llvm-project instantiated at:
  /home/sergey/tensorflow/WORKSPACE:15:14: in <toplevel>
  /home/sergey/tensorflow/tensorflow/workspace2.bzl:1098:21: in workspace
  /home/sergey/tensorflow/tensorflow/workspace2.bzl:657:9: in _tf_repositories
  /home/sergey/tensorflow/third_party/llvm/workspace.bzl:10:20: in repo
  /home/sergey/tensorflow/third_party/repo.bzl:113:21: in tf_http_archive
Repository rule _tf_http_archive defined at:
  /home/sergey/tensorflow/third_party/repo.bzl:66:35: in <toplevel>
INFO: Repository cudnn_frontend_archive instantiated at:
  /home/sergey/tensorflow/WORKSPACE:15:14: in <toplevel>
  /home/sergey/tensorflow/tensorflow/workspace2.bzl:1098:21: in workspace
  /home/sergey/tensorflow/tensorflow/workspace2.bzl:158:20: in _tf_repositories
  /home/sergey/tensorflow/third_party/repo.bzl:113:21: in tf_http_archive
Repository rule _tf_http_archive defined at:
  /home/sergey/tensorflow/third_party/repo.bzl:66:35: in <toplevel>
Analyzing: target //tensorflow/tools/pip_package:build_pip_package (427 packages loaded, 13776 targets configured)
INFO: Repository mkl_dnn_v1 instantiated at:
  /home/sergey/tensorflow/WORKSPACE:15:14: in <toplevel>
  /home/sergey/tensorflow/tensorflow/workspace2.bzl:1098:21: in workspace
  /home/sergey/tensorflow/tensorflow/workspace2.bzl:181:20: in _tf_repositories
  /home/sergey/tensorflow/third_party/repo.bzl:113:21: in tf_http_archive
Repository rule _tf_http_archive defined at:
  /home/sergey/tensorflow/third_party/repo.bzl:66:35: in <toplevel>
ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' failed; build aborted: Analysis failed
INFO: Elapsed time: 13.023s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (427 packages loaded, 13776 targets configured)
FAILED: Build did NOT complete successfully (427 packages loaded, 13776 targets configured)

Tensorflow 2.5.0 builds though on the same machine.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 17 (5 by maintainers)

Most upvoted comments

@sanjoy

There are many ways to circumvent this problem including pip or conda install tensorflow-gpu. Any thoughts how to resolve the error while building from source?