tensorflow: Still seeing 'missing dependency declarations' when building with Bazel 0.1.4/0.1.5

I’m still getting the error from #469, even with the fix in #788.

Debian, gcc 4.9.2, bazel 0.1.5 (tried 0.1.4 as well with the same results).

Command line:

bazel build -c opt --config=cuda --verbose_failures --spawn_strategy=standalone  //tensorflow/cc:tutorials_example_trainer

Error output:

ERROR: /usr/local/src/tensorflow/tensorflow/core/BUILD:334:1: undeclared inclusion(s) in rule '//tensorflow/core:gpu_kernels':
this rule is missing dependency declarations for the following files included by 'tensorflow/core/kernels/cwise_op_gpu_cos.cu.cc':
  '/usr/local/cuda/include/vector_types.h'
  '/usr/local/cuda/include/math_constants.h'
  '/usr/local/cuda/include/cuda_runtime.h'
  '/usr/local/cuda/include/curand_kernel.h'
  '/usr/local/cuda/include/curand.h'
  '/usr/local/cuda/include/curand_discrete.h'
  '/usr/local/cuda/include/curand_precalc.h'
  '/usr/local/cuda/include/curand_mrg32k3a.h'
  '/usr/local/cuda/include/curand_mtgp32_kernel.h'
  '/usr/local/cuda/include/cuda.h'
  '/usr/local/cuda/include/curand_mtgp32.h'
  '/usr/local/cuda/include/curand_philox4x32_x.h'
  '/usr/local/cuda/include/curand_globals.h'
  '/usr/local/cuda/include/curand_uniform.h'
  '/usr/local/cuda/include/curand_normal.h'
  '/usr/local/cuda/include/curand_normal_static.h'
  '/usr/local/cuda/include/curand_lognormal.h'
  '/usr/local/cuda/include/curand_poisson.h'
  '/usr/local/cuda/include/curand_discrete2.h'.
Target //tensorflow/cc:tutorials_example_trainer failed to build

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 21 (12 by maintainers)

Most upvoted comments

Adding

cxx_builtin_include_directory: "/usr/local/cuda/targets/x86_64-linux/include"

to file tensorflow/third_party/gpus/crosstool/CROSSTOOL

seem to be a workaround.

+12

ghost on Mar 5, 2016

@kirilg You save my day. In my case, adding cxx_builtin_include_directory: "/usr/local/cuda-7.5/include" to third_party/gpus/crosstool/CROSSTOOL works, but cxx_builtin_include_directory: "/usr/local/cuda/include" doesn’t.

Seems that the building system is not as smart as cmake.

ZiangYan on Jul 27, 2016

Hey guys I had the same problem and fixed it by:

Uninstalling and re-installing gcc and g++
Removing ~/.cache/bazel

I also restarted the machine but I think (2) was the one that really fixed it for me.

andrewor14 on May 15, 2019

@damienmg I’m having the same issue with TensorFlow Serving compiled with cuda. Adding cxx_builtin_include_directory: "/usr/local/cuda-7.0/include" to the CROSSTOOL file works, but changing it to "/usr/local/cuda/include (symlink) fails with the same error above (this rule is missing dependency declarations…) The path that works is user specific, so we can’t add it as is.

I tried to auto-configure using the link you provided, and got stuck in other places. The relevant part of the error message including the command it generated is: /usr/bin/gcc -shared -o bazel-out/local-opt/bin/tensorflow_serving/resources/libresource_values.so -no-canonical-prefixes -B/usr/bin -pass-exit-codes '-Wl,--build-id=md5' '-Wl,--hash-style=gnu' -Wl,rpath,/usr/local/cuda-7.0/lib -L/usr/local/cuda-7.0/lib -Wl,rpath,/usr/local/cuda-7.0/lib64 -L/usr/local/cuda-7.0/lib64 -Wl,--gc-sections -Wl,@bazel-out/local-opt/bin/tensorflow_serving/resources/libresource_values.so-2.params): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1. /usr/bin/ld: cannot find rpath: No such file or directory /usr/bin/ld: cannot find /usr/local/cuda-7.0/lib: File format not recognized /usr/bin/ld: cannot find rpath: No such file or directory /usr/bin/ld: cannot find /usr/local/cuda-7.0/lib64: File format not recognized collect2: error: ld returned 1 exit status

note that it’s using /usr/bin/gcc but TensorFlow’s crosstool config uses clang/bin/crosstool_wrapper_driver_is_not_gcc. I only added load("@bazel_tools//tools/cpp:cc_configure.bzl", "cc_configure") cc_configure() and not sure if there’s a way to configure the tool_path for gcc since cc_configure() is the only public function and it takes no parameters.

kirilg on Apr 7, 2016

Oh gosh, sorry @vrv, I totally missed that message.

Yes missing cxx_builtin_include_directory is the usual suspect when you have such message.

Note that I am rolling out a mechanism in Bazel to auto-configure those include dir (see https://groups.google.com/forum/#!msg/bazel-discuss/MSunz2ZUOq0/U5tE7uQLJQAJ), using it in TF would significantly reduce those bug reports

damienmg on Mar 5, 2016

@dbikard - You may have added your cxx_builtin_include_directory: entry to the wrong part of the config. Can you paste a snippet illustrating where in your CROSSTOOL config you added that entry to?

FYI, the CROSSTOOL config file is in text Protocol Buffer format and is parsed into a CToolchain Protocol Buffer message. Here is the message definition, which serves as the schema for the CROSSTOOL config file, and here is where the cxx_builtin_include_directory field is in the proto: https://github.com/bazelbuild/bazel/blob/master/src/main/protobuf/crosstool_config.proto#L370

davidzchen on Jul 26, 2016

@ZiangYan cxx_builtin_include_directory: “/usr/local/cuda-7.5/include” works for me too!

slowbull on Jul 21, 2016