tensorflow: Bazel Build Fails with "undeclared inclusion(s) in rule '@nccl_archive//:nccl'"

System information

The environment used is: tf_env.txt

The branch downloaded is tensorflow-master branch

When I run the configure script the configuration is .tf_configure.bazelrc

Describe the problem

When I run

bazel build --config=opt --config=cuda --config=mkl //tensorflow/tools/pip_package:build_pip_package

The build fails with the error

this rule is missing dependency declarations for the following files included by 'external/nccl_archive/src/core.cu.cc':
  '/usr/lib/gcc/x86_64-linux-gnu/4.9/include-fixed/limits.h'
  '/usr/lib/gcc/x86_64-linux-gnu/4.9/include-fixed/syslimits.h'
  '/usr/lib/gcc/x86_64-linux-gnu/4.9/include/stddef.h'
  '/usr/lib/gcc/x86_64-linux-gnu/4.9/include/stdarg.h'
Target //tensorflow/tools/pip_package:build_pip_package failed to build
FAILED: Build did NOT complete successfully

but the files are present:

ls /usr/lib/gcc/x86_64-linux-gnu/4.9/include-fixed/limits.h
/usr/lib/gcc/x86_64-linux-gnu/4.9/include-fixed/limits.h

There appears to be a problem with bazel build with MKL enabled. I have been able to compile this with MKL disabled. MKL for Ubuntu should be supported, yes?

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 37 (19 by maintainers)

Commits related to this issue

Don't pass '-no-canonical-prefixes' when collecting builtin includes. It matches the way bazel's autoconf works and seems to be the right thing to do. This change should fix #14380. PiperOrigin-RevI... — committed to tensorflow/tensorflow by ilya-biryukov 6 years ago

Most upvoted comments

I got bazel to successfully complete but without MKL support by running

bazel clean --expunge

then hand-editing the configure file to .tf_configure.bazelrc then running

bazel build --config=opt --config=cuda --config=mkl //tensorflow/tools/pip_package:build_pip_package

So I guess the problem is with the configuration and/or compilation of MKL.

My reading of the documentation indicates that the necessary files for MKL for Ubuntu should be automatically downloaded. Is this not the case?

+15

johnsrude on Nov 8, 2017

my system version: centos7+gcc4.8.5+bazel 0.8.1 I find its tricky to fix this , just change the default gcc path when running configure from /bin/gcc to /usr/bin/gcc

+13

argman on Jan 26, 2018

I apologize but I am having a hard time understanding what the problem is, where the problem is, and what version it affects. Please resubmit and pay attention to the issue template (https://github.com/tensorflow/tensorflow/issues/new) . Please provide all the information it asks. Thank you.

bignamehyp on Nov 8, 2017

@argman
Thanks. It does work by " just change the default gcc path when running configure from /bin/gcc to /usr/bin/gcc"

forest1990 on Feb 3, 2018

I think this is a problem with the default toolchain in bazel. Will wait for a response from bazel team.

gunan on Dec 29, 2017

Met the similar problem too. I removed the path of bazel cache files in “/root/.cache/bazel”(change root to your user path) and rebuild, build completed successfully.

bidai541 on Jan 30, 2018

Github auto-referenced the issue above. I filed the issue in bazel repository here: https://github.com/bazelbuild/bazel/issues/4365

gunan on Jan 2, 2018

I have the same problem, can anyone help to check " undeclared inclusion(s) in rule ‘@nccl_archive//:nccl’ " issue? it will be really appreciated. 😃

System information: CentOS 7.3 gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16) Build label: 0.8.1- (@non-git) cuda 9.0.176 cudnn 7.0.5 tensorflow 1.4.0

Build command: bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package

Error message: INFO: Found 1 target… ERROR: /root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/external/nccl_archive/BUILD:33:1: undeclared inclusion(s) in rule ‘@nccl_archive//:nccl’: this rule is missing dependency declarations for the following files included by ‘external/nccl_archive/src/libwrap.cu.cc’: ‘/usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/limits.h’ ‘/usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/syslimits.h’ ‘/usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/stddef.h’ ‘/usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/stdarg.h’ Target //tensorflow/tools/pip_package:build_pip_package failed to build Use --verbose_failures to see the command lines of failed build steps. INFO: Elapsed time: 6.973s, Critical Path: 4.56s FAILED: Build did NOT complete successfully

beanliao on Dec 28, 2017

I know this may seem confrontational but it is not. You could add a message to configure.py. It would not stop people from adding --config=cuda but you could give a warning based on the options they pick. On the flip side, I am not sure how many people run into this problem. If they see MKL on the performance page it says --config=cuda does not work. If they see it in the config.py then a message would help them and you could be a hero. (not sarcasm, the internet is a hard place to read intent)

I would not expect people to see everything on the website as that is crazy. I did document the --config=cuda does not work. And if you are interested in some gains with various config options on CPU.

tfboyd on Nov 9, 2017