tensorflow: Sym-links in nvcc C compiler path causes "undeclared inclusion(s) in rule '//tensorflow/core:lib_hash_crc32c_accelerate_internal'"

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Redhat Enterprise Linux 7.x
TensorFlow installed from (source or binary): source
TensorFlow version: 1.14.0
Python version: 3.7.4
Installed using virtualenv? pip? conda?: virtualenv
Bazel version (if compiling from source): 0.24.1
GCC/Compiler version (if compiling from source): 4.9.2
CUDA/cuDNN version: 10.0.130/7.4.2.24
GPU model and memory: Tesla P100

Describe the problem

We have compilers installed in /shared/ucl/apps/gcc/4.9.2. Unfortunately /shared is a sym-link to /lustre/shared. Attempting to build Tensorflow with Cuda support results in this sym-link being inconsistently de-referenced meaning that some rules refer to the /shared location while the system include paths refer to /lustre/shared. This seems to be an issue only with the C compiler used by nvcc, not with the one used to build the rest of the code.

Telling the configure script to use /lustre/shared/ucl/apps/gcc/4.9.2/bin/gcc as the nvcc c compiler works around the problem but is less than ideal as this path is different on different clusters. It culd also be an issue if a user on a mult-user system uses their own install of GCC in /homewhere /home is a sym-link. This appears to be a bug in the way the configure script and/or bazel deals with sym-links.

Provide the exact sequence of commands / steps that you executed before running into the problem

The script at https://github.com/owainkenwayucl/install-scripts/blob/master/scripts/tensorflow/tensorflow-1.14.0-py37-gpu_install

But effectively:

CONFIG_OPTS="--config=cuda --copt=-march=broadwell --cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0 --copt=-O3"
export TF_CUDA_PATHS=/shared/ucl/apps/cuda/10.0.130/gnu-4.9.2,/shared/ucl/apps/cudnn/7.4.2.24/10.0/cuda
./configure

bazel build --verbose_failures $CONFIG_OPTS //tensorflow/tools/pip_package:build_pip_package

Answering “yes” to build with cuda and selecting defaults for compute capability, nccl etc.

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 20 (11 by maintainers)

Commits related to this issue

Patch to solve sym-link problem https://github.com/tensorflow/tensorflow/issues/32776 — committed to owainkenwayucl/install-scripts by owainkenwayucl 5 years ago

Most upvoted comments

Having looked at configure.py I believe it’s as simple as adding:

gcc_host_compiler_path = os.path.realpath(gcc_host_compiler_path)

at line 842 in the current version, and 831 in 1.14.0.

I’m happy to do this and do a pull request if you prefer?

owainkenwayucl on Oct 4, 2019

Ok, that’s weird - our CUDA includes are also under the sym-link and we don’t get that problem. What error do you get?

ERROR: /cache/home/jz748/.cache/bazel/_bazel_jz748/9a5804611367d5a3465d7ad3decb2864/external/nccl_archive/BUILD.bazel:67:1: undeclared inclusion(s) in rule '@nccl_archive//:device_lib':
this rule is missing dependency declarations for the following files included by 'external/nccl_archive/collectives/device/max_all_gather.cu.cc':       '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/cuda_runtime.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/crt/host_config.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/builtin_types.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/device_types.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/crt/host_defines.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/driver_types.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/vector_types.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/surface_types.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/texture_types.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/library_types.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/channel_descriptor.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/cuda_runtime_api.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/cuda_device_runtime_api.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/driver_functions.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/vector_functions.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/vector_functions.hpp'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/crt/common_functions.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/crt/math_functions.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/crt/math_functions.hpp'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/cuda_surface_types.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/cuda_texture_types.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/crt/device_functions.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/crt/device_functions.hpp'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/device_atomic_functions.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/device_atomic_functions.hpp'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/crt/device_double_functions.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/crt/device_double_functions.hpp'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/sm_20_atomic_functions.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/sm_20_atomic_functions.hpp'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/sm_32_atomic_functions.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/sm_32_atomic_functions.hpp'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/sm_35_atomic_functions.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/sm_60_atomic_functions.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/sm_60_atomic_functions.hpp'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/sm_20_intrinsics.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/sm_20_intrinsics.hpp'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/sm_30_intrinsics.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/sm_30_intrinsics.hpp'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/sm_32_intrinsics.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/sm_32_intrinsics.hpp'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/sm_35_intrinsics.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/sm_61_intrinsics.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/sm_61_intrinsics.hpp'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/crt/sm_70_rt.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/crt/sm_70_rt.hpp'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/surface_functions.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/texture_fetch_functions.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/texture_indirect_functions.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/surface_indirect_functions.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/device_launch_parameters.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/cuda_fp16.h'
  '/scratch/jz748/cuda-10.1/targets/x86_64-linux/include/cuda_fp16.hpp'

Do you think it is a different issue? If so, I’ll create another issue.

njzjz on Oct 16, 2019

Build failure:

ERROR: /dev/shm/tensorflow/tmp.6ooBek4IW8/tensorflow/tensorflow/core/BUILD:2499:1: undeclared inclusion(s) in rule '//tensorflow/core:lib_hash_crc32c_accelerate_internal':
this rule is missing dependency declarations for the following files included by 'tensorflow/core/lib/hash/crc32c_accelerate.cc':
  '/shared/ucl/apps/gcc/4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include/stddef.h'
  '/shared/ucl/apps/gcc/4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include/stdint.h'
  '/shared/ucl/apps/gcc/4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include/nmmintrin.h'
  '/shared/ucl/apps/gcc/4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include/smmintrin.h'
  '/shared/ucl/apps/gcc/4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include/tmmintrin.h'
  '/shared/ucl/apps/gcc/4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include/pmmintrin.h'
  '/shared/ucl/apps/gcc/4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include/emmintrin.h'
  '/shared/ucl/apps/gcc/4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include/xmmintrin.h'
  '/shared/ucl/apps/gcc/4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include/mmintrin.h'
  '/shared/ucl/apps/gcc/4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include/mm_malloc.h'
  '/shared/ucl/apps/gcc/4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/include/popcntintrin.h'
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 108.730s, Critical Path: 1.51s
INFO: 42 processes: 42 local.
FAILED: Build did NOT complete successfully

owainkenwayucl on Sep 24, 2019