tensorflow: cannot build TensorFLow with --config=dbg

when building opensource TensorFlow with

bazel build --config=dbg --config=cuda --cxxopt=“-D_GLIBCXX_USE_CXX11_ABI=0” //tensorflow/tools/pip_package:build_pip_package

(for SM 7.0 only)

The build dies at link time with:

ERROR: /home/baarts/tensorflow-GH/tensorflow/python/BUILD:3373:24: Linking of rule '//tensorflow/python:_pywrap_tensorflow_internal.so' failed (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc @bazel-out/k8-dbg/bin/tensorflow/python/_pywrap_tensorflow_internal.so-2.params bazel-out/k8-dbg/bin/external/llvm-project/llvm/libScalar.a(AnnotationRemarks.pic.o):(.debug_aranges+0x6): relocation truncated to fit: R_X86_64_32 against .debug_info’ bazel-out/k8-dbg/bin/external/llvm-project/llvm/libScalar.a(BDCE.pic.o)😦.debug_aranges+0x6): relocation truncated to fit: R_X86_64_32 against .debug_info' bazel-out/k8-dbg/bin/external/llvm-project/llvm/libScalar.a(CallSiteSplitting.pic.o):(.debug_aranges+0x6): relocation truncated to fit: R_X86_64_32 against .debug_info’ bazel-out/k8-dbg/bin/external/llvm-project/llvm/libScalar.a(ConstantHoisting.pic.o)😦.debug_aranges+0x6): relocation truncated to fit: R_X86_64_32 against .debug_info' bazel-out/k8-dbg/bin/external/llvm-project/llvm/libScalar.a(ConstraintElimination.pic.o):(.debug_aranges+0x6): relocation truncated to fit: R_X86_64_32 against .debug_info’ bazel-out/k8-dbg/bin/external/llvm-project/llvm/libScalar.a(CorrelatedValuePropagation.pic.o)😦.debug_aranges+0x6): relocation truncated to fit: R_X86_64_32 against .debug_info' bazel-out/k8-dbg/bin/external/llvm-project/llvm/libScalar.a(DCE.pic.o):(.debug_aranges+0x6): relocation truncated to fit: R_X86_64_32 against .debug_info’ bazel-out/k8-dbg/bin/external/llvm-project/llvm/libScalar.a(DeadStoreElimination.pic.o)😦.debug_aranges+0x6): relocation truncated to fit: R_X86_64_32 against .debug_info' bazel-out/k8-dbg/bin/external/llvm-project/llvm/libScalar.a(DivRemPairs.pic.o):(.debug_aranges+0x6): relocation truncated to fit: R_X86_64_32 against .debug_info’ bazel-out/k8-dbg/bin/external/llvm-project/llvm/libScalar.a(EarlyCSE.pic.o)😦.debug_aranges+0x6): relocation truncated to fit: R_X86_64_32 against .debug_info' bazel-out/k8-dbg/bin/external/llvm-project/llvm/libScalar.a(FlattenCFGPass.pic.o):(.debug_aranges+0x6): additional relocation overflows omitted from the output collect2: error: ld returned 1 exit status

Adding -mcmodel=large makes no difference, as the overflow is in a debug section. I tried -gdwarf64 which is not supported by gcc

some platform info:

root@7fe23091cb5b:/opt/tensorflow# gcc --version
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

root@7fe23091cb5b:/opt/tensorflow# uname -a
Linux 7fe23091cb5b 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 43 (22 by maintainers)

Commits related to this issue

Fix build failure with --config=dbg. Before, the build would fail with errors such as: "relocation truncated to fit: R_X86_64_32 against .debug_info'". The issue was the debug info was too large. I b... — committed to tensorflow/tensorflow by reedwm 3 years ago
Add documentation on the --config=dbg option. Resolves https://github.com/tensorflow/tensorflow/issues/48919. PiperOrigin-RevId: 380915403 — committed to tensorflow/docs by reedwm 3 years ago
[create-pull-request] automated change — committed to jessecantu/jamstack-cfp by github-actions[bot] 4 years ago

Most upvoted comments

so I added --distinct_host_configuration=false to the bazel build just to see. Build was so much faster, as only the build is done. So far, seems like stuff is working.

Is this related to this debug build or it is going to impact also regular build?

While i have not verified, this should not be debug related

This could be interesting /cc @angerson @perfinion

bhack on Jun 12, 2021

I plan on adding the following to .bazelrc:

build:dbg --per_file_copt=+.*,-tensorflow.*@-g0
build:dbg --per_file_copt=+tensorflow/core/kernels.*@-g0

.bazelrc already has the line build:dbg -c dbg, so I think adding -O0 -g is unnecessary.

reedwm on Jun 10, 2021

I can reproduce on Ubuntu 20.04. I am not familiar with how debugging symbols work, but based on what you said it seems they are exceeding 2 GB which is causing the issue? You mention -gdwarf64, which is in gcc 11.1 but not in gcc 9.3, which is what Ubuntu 20.04 uses.

/CC @chsigg any ideas on what to do here? If it’s impossible to support compiling all files with debug symbols, perhaps we should provide a config option and instructions on only building a subset of files with debugging symbols.

reedwm on May 6, 2021