tensorflow: Failing to build version 2.14.0 with numa

Issue type

Build/Install

Have you reproduced the bug with TensorFlow Nightly?

No

Source

source

TensorFlow version

2.14.0

Custom code

No

OS platform and distribution

linux rocky8

Mobile device

No response

Python version

3.8.7

Bazel version

6.1.0

GCC/compiler version

gcc 11.3.0

CUDA/cuDNN version

cuda 11.8.0

GPU model and memory

No response

Current behavior?

tensorflow v2.14.0 seems to fail configuring when adding --config=numa

bazel' '--nohome_rc' '--nosystem_rc' '--output_user_root=/tmp/spackweoy5szc' 'build' '--color=no' '--jobs=10' '--config=opt' '--verbose_failures' '--config=dynamic_kernels' '--config=cuda' '--config=noaws' '--config=nogcp' '--config=nohdfs' '--config=numa' '--config=v2' '//tensorflow/tools/pip_package:build_pip_package'
...
ERROR: Config value 'numa' is not defined in any .rc file

Standalone code to reproduce the issue

bazel' '--nohome_rc' '--nosystem_rc' '--output_user_root=/tmp/spackweoy5szc' 'build' '--color=no' '--jobs=10' '--config=opt' '--verbose_failures' '--config=dynamic_kernels' '--config=cuda' '--config=noaws' '--config=nogcp' '--config=nohdfs' '--config=numa' '--config=v2' '//tensorflow/tools/pip_package:build_pip_package'

Relevant log output

No response

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 17 (7 by maintainers)

Most upvoted comments

The above mentioned commit is the cause of the issue, not the fix.

Yes. The --config=numa flag was used as a convenient shorthand for grouping options related to NUMA support, initially containing just the --define=with_numa_support=true option. If we anticipate adding more NUMA-specific options in the future, it would make sense to assess the potential impact on users and determine if reintroducing the --config=numa flag would improve the overall experience for a substantial number of users.

Hi @learning-to-play ,

Yes, i had tried also 2.16.

The issue is caused by this commit https://github.com/tensorflow/tensorflow/commit/ffd06c0782a31766b15b6730aac48d124cdf4a11 which removes the numa config (and many others) from the bazelrc. If you diff the bazelrc from that commit onwards you’ll see it has never been put back.

Now it would be interesting to understand why numa (and the other options) has been removed from the bazelrc. Is it because it is always implicitly active? Is it automatically detected? Was the corresponding feature removed completely? As it is still present in the configure.py, making the two files incoherent with each other, this is pretty confusing and the comment of the mentioned commit does not explain why they are deemed no longer relevant.

@kanglant Tagging you here because i found this commit which removes the numa config: https://github.com/tensorflow/tensorflow/commit/ffd06c0782a31766b15b6730aac48d124cdf4a11

This command: bazel --nohome_rc --nosystem_rc --output_user_root=/tmp/spackd8v53qm6 build --color=no --jobs=16 --config=opt --verbose_failures --config=dynamic_kernels --config=cuda --config=noaws --config=nogcp --config=nohdfs --config=numa --config=v2 //tensorflow/tools/pip_package:build_pip_package produces: ERROR: Config value 'numa' is not defined in any .rc file

This command: bazel --nohome_rc --nosystem_rc --output_user_root=/tmp/spackd8v53qm6 build --color=no --jobs=16 --config=opt --verbose_failures --config=dynamic_kernels --config=cuda --config=noaws --config=nogcp --config=nohdfs --config=v2 //tensorflow/tools/pip_package:build_pip_package (which is the same without --config=numa) passes the configuration stage.

Trying to grep for numa in the sources directory yelds:

$ grep  numa *
configure.py:  config_info_line('numa', 'Build with NUMA support.')
RELEASE.md:    *   Removing the `experimental_numa_aware` option from `tf.data.Options`.
$ grep numa .bazel*

Up to version 2.13.1, this was present in .bazelrc and is no longer present in 2.14

# Options extracted from configure script
build:numa --define=with_numa_support=true

Does this means that --config=numa is no longer supported from version 2.14 and that the following line should be removed from configure.py?

 config_info_line('numa', 'Build with NUMA support.')

(it’s still present in 2.15 but same stuff missing in .bazelrc)

I thinks all this is independent from python or compiler type/version as we are still in bazel configuration and haven’t started building anything yet.