tensorflow: Failing to build version 2.14.0 with numa
Issue type
Build/Install
Have you reproduced the bug with TensorFlow Nightly?
No
Source
source
TensorFlow version
2.14.0
Custom code
No
OS platform and distribution
linux rocky8
Mobile device
No response
Python version
3.8.7
Bazel version
6.1.0
GCC/compiler version
gcc 11.3.0
CUDA/cuDNN version
cuda 11.8.0
GPU model and memory
No response
Current behavior?
tensorflow v2.14.0 seems to fail configuring when adding --config=numa
bazel' '--nohome_rc' '--nosystem_rc' '--output_user_root=/tmp/spackweoy5szc' 'build' '--color=no' '--jobs=10' '--config=opt' '--verbose_failures' '--config=dynamic_kernels' '--config=cuda' '--config=noaws' '--config=nogcp' '--config=nohdfs' '--config=numa' '--config=v2' '//tensorflow/tools/pip_package:build_pip_package'
...
ERROR: Config value 'numa' is not defined in any .rc file
Standalone code to reproduce the issue
bazel' '--nohome_rc' '--nosystem_rc' '--output_user_root=/tmp/spackweoy5szc' 'build' '--color=no' '--jobs=10' '--config=opt' '--verbose_failures' '--config=dynamic_kernels' '--config=cuda' '--config=noaws' '--config=nogcp' '--config=nohdfs' '--config=numa' '--config=v2' '//tensorflow/tools/pip_package:build_pip_package'
Relevant log output
No response
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 17 (7 by maintainers)
The above mentioned commit is the cause of the issue, not the fix.
Yes. The
--config=numaflag was used as a convenient shorthand for grouping options related to NUMA support, initially containing just the--define=with_numa_support=trueoption. If we anticipate adding more NUMA-specific options in the future, it would make sense to assess the potential impact on users and determine if reintroducing the--config=numaflag would improve the overall experience for a substantial number of users.Hi @learning-to-play ,
Yes, i had tried also 2.16.
The issue is caused by this commit https://github.com/tensorflow/tensorflow/commit/ffd06c0782a31766b15b6730aac48d124cdf4a11 which removes the numa config (and many others) from the bazelrc. If you diff the bazelrc from that commit onwards you’ll see it has never been put back.
Now it would be interesting to understand why numa (and the other options) has been removed from the bazelrc. Is it because it is always implicitly active? Is it automatically detected? Was the corresponding feature removed completely? As it is still present in the configure.py, making the two files incoherent with each other, this is pretty confusing and the comment of the mentioned commit does not explain why they are deemed no longer relevant.
@kanglant Tagging you here because i found this commit which removes the numa config: https://github.com/tensorflow/tensorflow/commit/ffd06c0782a31766b15b6730aac48d124cdf4a11
This command:
bazel --nohome_rc --nosystem_rc --output_user_root=/tmp/spackd8v53qm6 build --color=no --jobs=16 --config=opt --verbose_failures --config=dynamic_kernels --config=cuda --config=noaws --config=nogcp --config=nohdfs --config=numa --config=v2 //tensorflow/tools/pip_package:build_pip_packageproduces:ERROR: Config value 'numa' is not defined in any .rc fileThis command:
bazel --nohome_rc --nosystem_rc --output_user_root=/tmp/spackd8v53qm6 build --color=no --jobs=16 --config=opt --verbose_failures --config=dynamic_kernels --config=cuda --config=noaws --config=nogcp --config=nohdfs --config=v2 //tensorflow/tools/pip_package:build_pip_package(which is the same without--config=numa) passes the configuration stage.Trying to grep for
numain the sources directory yelds:Up to version 2.13.1, this was present in
.bazelrcand is no longer present in 2.14Does this means that
--config=numais no longer supported from version 2.14 and that the following line should be removed fromconfigure.py?(it’s still present in 2.15 but same stuff missing in
.bazelrc)I thinks all this is independent from python or compiler type/version as we are still in bazel configuration and haven’t started building anything yet.