tensorflow: build failed with cuda 10.2
Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template
System information
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): 18.04 x64
- Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A
- TensorFlow installed from (source or binary): source
- TensorFlow version: 2.0.0
- Python version: 3.7.5 x64
- Installed using virtualenv? pip? conda?: N/A
- Bazel version (if compiling from source): 1.1.0 x64
- GCC/Compiler version (if compiling from source): 7.4.0
- CUDA/cuDNN version: 10.2 / 7.6.5
- GPU model and memory: GTX1080Ti GDDR5X 11GB
Describe the problem
Provide the exact sequence of commands / steps that you executed before running into the problem bazel build
Any other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
ERROR: /home/vai/.cache/bazel/_bazel_vai/964c7018fd2d0d2d2cf98e15f592d3c8/external/nccl_archive/BUILD.bazel:53:1: fatbinary external/nccl_archive/device_dlink_hdrs.fatbin failed (Exit 1)
fatbinary fatal : Unknown option '-bin2c-path'
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /home/vai/repo/tensorflow/tensorflow/python/tools/BUILD:81:1 fatbinary external/nccl_archive/device_dlink_hdrs.fatbin failed (Exit 1)
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 29 (14 by maintainers)
Commits related to this issue
- Update build_defs.bzl.tpl * delete "--bin2c-path=%s" % bin2c.dirname * ref https://github.com/tensorflow/tensorflow/issues/34429#issuecomment-557408498 — committed to joapolarbear/tensorflow by joapolarbear 4 years ago
I was able to successfully compile on v2.0.0 with cuda-10.2.
I had the same error. My workaround was to remove the line
"--bin2c-path=%s" % bin2c.dirname,from the filethird_party/nccl/build_defs.bzl.tplseems the ‘-bin2c-path’ option is not required anymore.
Can you downgrade to cuda 10.0 and see how it progress.Thanks!
For the record, it was fixed in this commit:
commit 67edc16326d6328e7ef096e1b06f81dae1bfb816 Author: Sami skama@nvidia.com Date: Fri Dec 6 08:49:20 2019
@alanpurple
Yes! Windows 10, grpcio==1.26.0rc1, Python 3.8 and Cuda 10.1 compiles and runs perfectly fine now without any tweaks with tensorflow v2.1.0-rc0.
It is worth mentioning that although tensorflow works now fine with Python 3.8, there are closely related projects like Hugging Face Transformers with it’s own upstream and downstream dependencies waiting for the wheels.
The scope of this issue was cuda 10.2, but for now I’d still stay on cuda 10.0 or 10.1. Looking at the source and recent changes, simply removing bin2c could have it’s own less visible consequences not worth the risk.
Not really. Release branches only get security patches and build fixes on the same toolchain. This seems to change cuda toolchain, which I don’t think is low risk enough to warrant cherry-picking
also failed build of version 1.15.0 with cuda-10.2
fatbinary fatal : Unknown option ‘-bin2c-path’
here is a comparison of “fatbinary --help” output from cuda-10.0 and cuda-10.2
https://pastebin.com/E1SXpuqh
seems that there is no ‘-bin2c-path’ option anymore