tensorflow: Building TensorFlow 1.2 from source results in catastrophic error?

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04 LTS
  • TensorFlow installed from (source or binary): Source
  • TensorFlow version (use command below): 1.2
  • Bazel version (if compiling from source): 4.5.1
  • CUDA/cuDNN version: 8.0/5.1
  • GPU model and memory: GTX 860M
  • Exact command to reproduce: bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package --verbose_failures

Describe the problem

I have successfully installed TensorFlow 1.2 by upgrading the 1.1 version via Pip, but I wanted to install 1.2 from source to get the full benefits. However, strangely I received this error when building the pip package from bazel.

INFO: From Compiling tensorflow/core/kernels/batch_norm_op_gpu.cu.cc: external/eigen_archive/unsupported/Eigen/CXX11/src/Tensor/TensorBroadcasting.h(271): internal error: assertion failed at: “/dvs/p4/build/sw/rel/gpu_drv/r361/r361_00/drivers/compiler/edg/EDG_4.10/src/folding.c”, line 9819

1 catastrophic error detected in the compilation of "/tmp/tmpxft_00007cc2_00000000-7_batch_norm_op_gpu.cu.cpp1.ii".
Compilation aborted.
Aborted (core dumped)
ERROR: /home/kwotsin/tensorflow_official/tensorflow/tensorflow/core/kernels/BUILD:2730:1: output 'tensorflow/core/kernels/_objs/batch_norm_op_gpu/tensorflow/core/kernels/batch_norm_op_gpu.cu.pic.o' was not created.
ERROR: /home/kwotsin/tensorflow_official/tensorflow/tensorflow/core/kernels/BUILD:2730:1: not all outputs were created or valid.
Target //tensorflow/tools/pip_package:build_pip_package failed to build

I’ve not seen this error before - is it because I did not git check out the 1.2 version but instead the master branch?

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 18 (9 by maintainers)

Most upvoted comments

@sugartom I also running an experiment and manually copied the cuda-repo-l4t-8-0-local_8.0.64-1_arm64.deb package to my TX1. sudo dpkg -i cuda-repo-l4t-8-0-local_8.0.64-1_arm64.deb && sudo apt-get update && sudo apt-get upgrade

I’ll try rebuilding tensorflow v1.1.0 throughout the day and post the results.