sonnet: internal compiler error: in tsubst_copy, at cp/pt.c:13970 using gcc 6.2

I have been unable to install Sonnet because of a compiler error. I tried to find if this was a bug with gcc and I found similar entries, however updating gcc did not work. This error happened to me using verions 5.3 and 6.2 of gcc (output of the latter below).

abermea@host:~/Projects/Sonnet/sonnet$ bazel build --config=opt :install
.
WARNING: Config values are not defined in any .rc file: opt
WARNING: /home/abermea/.cache/bazel/_bazel_abermea/ca99c09533717eb94266b31b726808fb/external/org_tensorflow/tensorflow/workspace.bzl:72:5: tf_repo_name was specified to tf_workspace but is no longer used and will be removed in the future.
INFO: Found 1 target...
ERROR: /home/abermea/Projects/Sonnet/sonnet/sonnet/cc/kernels/BUILD:19:1: C++ compilation of rule '//sonnet/cc/kernels:resampler_op' failed: Process exited with status 1 [sandboxed].
sonnet/cc/kernels/resampler_op.cc: In instantiation of 'deepmind::tensorflow::sonnet::functor::ResamplerGrad2DFunctor<Eigen::ThreadPoolDevice, T>::operator()(tensorflow::OpKernelContext*, const CPUDevice&, const T*, const T*, const T*, T*, T*, int, int, int, int, int)::<lambda(int, int)>::<lambda(int, int, int, T)> [with T = double]':
sonnet/cc/kernels/resampler_op.cc:269:23:   required from 'struct deepmind::tensorflow::sonnet::functor::ResamplerGrad2DFunctor<Eigen::ThreadPoolDevice, T>::operator()(tensorflow::OpKernelContext*, const CPUDevice&, const T*, const T*, const T*, T*, T*, int, int, int, int, int)::<lambda(int, int)> [with T = double]::<lambda(int, int, int, double)>'
sonnet/cc/kernels/resampler_op.cc:272:9:   required from 'deepmind::tensorflow::sonnet::functor::ResamplerGrad2DFunctor<Eigen::ThreadPoolDevice, T>::operator()(tensorflow::OpKernelContext*, const CPUDevice&, const T*, const T*, const T*, T*, T*, int, int, int, int, int)::<lambda(int, int)> [with T = double]'
sonnet/cc/kernels/resampler_op.cc:317:38:   required from 'struct deepmind::tensorflow::sonnet::functor::ResamplerGrad2DFunctor<Eigen::ThreadPoolDevice, T>::operator()(tensorflow::OpKernelContext*, const CPUDevice&, const T*, const T*, const T*, T*, T*, int, int, int, int, int) [with T = double; deepmind::tensorflow::sonnet::CPUDevice = Eigen::ThreadPoolDevice]::<lambda(int, int)>'
sonnet/cc/kernels/resampler_op.cc:338:5:   required from 'void deepmind::tensorflow::sonnet::functor::ResamplerGrad2DFunctor<Eigen::ThreadPoolDevice, T>::operator()(tensorflow::OpKernelContext*, const CPUDevice&, const T*, const T*, const T*, T*, T*, int, int, int, int, int) [with T = double; deepmind::tensorflow::sonnet::CPUDevice = Eigen::ThreadPoolDevice]'
sonnet/cc/kernels/resampler_op.cc:407:51:   required from 'void deepmind::tensorflow::sonnet::ResamplerGradOp<Device, T>::Compute(tensorflow::OpKernelContext*) [with Device = Eigen::ThreadPoolDevice; T = double]'
sonnet/cc/kernels/resampler_op.cc:443:1:   required from here
sonnet/cc/kernels/resampler_op.cc:239:47: internal compiler error: in tsubst_copy, at cp/pt.c:13970
     const int data_batch_stride = data_height * data_width * data_channels;
                                   ~~~~~~~~~~~~^~~~~~~~~~~~
0x60e858 tsubst_copy
	../../src/gcc/cp/pt.c:13970
0x60efb1 tsubst_copy_and_build(tree_node*, tree_node*, int, tree_node*, bool, bool)
	../../src/gcc/cp/pt.c:17067
0x6102b8 tsubst_copy_and_build(tree_node*, tree_node*, int, tree_node*, bool, bool)
	../../src/gcc/cp/pt.c:16252
0x6102b8 tsubst_copy_and_build(tree_node*, tree_node*, int, tree_node*, bool, bool)
	../../src/gcc/cp/pt.c:16252
0x60aa58 tsubst_expr(tree_node*, tree_node*, int, tree_node*, bool)
	../../src/gcc/cp/pt.c:15876
0x60bab5 tsubst_init
	../../src/gcc/cp/pt.c:13916
0x60e8e6 tsubst_copy
	../../src/gcc/cp/pt.c:14109
0x60efb1 tsubst_copy_and_build(tree_node*, tree_node*, int, tree_node*, bool, bool)
	../../src/gcc/cp/pt.c:17067
0x6102d6 tsubst_copy_and_build(tree_node*, tree_node*, int, tree_node*, bool, bool)
	../../src/gcc/cp/pt.c:16253
0x6102b8 tsubst_copy_and_build(tree_node*, tree_node*, int, tree_node*, bool, bool)
	../../src/gcc/cp/pt.c:16252
0x6102b8 tsubst_copy_and_build(tree_node*, tree_node*, int, tree_node*, bool, bool)
	../../src/gcc/cp/pt.c:16252
0x60f2bf tsubst_copy_and_build(tree_node*, tree_node*, int, tree_node*, bool, bool)
	../../src/gcc/cp/pt.c:16285
0x60fa64 tsubst_copy_and_build(tree_node*, tree_node*, int, tree_node*, bool, bool)
	../../src/gcc/cp/pt.c:16390
0x60aa58 tsubst_expr(tree_node*, tree_node*, int, tree_node*, bool)
	../../src/gcc/cp/pt.c:15876
0x609686 tsubst_expr(tree_node*, tree_node*, int, tree_node*, bool)
	../../src/gcc/cp/pt.c:15192
0x60a903 tsubst_expr(tree_node*, tree_node*, int, tree_node*, bool)
	../../src/gcc/cp/pt.c:15364
0x609980 tsubst_expr(tree_node*, tree_node*, int, tree_node*, bool)
	../../src/gcc/cp/pt.c:15344
0x60a8bc tsubst_expr(tree_node*, tree_node*, int, tree_node*, bool)
	../../src/gcc/cp/pt.c:15178
0x60a903 tsubst_expr(tree_node*, tree_node*, int, tree_node*, bool)
	../../src/gcc/cp/pt.c:15364
0x60a8bc tsubst_expr(tree_node*, tree_node*, int, tree_node*, bool)
	../../src/gcc/cp/pt.c:15178
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <file:///usr/share/doc/gcc-6/README.Bugs> for instructions.
Use --strategy=CppCompile=standalone to disable sandboxing for the failing actions.
Target //:install failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 34.787s, Critical Path: 11.86s

==========================================

abermea@host:~/Projects/Sonnet/sonnet$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/6/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 6.2.0-3ubuntu11~16.04' --with-bugurl=file:///usr/share/doc/gcc-6/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-6 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-6-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-6-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-6-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 6.2.0 20160901 (Ubuntu 6.2.0-3ubuntu11~16.04)

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 23 (4 by maintainers)

Most upvoted comments

@kosklain: Yes, I am referring to the submodule and these are steps which worked for me together with the patch #6:

cd sonnet/tensorflow
git checkout r1.1
./configure
bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
sudo -H pip uninstall tensorflow
sudo -H pip install /tmp/tensorflow_pkg/tensorflow-1.1.0rc1-cp27-cp27mu-linux_x86_64.whl
cd ../
bazel build --config=opt //:install
bazel-bin/install /tmp/sonnet_pkg
sudo -H pip uninstall sonnet
sudo -H pip install /tmp/sonnet_pkg/sonnet-1.0-py2-none-any.whl

roman3017 on Apr 9, 2017

Got it working after using r1.1 branch for tensorflow, recompiled and installed from source. The master branch for tensorflow did not work for me. Also had to augment LD_LIBRARY_PATH and use the patch https://github.com/deepmind/sonnet/issues/6, both are described above.

>>> snt.resampler(tf.constant([0.]), tf.constant([0.]))
Tensor("resampler/Resampler:0", shape=(1,), dtype=float32)

~/projects/tensor/sonnet/sonnet/examples$ python rnn_shakespeare_test.py 
2017-04-08 16:08:26.416213: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2017-04-08 16:08:26.416596: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties: 
name: GeForce GT 750M
major: 3 minor: 0 memoryClockRate (GHz) 0.967
pciBusID 0000:02:00.0
Total memory: 1.95GiB
Free memory: 1.10GiB
2017-04-08 16:08:26.416616: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0 
2017-04-08 16:08:26.416622: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0:   Y 
2017-04-08 16:08:26.416633: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GT 750M, pci bus id: 0000:02:00.0)
..
----------------------------------------------------------------------
Ran 2 tests in 59.776s

OK

roman3017 on Apr 9, 2017

Ideally we want to not have to compile tensorflow (given that it can get complex and might take a lot of time) even with new compilers. We are resolving the issues for people that compile sonnet with gcc 5x, partly solved by using:

bazel build --config=opt --copt="-D_GLIBCXX_USE_CXX11_ABI=0" :install

For now I think it’s safe to say that this issue will be resolved when #25 is resolved. Marking as duplicate to track progress there.

adria-p on May 25, 2017

I’ve pushed the image to docker hub in case anyone wants to use it for experimentation

docker run -p 8888:8888 -it davemssavage/sonnet

davemssavage on Apr 9, 2017