tensorflow: Eigen version bump breaks nightly AVX512 build with gcc 6.3
System information
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Debian 9.9
- Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy): N/A
- TensorFlow installed from (source or binary): source
- TensorFlow version: e432bf03931f4062f7c5e3a1553aff61a7294751
- Python version: 2.7.13
- Installed using virtualenv? pip? conda?: virtualenv
- Bazel version (if compiling from source): 0.24.1
- GCC/Compiler version (if compiling from source): (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
- CUDA/cuDNN version: N/A
- GPU model and memory: N/A
Describe the problem
In a GCE VM with AVX512 supported CPU:
bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package
Any other info / logs
In file included from /usr/lib/gcc/x86_64-linux-gnu/6/include/immintrin.h:59:0,
from external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/Core/util/ConfigureVectorization.h:318,
from external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/Core:22,
from external/eigen_archive/unsupported/Eigen/CXX11/Tensor:14,
from ./third_party/eigen3/unsupported/Eigen/CXX11/Tensor:1,
from ./tensorflow/core/framework/tensor_shape.h:21,
from ./tensorflow/core/kernels/conv_grad_ops.h:164,
from tensorflow/core/kernels/conv_grad_filter_ops.cc:21:
/usr/lib/gcc/x86_64-linux-gnu/6/include/avx512vlbwintrin.h: In function 'typename Eigen::internal::enable_if<Eigen::internal::unpacket_traits<T>::masked_load_available, Packet>::type Eigen::internal::ploadu(const typename Eigen::internal::unpacket_traits<Packet>::type*, typename Eigen::internal::unpacket_traits<Packet>::mask_t) [with Packet = Eigen::internal::Packet16h]':
/usr/lib/gcc/x86_64-linux-gnu/6/include/avx512vlbwintrin.h:105:1: error: inlining failed in call to always_inline '__m256i _mm256_maskz_loadu_epi16(__mmask16, const void*)': target specific option mismatch
_mm256_maskz_loadu_epi16 (__mmask16 __U, void const *__P)
^~~~~~~~~~~~~~~~~~~~~~~~
In file included from external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/Core:202:0,
from external/eigen_archive/unsupported/Eigen/CXX11/Tensor:14,
from ./third_party/eigen3/unsupported/Eigen/CXX11/Tensor:1,
from ./tensorflow/core/framework/tensor_shape.h:21,
from ./tensorflow/core/kernels/conv_grad_ops.h:164,
from tensorflow/core/kernels/conv_grad_filter_ops.cc:21:
external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/Core/arch/GPU/PacketMathHalf.h:598:38: note: called from here
result.x = _mm256_maskz_loadu_epi16(mask, from);
~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 387.379s, Critical Path: 200.24s
INFO: 763 processes: 763 local.
FAILED: Build did NOT complete successfully
Bisect to b53fd7648b5ca7eaeceb602617433be6e9a4abec reports the following error:
In file included from /usr/lib/gcc/x86_64-linux-gnu/6/include/immintrin.h:59:0,
from external/eigen_archive/Eigen/src/Core/util/ConfigureVectorization.h:318,
from external/eigen_archive/Eigen/Core:22,
from ./third_party/eigen3/Eigen/Core:1,
from ./tensorflow/compiler/xla/service/cpu/runtime_conv2d.h:19,
from tensorflow/compiler/xla/service/cpu/runtime_conv2d.cc:16:
/usr/lib/gcc/x86_64-linux-gnu/6/include/avx512vlbwintrin.h: In function 'typename Eigen::internal::enable_if<Eigen::internal::unpacket_traits<T>::masked_load_available, Packet>::type Eigen::internal::ploadu(const typename Eigen::internal::unpacket_traits<Packet>::type*, typename Eigen::internal::unpacket_traits<Packet>::mask_t) [with Packet = Eigen::internal::Packet16h]':
/usr/lib/gcc/x86_64-linux-gnu/6/include/avx512vlbwintrin.h:105:1: error: inlining failed in call to always_inline '__m256i _mm256_maskz_loadu_epi16(__mmask16, const void*)': target specific option mismatch
_mm256_maskz_loadu_epi16 (__mmask16 __U, void const *__P)
^~~~~~~~~~~~~~~~~~~~~~~~
In file included from external/eigen_archive/Eigen/Core:202:0,
from ./third_party/eigen3/Eigen/Core:1,
from ./tensorflow/compiler/xla/service/cpu/runtime_conv2d.h:19,
from tensorflow/compiler/xla/service/cpu/runtime_conv2d.cc:16:
external/eigen_archive/Eigen/src/Core/arch/GPU/PacketMathHalf.h:598:38: note: called from here
result.x = _mm256_maskz_loadu_epi16(mask, from);
~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 639.500s, Critical Path: 90.25s
INFO: 4690 processes: 4690 local.
FAILED: Build did NOT complete successfully
Reverting b53fd7648b5ca7eaeceb602617433be6e9a4abec resolves the issue.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 16 (7 by maintainers)
I’ve just mirrored Eigen to http://mirror.tensorflow.org/bitbucket.org/eigen/eigen/get/a0d250e79c79.tar.gz, either we never mirrored it before, or lost it.
The change is now merged in Eigen. The next EIgen bump in TensorFlow should pull it in.
Eigen PR is out: https://bitbucket.org/eigen/eigen/pull-requests/639/fix-avx512-gcc-63-compilation/diff
I guess
frompointer has to be reinterpret_casted similar to the method above: https://bitbucket.org/eigen/eigen/src/1984a10c33ea299487e6492602076cccc56a97be/Eigen/src/Core/arch/GPU/PacketMathHalf.h#lines-588:600I’ll submit this change to Eigen and will bump the version.