tensorflow: Eigen version bump breaks nightly AVX512 build with gcc 6.3

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Debian 9.9
  • Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy): N/A
  • TensorFlow installed from (source or binary): source
  • TensorFlow version: e432bf03931f4062f7c5e3a1553aff61a7294751
  • Python version: 2.7.13
  • Installed using virtualenv? pip? conda?: virtualenv
  • Bazel version (if compiling from source): 0.24.1
  • GCC/Compiler version (if compiling from source): (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
  • CUDA/cuDNN version: N/A
  • GPU model and memory: N/A

Describe the problem

In a GCE VM with AVX512 supported CPU:

bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package

Any other info / logs

In file included from /usr/lib/gcc/x86_64-linux-gnu/6/include/immintrin.h:59:0,
                 from external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/Core/util/ConfigureVectorization.h:318,
                 from external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/Core:22,
                 from external/eigen_archive/unsupported/Eigen/CXX11/Tensor:14,
                 from ./third_party/eigen3/unsupported/Eigen/CXX11/Tensor:1,
                 from ./tensorflow/core/framework/tensor_shape.h:21,
                 from ./tensorflow/core/kernels/conv_grad_ops.h:164,
                 from tensorflow/core/kernels/conv_grad_filter_ops.cc:21:
/usr/lib/gcc/x86_64-linux-gnu/6/include/avx512vlbwintrin.h: In function 'typename Eigen::internal::enable_if<Eigen::internal::unpacket_traits<T>::masked_load_available, Packet>::type Eigen::internal::ploadu(const typename Eigen::internal::unpacket_traits<Packet>::type*, typename Eigen::internal::unpacket_traits<Packet>::mask_t) [with Packet = Eigen::internal::Packet16h]':
/usr/lib/gcc/x86_64-linux-gnu/6/include/avx512vlbwintrin.h:105:1: error: inlining failed in call to always_inline '__m256i _mm256_maskz_loadu_epi16(__mmask16, const void*)': target specific option mismatch
 _mm256_maskz_loadu_epi16 (__mmask16 __U, void const *__P)
 ^~~~~~~~~~~~~~~~~~~~~~~~
In file included from external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/Core:202:0,
                 from external/eigen_archive/unsupported/Eigen/CXX11/Tensor:14,
                 from ./third_party/eigen3/unsupported/Eigen/CXX11/Tensor:1,
                 from ./tensorflow/core/framework/tensor_shape.h:21,
                 from ./tensorflow/core/kernels/conv_grad_ops.h:164,
                 from tensorflow/core/kernels/conv_grad_filter_ops.cc:21:
external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/Core/arch/GPU/PacketMathHalf.h:598:38: note: called from here
   result.x = _mm256_maskz_loadu_epi16(mask, from);
              ~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 387.379s, Critical Path: 200.24s
INFO: 763 processes: 763 local.
FAILED: Build did NOT complete successfully

Bisect to b53fd7648b5ca7eaeceb602617433be6e9a4abec reports the following error:

In file included from /usr/lib/gcc/x86_64-linux-gnu/6/include/immintrin.h:59:0,
                 from external/eigen_archive/Eigen/src/Core/util/ConfigureVectorization.h:318,
                 from external/eigen_archive/Eigen/Core:22,
                 from ./third_party/eigen3/Eigen/Core:1,
                 from ./tensorflow/compiler/xla/service/cpu/runtime_conv2d.h:19,
                 from tensorflow/compiler/xla/service/cpu/runtime_conv2d.cc:16:
/usr/lib/gcc/x86_64-linux-gnu/6/include/avx512vlbwintrin.h: In function 'typename Eigen::internal::enable_if<Eigen::internal::unpacket_traits<T>::masked_load_available, Packet>::type Eigen::internal::ploadu(const typename Eigen::internal::unpacket_traits<Packet>::type*, typename Eigen::internal::unpacket_traits<Packet>::mask_t) [with Packet = Eigen::internal::Packet16h]':
/usr/lib/gcc/x86_64-linux-gnu/6/include/avx512vlbwintrin.h:105:1: error: inlining failed in call to always_inline '__m256i _mm256_maskz_loadu_epi16(__mmask16, const void*)': target specific option mismatch
 _mm256_maskz_loadu_epi16 (__mmask16 __U, void const *__P)
 ^~~~~~~~~~~~~~~~~~~~~~~~
In file included from external/eigen_archive/Eigen/Core:202:0,
                 from ./third_party/eigen3/Eigen/Core:1,
                 from ./tensorflow/compiler/xla/service/cpu/runtime_conv2d.h:19,
                 from tensorflow/compiler/xla/service/cpu/runtime_conv2d.cc:16:
external/eigen_archive/Eigen/src/Core/arch/GPU/PacketMathHalf.h:598:38: note: called from here
   result.x = _mm256_maskz_loadu_epi16(mask, from);
              ~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 639.500s, Critical Path: 90.25s
INFO: 4690 processes: 4690 local.
FAILED: Build did NOT complete successfully

Reverting b53fd7648b5ca7eaeceb602617433be6e9a4abec resolves the issue.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 16 (7 by maintainers)

Most upvoted comments

I’ve just mirrored Eigen to http://mirror.tensorflow.org/bitbucket.org/eigen/eigen/get/a0d250e79c79.tar.gz, either we never mirrored it before, or lost it.

The change is now merged in Eigen. The next EIgen bump in TensorFlow should pull it in.

I guess from pointer has to be reinterpret_casted similar to the method above: https://bitbucket.org/eigen/eigen/src/1984a10c33ea299487e6492602076cccc56a97be/Eigen/src/Core/arch/GPU/PacketMathHalf.h#lines-588:600

I’ll submit this change to Eigen and will bump the version.