tensorflow: libstdc++.so.6: version `CXXABI_1.3.8' not found

PS, happy new year!

I get the same error on Centos 7

Oh, for posterity, here are some facts we uncovered for anyone who’s interested in using Debian 8 with Clang. This could potentially be useful for other people in the industry who are interested in distributing TensorFlow.

---

Debian 8 Jessie (lifetime 2015→2020) is my personal favorite. Clang team maintains an apt repo for Debian 8. The two have a surprising degree of synergy.

Unlike the GCC 4.9.2 which comes included with Debian 8, Clang 6.0 is able to generate opcodes for more recent instruction sets, e.g. AVX-512. Those can make TF potentially 10x faster on CPUs like Skylake Xeons (c. 2015) assuming -march=native is used.

If one’s goal is compatibility, -msse3 is a safe optimization that works on nearly every x86_64 CPU in existence. For example, the optional TF Debian 8 Clang 6 Bazel CROSSTOOL should in theory create binaries that are compatible with:

• Ubuntu 14+
• CentOS 7+
• Debian 8+
• SuSE 13.2+
• Mint 17.3+
• Manjaro 0.8.11

Proof:

jart@tomservo:/tmp/pip$ find . -name \*.so | xargs ldd -v | grep -Po '(?:GCC|GLIBC|CXXABI)_[.\d]+' | sort -n | sort -u
CXXABI_1.3.7 (GCC 4.8.3)
GCC_4.2.0
GLIBC_2.18
jart@tomservo:/tmp/pip$ ldd tensorflow-1.5.0rc0.data/purelib/tensorflow/libtensorflow_framework.so 
        linux-vdso.so.1 (0x00007ffe70f80000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f4487259000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f4486f58000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4486d3b000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f4486a30000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4486685000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f448801b000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f448646f000)

Some facts we uncovered about Linux distributions:

# Linux distro versions
2014 CentOS 5.11 (EOL 2019-08)    gcc4.1 libc2.5 libc++4.1.2 sqlite3.3.6
2015 Wheezy 7.0 (EOL 2018-05)     gcc4.7 libc2.13 libc++4.7.2 sqlite3.7.13
2014 CentOS 7 (EOL 2019-08)       gcc4.8 libc2.17 libc++4.8.5 sqlite3.7.17
2014 Trusty 14.04 (EOL 2019-04)   gcc4.8 libc2.19 libc++4.8.4 sqlite3.8.2
2015 Jessie 8.0 (EOL 2020-05)     gcc4.9 libc2.19 libc++4.8.4 sqlite3.8.7
2016 Xenial 16.04 (EOL 2021-04)   gcc5.3 libc2.23 libc++5.4.0 sqlite3.11.0

# C++ ABIs
CXXABI_1.3.7 <-- max on ubuntu 14?
CXXABI_1.3.8 <-- where the C++11 ABI break happened for GCC 5.x
strings /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep LIBCXX

Some facts we uncovered about x86 CPUs (w.r.t. scientific computing):

# Noteworthy CPU Features
SSE      128-bit vector instructions (e.g. addps http://www.felixcloutier.com/x86/ADDPS.html)
SSE4.1   Makes SSE good
SSE4.2   Makes SSE do strings and CRC32
AVX      Makes SSE 256-bit (e.g. vaddps)
AVX2     Adds features to AVX
AVX-512  Makes SSE 512-bit
SHA      Makes SSE do SHA1 and SHA256
FMA      Fused multiply–add
CLMUL    Carry-less multiplication https://goo.gl/ritKJX
TSX      Transactional memory instructions for threads http://www.intel.com/software/tsx

# Intel CPU Line
2003 P6 M           SSE SSE2
2004 prescott       SSE3 SSSE3 (-march=prescott)
2006 core           X64 SSE4.1 (only on 45nm variety) (-march=core2)
2008 nehalem        SSE4.2 VT-x VT-d (-march=nehalem)
2010 westmere       CLMUL AES (-march=westmere)
2012 sandybridge    AVX TXT (-march=sandybridge)
2012 ivybridge      F16C MOVBE (-march=ivybridge)
2013 haswell        AVX2 TSX BMI2 FMA (-march=haswell)
2014 broadwell      RDSEED ADCX PREFETCHW (-march=broadwell - works on trusty gcc4.9)
2015 skylake        SGX ADX MPX AVX-512[xeon-only] (-march=skylake / -march=skylake-avx512 - needs gcc7)
2018 cannonlake     AVX-512 SHA (-march=cannonlake - needs clang5)

# Intel Low Power CPU Line
2013 silvermont     SSE4.1 SSE4.2 VT-x (-march=silvermont)
2016 goldmont       SHA (-march=goldmont - needs clang5)

# AMD CPU Line
2003 k8             SSE SSE2 (-march=k8)
2005 k8 (Venus)     SSE3 (-march=k8-sse3)
2008 barcelona      SSE4a?! (-march=barcelona)
2011 bulldozer      SSE4.1 SSE4.2 CLMUL AVX AES FMA4?! (-march=bdver1)
2011 piledriver     FMA (-march=bdver2)
2015 excavator      AVX2 BMI2 MOVBE (-march=bdver4)

# Google Compute Engine Supported CPUs
# https://cloud.google.com/compute/docs/cpu-platforms
2012 sandybridge 2.6gHz -march=sandybridge
2012 ivybridge   2.5gHz -march=ivybridge
2013 haswell     2.3gHz -march=haswell
2014 broadwell   2.2gHz -march=broadwell
2015 skylake     2.0gHz -march=skylake-avx512

One possibility Debian/Ubuntu distributors could explore is:

# We might be able to say something like this in /etc/apt/sources.list
# so folks can get a package for their specific microarchitecture.
deb [arch=skylake] https://apt.tensorflow.org/ stretch stable

Please note -msse3 or -march=native is not a binary dilemma. It’s possible to use a single Debian 8 machine to cross-compile to -march=whatever.

There also appear to be sweet-spot combinations where you can build binaries that support large subsets in-between. For example, the following optimized CFLAGS have been carefully selected to represent the points in time where Intel and AMD microarchitectures came into alignment.

oldcpu: Nearly all x86_64 microarchitectures
  $ intersect <(march-copts core2) <(march-copts k8-sse3)
  -msse3
default: x86_64 + SSE4
  $ intersect <(march-copts nehalem) <(march-copts bdver1)
  -mcx16
  -mpopcnt
  -msahf
  -msse3
  -msse4
  -msse4.1
  -msse4.2
  -mssse3
avx: x86_64 + AVX
  $ intersect <(march-copts sandybridge) <(march-copts bdver1)
  -maes
  -mavx
  -mavx256-split-unaligned-store
  -mcx16
  -mpclmul
  -mpopcnt
  -msahf
  -msse3
  -msse4
  -msse4.1
  -msse4.2
  -mssse3
  -mxsave
avx2: x86_64 + AVX2
  $ intersect <(march-copts haswell) <(march-copts bdver4)
  -maes
  -mavx
  -mavx2
  -mbmi
  -mbmi2
  -mcx16
  -mf16c
  -mfma
  -mfsgsbase
  -mlzcnt
  -mmovbe
  -mpclmul
  -mpopcnt
  -mrdrnd
  -msahf
  -msse3
  -msse4
  -msse4.1
  -msse4.2
  -mssse3
  -mxsave
  -mxsaveopt
avx3: x86_64 + AVX-512
  $ march-copts skylake-avx512
  -madx
  -maes
  -mavx
  -mavx2
  -mavx512bw
  -mavx512cd
  -mavx512dq
  -mavx512f
  -mavx512vl
  -mbmi
  -mbmi2
  -mclflushopt
  -mcx16
  -mf16c
  -mfma
  -mfsgsbase
  -mhle
  -mlzcnt
  -mmovbe
  -mpclmul
  -mpku
  -mpopcnt
  -mprfchw
  -mrdrnd
  -mrdseed
  -msahf
  -msse3
  -msse4
  -msse4.1
  -msse4.2
  -mssse3
  -mxsave
  -mxsavec
  -mxsaveopt
  -mxsaves

See also:

• https://distrowatch.com/table.php?distribution=ubuntu
• https://distrowatch.com/table.php?distribution=redhat
• http://vault.centos.org/5.11/os/x86_64/CentOS/

I’m pretty sure at this point it’s impossible to build TF binaries on Ubuntu 16 that work on Ubuntu 14. The dual ABI macro is able to remove a lot of the newer __cxx11 symbols, but no matter what I try, dependencies on new libc and libc++ interfaces just keep sneaking in.

Probably the only way it’d be possible to maintain Ubuntu 14 support, would be if our Ubuntu 16 machines used an Ubuntu 14 docker container to build the CPU package.

For posterity’s sake, here’s what I did to experiment:

# Build TensorFlow pip package on Ubuntu 16.04
docker run --name=build -v /home/jart/filez:/filez -t -d ubuntu:16.04
docker exec -i -t build /bin/bash
cd /root
apt-get update
apt-get install -y --no-install-recommends build-essential python python-dev pkg-config zip g++ zlib1g-dev unzip wget git-core python-numpy python-pip
wget https://github.com/bazelbuild/bazel/releases/download/0.9.0/bazel-0.9.0-installer-linux-x86_64.sh
chmod +x bazel-0.9.0-installer-linux-x86_64.sh
./bazel-0.9.0-installer-linux-x86_64.sh
git clone https://github.com/tensorflow/tensorflow
cd tensorflow
./configure
bazel build -s --copt=-fabi-version=7 --host_copt=-fabi-version=7 --copt=-D_GLIBCXX_USE_CXX11_ABI=0 --host_copt=-D_GLIBCXX_USE_CXX11_ABI=0 --config=opt //tensorflow/tools/pip_package:build_pip_package
bazel-bin/tensorflow/tools/pip_package/build_pip_package /filez/tf

# Test it works with Ubuntu 14.04 C++ ABI
docker run --name=user -v /usr/local/google/home/jart/filez:/filez -t -d ubuntu:14.04
docker exec -i -t user /bin/bash
cd /root
apt-get update
apt-get install -y --no-install-recommends python python-virtualenv
virtualenv tf
cd tf
source bin/activate
easy_install -U pip
pip install -U /filez/tf/tensorflow-1.5.0rc0-cp27-cp27mu-linux_x86_64.whl
python -c 'import tensorflow'

# Cleanup
docker stop build
docker rm build
docker stop user
docker rm user

Ubuntu 14 LTE doesn’t become EOL until April 2019. Are we planning to move it out of our support matrix? (See go/tf-issues-triage) What about CentOS 7, which has gcc 4.8.5? What are the benefits of this new ABI?

If changing our support matrix is the consensus here, I’d suggest we go a step further and compile prebuilt binaries with SSE4 (maybe AVX2, possibly even AVX512) since that makes TensorFlow CPU 4x faster.

Otherwise, maybe it’s possible to use _GLIBCXX_USE_CXX11_ABI to support the old ABI? According to this page, the new ABI is mostly meant to resolve the concerns of the C++ standards committee.

IMHO it’s too early to drop support for Ubuntu 14.04 and Docker still needs time to mature (ie, 3 months ago you couldn’t use GPUs properly with docker)