bazel: Version `GLIBCXX_3.4.18' not found when compiling tensorflow example

Hi all,

I’m trying to install bazel and tensorflow with cuda support in my home directory on a network drive of a research cluster. I am building them using tools from a conda virtual environment, so almost nothing I have is in a default location.

Bazel version: 0.2.3 tag from the master branch Gcc version: gcc version 4.8.5 (GCC)

When building the first tensorflow example, I get this error:

/homes/krheuton/bazel/output/bazel build -c opt --config=cuda //tensorflow/cc:tutorials_example_trainer --verbose_failures …

WARNING: Sandboxed execution is not supported on your system and thus hermeticity of actions cannot be guaranteed. See http://bazel.io/docs/bazel-user-manual.html#sandboxing for more information. You can turn off this warning via --ignore_unsupported_sandboxing.

WARNING: /snfs2/HOME/krheuton/.cache/bazel/_bazel_krheuton/4bbcdf630a5812f0ffcb10ef621943b3/external/protobuf/WORKSPACE:1: Workspace name in /snfs2/HOME/krheuton/.cache/bazel/_bazel_krheuton/4bbcdf630a5812f0ffcb10ef621943b3/external/protobuf/WORKSPACE (@main) does not match the name given in the repository’s definition (@protobuf); this will cause a build error in future versions.

WARNING: /snfs2/HOME/krheuton/.cache/bazel/_bazel_krheuton/4bbcdf630a5812f0ffcb10ef621943b3/external/re2/WORKSPACE:1: Workspace name in /snfs2/HOME/krheuton/.cache/bazel/_bazel_krheuton/4bbcdf630a5812f0ffcb10ef621943b3/external/re2/WORKSPACE (@main) does not match the name given in the repository’s definition (@re2); this will cause a build error in future versions.

WARNING: /snfs2/HOME/krheuton/.cache/bazel/_bazel_krheuton/4bbcdf630a5812f0ffcb10ef621943b3/external/highwayhash/WORKSPACE:1: Workspace name in /snfs2/HOME/krheuton/.cache/bazel/_bazel_krheuton/4bbcdf630a5812f0ffcb10ef621943b3/external/highwayhash/WORKSPACE (@main) does not match the name given in the repository’s definition (@highwayhash); this will cause a build error in future versions.

INFO: Found 1 target…

Slow read: a 111231960-byte read from /snfs2/HOME/krheuton/.cache/bazel/_bazel_krheuton/4bbcdf630a5812f0ffcb10ef621943b3/tensorflow/bazel-out/local_linux-opt/genfiles/third_party/gpus/cuda/lib64/libcufft.so took 8004ms. Slow read: a 111231960-byte read from /snfs2/HOME/krheuton/.cache/bazel/_bazel_krheuton/4bbcdf630a5812f0ffcb10ef621943b3/tensorflow/bazel-out/host/genfiles/third_party/gpus/cuda/lib64/libcufft.so took 7987ms. Slow read: a 111231960-byte read from /snfs2/HOME/krheuton/.cache/bazel/_bazel_krheuton/4bbcdf630a5812f0ffcb10ef621943b3/tensorflow/bazel-out/local_linux-opt/bin/_solib_local/_U_S_Sthird_Uparty_Sgpus_Scuda_Ccufft___Uthird_Uparty_Sgpus_Scuda_Slib64/libcufft.so took 7999ms.

ERROR: /snfs2/HOME/krheuton/tensorflow/tensorflow/core/BUILD:90:1: null failed: protoc failed: error executing command (cd /snfs2/HOME/krheuton/.cache/bazel/_bazel_krheuton/4bbcdf630a5812f0ffcb10ef621943b3/tensorflow &&
exec env -
bazel-out/host/bin/external/protobuf/protoc ‘–cpp_out=bazel-out/local_linux-opt/genfiles/’ -I. -Iexternal/protobuf/src -Ibazel-out/local_linux-opt/genfiles/external/protobuf/src tensorflow/core/example/example.proto tensorflow/core/example/feature.proto tensorflow/core/framework/allocation_description.proto tensorflow/core/framework/attr_value.proto tensorflow/core/framework/cost_graph.proto tensorflow/core/framework/device_attributes.proto tensorflow/core/framework/function.proto tensorflow/core/framework/graph.proto tensorflow/core/framework/kernel_def.proto tensorflow/core/framework/log_memory.proto tensorflow/core/framework/op_def.proto tensorflow/core/framework/step_stats.proto tensorflow/core/framework/summary.proto tensorflow/core/framework/tensor.proto tensorflow/core/framework/tensor_description.proto tensorflow/core/framework/tensor_shape.proto tensorflow/core/framework/tensor_slice.proto tensorflow/core/framework/types.proto tensorflow/core/framework/variable.proto tensorflow/core/framework/versions.proto tensorflow/core/lib/core/error_codes.proto tensorflow/core/protobuf/config.proto tensorflow/core/protobuf/meta_graph.proto tensorflow/core/protobuf/named_tensor.proto tensorflow/core/protobuf/queue_runner.proto tensorflow/core/protobuf/saver.proto tensorflow/core/protobuf/tensorflow_server.proto tensorflow/core/util/event.proto tensorflow/core/util/memmapped_file_system.proto tensorflow/core/util/saved_tensor_slice.proto tensorflow/core/util/test_log.proto): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1. bazel-out/host/bin/external/protobuf/protoc: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.18’ not found (required by bazel-out/host/bin/external/protobuf/protoc) Target //tensorflow/cc:tutorials_example_trainer failed to build INFO: Elapsed time: 154.944s, Critical Path: 83.57s

This seems to mirror issue #898 but I can’t quite determine what the resolution was there.

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 19 (6 by maintainers)

Most upvoted comments

@kheuton You can add env=ctx.configuration.default_shell_env to the ctx.action call in bazel-tensorflow/external/protobuf/protobuf.bzl to get past the problem.

+12

gbkedar on Jul 12, 2016

@gbkedar Your comment from Jul 12 got me through this error after several hours of adjusting CROSSTOOL and recompile attempts! Thanks! (I was using the latest bazel build 0.3.2-2016-11-02 (@03afc7d) and tensorflow v0.11.0rc1)

I wanted to document my steps for other users attempting to compile tensorflow without root on a cluster. This is the first link that pops up in a google search of the error and the only one that helped so I thought I’d post the info here in hopes of saving others the hours I spent trying to get past this error. I was able to compile bazel from source, compile tensorflow from source, and successfully pip install from the created .whl file after adding the line from the Jul 12 comment above. Please find a more detailed explanation of my steps here:

Installing Tensorflow on CENTOS 6.8 without Root

taylorpaul on Nov 4, 2016

@kheuton in third_party/gpus/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc change the line PREFIX_DIR = os.path.dirname(GCC_HOST_COMPILER_PATH) to PREFIX_DIR = os.path.dirname(“Path to AS”)

gbkedar on Jul 25, 2016

@taylorpaul 's guide is nice, but it looks like there’s a simpler way to fix version 'GLIBCXX_3.4.20' not found.

I’ve not tried his trick with copying “as”, though (if it works, it should be faster and simplier workaround), instead, I’ve recompiled gcc, hardcoding paths to as, nm and ld. build tensorflow 0.12rc0 on CentOS6.5, which only had gcc-4.4.7 compiler by default.

i3v on Dec 7, 2016