envoy: Envoy FIPS v1.29.x x86_64 build failures

Description

The x86_64 build for release.server_only of FIPS Envoy (v1.29.X) seems to error out at the same stage, and looks to be related to https://github.com/envoyproxy/envoy/issues/31874 and https://github.com/envoyproxy/envoy/pull/32901

Setup Env

export BAZEL_BUILD_EXTRA_OPTIONS="--define boringssl=fips --local_cpu_resources=12 --build_tag_filters=-nofips --local_ram_resources=HOST_RAM*0.80 --jobs=12 --progress_report_interval=60"
export BAZEL_EXTRA_TEST_OPTIONS="--test_tag_filters=-nofips --test_env=ENVOY_IP_TEST_VERSIONS=v4only --local_ram_resources=HOST_RAM*1 --local_cpu_resources=12 --test_output=errors --test_summary=detailed"

Command:

ci/run_envoy_docker.sh ci/do_ci.sh release.server_only

Error logs

v1.29.1

ld.lld: error: cannot preempt symbol: OPENSSL_ia32cap_P
>>> defined in /build/bazel_root/base/sandbox/processwrapper-sandbox/6837/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a(crypto.c.o)
>>> referenced by cpu-intel.c:287 (../crypto/cpu-intel.c:287)
>>>               cpu-intel.c.o:(OPENSSL_cpuid_setup) in archive /build/bazel_root/base/sandbox/processwrapper-sandbox/6837/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a

ld.lld: error: relocation R_X86_64_32 cannot be used against local symbol; recompile with -fPIC
>>> defined in /build/bazel_root/base/sandbox/processwrapper-sandbox/6837/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a(cpu-intel.c.o)
>>> referenced by cpu-intel.c:133 (../crypto/cpu-intel.c:133)
>>>               cpu-intel.c.o:(handle_cpu_env) in archive /build/bazel_root/base/sandbox/processwrapper-sandbox/6837/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a

ld.lld: error: relocation R_X86_64_32 cannot be used against local symbol; recompile with -fPIC
>>> defined in /build/bazel_root/base/sandbox/processwrapper-sandbox/6837/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a(cpu-intel.c.o)
>>> referenced by cpu-intel.c:0 (../crypto/cpu-intel.c:0)
>>>               cpu-intel.c.o:(handle_cpu_env) in archive /build/bazel_root/base/sandbox/processwrapper-sandbox/6837/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a

ld.lld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
clang-14: ·[0;1;31merror: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [Makefile:493: qzip] Error 1
make[1]: Leaving directory '/build/bazel_root/base/sandbox/processwrapper-sandbox/6837/execroot/envoy/bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip.build_tmpdir/utils'
make: *** [Makefile:568: all-recursive] Error 1

v1.29.2

ld.lld: error: relocation R_X86_64_32 cannot be used against local symbol; recompile with -fPIC
>>> defined in /build/bazel_root/base/sandbox/processwrapper-sandbox/6753/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a(cpu-intel.c.o)
>>> referenced by cpu-intel.c:0 (../crypto/cpu-intel.c:0)
>>>               cpu-intel.c.o:(handle_cpu_env) in archive /build/bazel_root/base/sandbox/processwrapper-sandbox/6753/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a

ld.lld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
clang-14: ·[0;1;31merror: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [Makefile:493: qzip] Error 1
make[1]: Leaving directory '/build/bazel_root/base/sandbox/processwrapper-sandbox/6753/execroot/envoy/bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip.build_tmpdir/utils'
make: *** [Makefile:568: all-recursive] Error 1
_____ END BUILD LOGS _____
rules_foreign_cc: Build wrapper script location: bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip_foreign_cc/wrapper_build_script.sh
rules_foreign_cc: Build script location: bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip_foreign_cc/build_script.sh
rules_foreign_cc: Build log location: bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip_foreign_cc/Configure.log

Target //distribution/binary:release failed to build
ERROR: /source/contrib/exe/BUILD:23:16 Linking contrib/exe/envoy-static failed: (Exit 2): bash failed: error executing command (from target //contrib/qat/compression/qatzip/compressor/source:qatzip) 
  (cd /build/bazel_root/base/sandbox/processwrapper-sandbox/6753/execroot/envoy && \
  exec env - \
    BAZEL_COMPILER=clang \
    BAZEL_CXXOPTS='-stdlib=libc++' \
    BAZEL_LINKLIBS=-l%:libc++.a:-l%:libc++abi.a \
    BAZEL_LINKOPTS=-lm:-pthread \
    CC=clang \
    CXX=clang++ \
    CXXFLAGS='-stdlib=libc++' \
    LDFLAGS='-stdlib=libc++' \
    LLVM_CONFIG=/opt/llvm/bin/llvm-config \
    PATH=/opt/llvm/bin:/opt/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
  /bin/bash -c bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip_foreign_cc/wrapper_build_script.sh)
# Configuration: 7e5fb22f0622a366157af3db174db77e4136215ff4d0bf9d71bc7eb0ac911359
# Execution platform: @local_config_platform//:host

v1.29.3

ld.lld: error: relocation R_X86_64_32 cannot be used against local symbol; recompile with -fPIC
>>> defined in /build/bazel_root/base/sandbox/processwrapper-sandbox/6394/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a(cpu-intel.c.o)
>>> referenced by cpu-intel.c:0 (../crypto/cpu-intel.c:0)
>>>               cpu-intel.c.o:(handle_cpu_env) in archive /build/bazel_root/base/sandbox/processwrapper-sandbox/6394/execroot/envoy/bazel-out/k8-opt/bin/external/boringssl_fips/crypto/libcrypto.a

ld.lld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
clang-14: ·[0;1;31merror: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [Makefile:493: qzip] Error 1
make[1]: Leaving directory '/build/bazel_root/base/sandbox/processwrapper-sandbox/6394/execroot/envoy/bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip.build_tmpdir/utils'
make: *** [Makefile:568: all-recursive] Error 1
_____ END BUILD LOGS _____
rules_foreign_cc: Build wrapper script location: bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip_foreign_cc/wrapper_build_script.sh
rules_foreign_cc: Build script location: bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip_foreign_cc/build_script.sh
rules_foreign_cc: Build log location: bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip_foreign_cc/Configure.log

[12,845 / 14,338] Compiling src/codegen/compiler.cc; 15s processwrapper-sandbox ... (25 actions running)
Target //distribution/binary:release failed to build
ERROR: /source/contrib/exe/BUILD:31:16 Linking contrib/exe/envoy-static failed: (Exit 2): bash failed: error executing command (from target //contrib/qat/compression/qatzip/compressor/source:qatzip) 
  (cd /build/bazel_root/base/sandbox/processwrapper-sandbox/6394/execroot/envoy && \
  exec env - \
    BAZEL_COMPILER=clang \
    BAZEL_CXXOPTS='-stdlib=libc++' \
    BAZEL_LINKLIBS=-l%:libc++.a:-l%:libc++abi.a \
    BAZEL_LINKOPTS=-lm:-pthread \
    CC=clang \
    CXX=clang++ \
    CXXFLAGS='-stdlib=libc++' \
    LDFLAGS='-stdlib=libc++' \
    LLVM_CONFIG=/opt/llvm/bin/llvm-config \
    PATH=/opt/llvm/bin:/opt/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
  /bin/bash -c bazel-out/k8-opt/bin/contrib/qat/compression/qatzip/compressor/source/qatzip_foreign_cc/wrapper_build_script.sh)
# Configuration: 7e5fb22f0622a366157af3db174db77e4136215ff4d0bf9d71bc7eb0ac911359
# Execution platform: @local_config_platform//:host

About this issue

  • Original URL
  • State: open
  • Created 3 months ago
  • Reactions: 1
  • Comments: 15 (9 by maintainers)

Most upvoted comments

I can raise a new issue for the v1.30.0 build issue

rather raise any issue against main as the issue must be present there also

any ideas on when this backfix will be available on the tagged releases for v1.29.X?

there isnt a current plan to cut a release but im guessing a month or so - there is nothing stopping us cutting a release sooner - just needs a bit of work

I’ve also discovered with the new v1.30.0 release, build errors as well with this backport cherry picked and without it.

im a bit confused here - the backport was cherry-picked from main and 1.30 was cut from main so it should be on that branch already

Ahh you’re right, I missed reverting my troubleshooting changes, and have added --build_tag_filters=-nofips back This works now 👍

Thanks for your help @phlax