tensorflow: TF 2.5.0 with CUDA 10.2 (in container) Build error extract_volume_patches_op_gpu failed: crosstool_wrapper_driver_is_not_gcc

System information

• OS Platform and Distribution: Linux Ubuntu 18.04 (base container, see below)
• Docker base container: nvidia/cuda:10.2-cudnn8-devel-ubuntu18.04
• TensorFlow installed from:source
• TensorFlow version: 2.5.0
• Python version: 3.8
• Installed using: building for pip install
• Bazel version: 3.7.2
• GCC/Compiler version: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
• CUDA/cuDNN version: 10.2 / 8.2
• GPU model and memory: RTX 3090 / 24GB

Describe the problem

When trying to build TF 2.5.0 with Cuda 10.2, after applying the patch proposed in https://github.com/tensorflow/tensorflow/pull/48393 I get the following build error:

- Environment variables set:
TF_CUDA_CLANG=0
TF_CUDA_COMPUTE_CAPABILITIES=6.0,6.1,7.0,7.5
TF_CUDA_VERSION=10.2
TF_CUDNN_VERSION=8
TF_DOWNLOAD_CLANG=0
TF_DOWNLOAD_MKL=0
TF_ENABLE_XLA=0
TF_NCCL_VERSION=2
TF_NEED_AWS=0
TF_NEED_COMPUTECPP=0
TF_NEED_CUDA=1
TF_NEED_GCP=0
TF_NEED_GDR=0
TF_NEED_HDFS=0
TF_NEED_JEMALLOC=1
TF_NEED_KAFKA=0
TF_NEED_MKL=0
TF_NEED_MPI=0
TF_NEED_OPENCL=0
TF_NEED_OPENCL_SYCL=0
TF_NEED_ROCM=0
TF_NEED_S3=0
TF_NEED_TENSORRT=0
TF_NEED_VERBS=0
TF_SET_ANDROID_WORKSPACE=0
GCC_HOST_COMPILER_PATH=/usr/bin/gcc
CC_OPT_FLAGS=
PYTHON_BIN_PATH=/usr/local/bin/python
PYTHON_LIB_PATH=/usr/local/lib/python3.8/dist-packages
-- ./configure output:
You have bazel 3.7.2 installed.
Found CUDA 10.2 in:
    /usr/local/cuda-10.2/targets/x86_64-linux/lib
    /usr/local/cuda-10.2/targets/x86_64-linux/include
Found cuDNN 8 in:
    /usr/lib/x86_64-linux-gnu
    /usr/include
Found NCCL 2 in:
    /usr/lib/x86_64-linux-gnu
    /usr/include

Built using bazel build --verbose_failures --config=opt --config=v2 --config=cuda //tensorflow/tools/pip_package:build_pip_package

Error log

[25,186 / 36,992] Compiling tensorflow/lite/toco/tflite/operator.cc; 13s local ... (32 actions, 31 running)
ERROR: /usr/local/src/tensorflow/tensorflow/core/kernels/image/BUILD:241:18: C++ compilation of rule '//tensorflow/core/kernels/image:extract_volume_patches_op_gpu' failed (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command 
  (cd /root/.cache/bazel/_bazel_root/bbcc73fcc5c2b01ab08b6bcf7c29e42e/execroot/org_tensorflow && \
  exec env - \
    CUDA_TOOLKIT_PATH=/usr/local/cuda-10.2 \
    GCC_HOST_COMPILER_PATH=/usr/bin/x86_64-linux-gnu-gcc-7 \
    LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 \
    PATH=/root/.cache/bazelisk/downloads/bazelbuild/bazel-3.7.2-linux-x86_64/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    PWD=/proc/self/cwd \
    PYTHON_BIN_PATH=/usr/local/bin/python \
    PYTHON_LIB_PATH=/usr/local/lib/python3.8/dist-packages \
    TF2_BEHAVIOR=1 \
    TF_CUDA_COMPUTE_CAPABILITIES=6.0,6.1,7.0,7.5 \
    TF_CUDA_VERSION=10.2 \
    TF_CUDNN_VERSION=8 \
    TF_NCCL_VERSION=2 \
  external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/k8-opt/bin/tensorflow/core/kernels/image/_objs/extract_volume_patches_op_gpu/extract_volume_patches_op_gpu.cu.pic.d '-frandom-seed=bazel-out/k8-opt/bin/tensorflow/core/kernels/image/_objs/extract_volume_patches_op_gpu/extract_volume_patches_op_gpu.cu.pic.o' -DTENSORFLOW_USE_CUSTOM_CONTRACTION_KERNEL -DTENSORFLOW_USE_MKLDNN_CONTRACTION_KERNEL -DEIGEN_MPL2_ONLY '-DEIGEN_MAX_ALIGN_BYTES=64' -iquote . -iquote bazel-out/k8-opt/bin -iquote external/com_google_absl -iquote bazel-out/k8-opt/bin/external/com_google_absl -iquote external/nsync -iquote bazel-out/k8-opt/bin/external/nsync -iquote external/eigen_archive -iquote bazel-out/k8-opt/bin/external/eigen_archive -iquote external/gif -iquote bazel-out/k8-opt/bin/external/gif -iquote external/libjpeg_turbo -iquote bazel-out/k8-opt/bin/external/libjpeg_turbo -iquote external/com_google_protobuf -iquote bazel-out/k8-opt/bin/external/com_google_protobuf -iquote external/com_googlesource_code_re2 -iquote bazel-out/k8-opt/bin/external/com_googlesource_code_re2 -iquote external/farmhash_archive -iquote bazel-out/k8-opt/bin/external/farmhash_archive -iquote external/fft2d -iquote bazel-out/k8-opt/bin/external/fft2d -iquote external/highwayhash -iquote bazel-out/k8-opt/bin/external/highwayhash -iquote external/zlib -iquote bazel-out/k8-opt/bin/external/zlib -iquote external/local_config_cuda -iquote bazel-out/k8-opt/bin/external/local_config_cuda -iquote external/local_config_rocm -iquote bazel-out/k8-opt/bin/external/local_config_rocm -iquote external/local_config_tensorrt -iquote bazel-out/k8-opt/bin/external/local_config_tensorrt -iquote external/mkl_dnn_v1 -iquote bazel-out/k8-opt/bin/external/mkl_dnn_v1 -Ibazel-out/k8-opt/bin/external/local_config_cuda/cuda/_virtual_includes/cuda_headers_virtual -Ibazel-out/k8-opt/bin/external/local_config_tensorrt/_virtual_includes/tensorrt_headers -isystem external/nsync/public -isystem bazel-out/k8-opt/bin/external/nsync/public -isystem third_party/eigen3/mkl_include -isystem bazel-out/k8-opt/bin/third_party/eigen3/mkl_include -isystem external/eigen_archive -isystem bazel-out/k8-opt/bin/external/eigen_archive -isystem external/gif -isystem bazel-out/k8-opt/bin/external/gif -isystem external/com_google_protobuf/src -isystem bazel-out/k8-opt/bin/external/com_google_protobuf/src -isystem external/farmhash_archive/src -isystem bazel-out/k8-opt/bin/external/farmhash_archive/src -isystem external/zlib -isystem bazel-out/k8-opt/bin/external/zlib -isystem external/local_config_cuda/cuda -isystem bazel-out/k8-opt/bin/external/local_config_cuda/cuda -isystem external/local_config_cuda/cuda/cuda/include -isystem bazel-out/k8-opt/bin/external/local_config_cuda/cuda/cuda/include -isystem external/local_config_rocm/rocm -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm -isystem external/local_config_rocm/rocm/rocm/include -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm/rocm/include -isystem external/local_config_rocm/rocm/rocm/include/rocrand -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm/rocm/include/rocrand -isystem external/local_config_rocm/rocm/rocm/include/roctracer -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm/rocm/include/roctracer -isystem external/mkl_dnn_v1/include -isystem bazel-out/k8-opt/bin/external/mkl_dnn_v1/include -isystem external/mkl_dnn_v1/src -isystem bazel-out/k8-opt/bin/external/mkl_dnn_v1/src -isystem external/mkl_dnn_v1/src/common -isystem bazel-out/k8-opt/bin/external/mkl_dnn_v1/src/common -isystem external/mkl_dnn_v1/src/common/ittnotify -isystem bazel-out/k8-opt/bin/external/mkl_dnn_v1/src/common/ittnotify -isystem external/mkl_dnn_v1/src/cpu -isystem bazel-out/k8-opt/bin/external/mkl_dnn_v1/src/cpu -isystem external/mkl_dnn_v1/src/cpu/gemm -isystem bazel-out/k8-opt/bin/external/mkl_dnn_v1/src/cpu/gemm -isystem external/mkl_dnn_v1/src/cpu/x64/xbyak -isystem bazel-out/k8-opt/bin/external/mkl_dnn_v1/src/cpu/x64/xbyak -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fPIC -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -fno-omit-frame-pointer -no-canonical-prefixes -fno-canonical-system-headers -DNDEBUG -g0 -O2 -ffunction-sections -fdata-sections -w -DAUTOLOAD_DYNAMIC_KERNELS -Wno-sign-compare '-std=c++14' -x cuda '-DGOOGLE_CUDA=1' '-Xcuda-fatbinary=--compress-all' '--no-cuda-include-ptx=all' '--cuda-include-ptx=sm_60' '--cuda-gpu-arch=sm_60' '--cuda-include-ptx=sm_61' '--cuda-gpu-arch=sm_61' '--cuda-include-ptx=sm_70' '--cuda-gpu-arch=sm_70' '--cuda-include-ptx=sm_75' '--cuda-gpu-arch=sm_75' -DEIGEN_AVOID_STL_ARRAY -Iexternal/gemmlowp -Wno-sign-compare '-ftemplate-depth=900' -fno-exceptions '-DGOOGLE_CUDA=1' '-DTENSORFLOW_USE_NVCC=1' -DINTEL_MKL -msse3 -pthread '-nvcc_options=relaxed-constexpr' '-nvcc_options=ftz=true' -c tensorflow/core/kernels/image/extract_volume_patches_op_gpu.cu.cc -o bazel-out/k8-opt/bin/tensorflow/core/kernels/image/_objs/extract_volume_patches_op_gpu/extract_volume_patches_op_gpu.cu.pic.o)
Execution platform: @local_execution_config_platform//:platform
external/com_google_absl/absl/functional/function_ref.h:100:29: error: parameter packs not expanded with ‘...’:
   template <typename F, typename = EnableIfCompatible<const F&>>
                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                                                
external/com_google_absl/absl/functional/function_ref.h:100:29: note:         ‘Args’
external/com_google_absl/absl/functional/function_ref.h:114:13: error: parameter packs not expanded with ‘...’:
       typename F, typename = EnableIfCompatible<F*>,
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                                
external/com_google_absl/absl/functional/function_ref.h:114:13: note:         ‘Args’
Target //tensorflow/tools/pip_package:build_pip_package failed to build
ERROR: /usr/local/src/tensorflow/tensorflow/lite/toco/python/BUILD:89:10 C++ compilation of rule '//tensorflow/core/kernels/image:extract_volume_patches_op_gpu' failed (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command 
  (cd /root/.cache/bazel/_bazel_root/bbcc73fcc5c2b01ab08b6bcf7c29e42e/execroot/org_tensorflow && \
  exec env - \
    CUDA_TOOLKIT_PATH=/usr/local/cuda-10.2 \
    GCC_HOST_COMPILER_PATH=/usr/bin/x86_64-linux-gnu-gcc-7 \
    LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 \
    PATH=/root/.cache/bazelisk/downloads/bazelbuild/bazel-3.7.2-linux-x86_64/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    PWD=/proc/self/cwd \
    PYTHON_BIN_PATH=/usr/local/bin/python \
    PYTHON_LIB_PATH=/usr/local/lib/python3.8/dist-packages \
    TF2_BEHAVIOR=1 \
    TF_CUDA_COMPUTE_CAPABILITIES=6.0,6.1,7.0,7.5 \
    TF_CUDA_VERSION=10.2 \
    TF_CUDNN_VERSION=8 \
    TF_NCCL_VERSION=2 \
  external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/k8-opt/bin/tensorflow/core/kernels/image/_objs/extract_volume_patches_op_gpu/extract_volume_patches_op_gpu.cu.pic.d '-frandom-seed=bazel-out/k8-opt/bin/tensorflow/core/kernels/image/_objs/extract_volume_patches_op_gpu/extract_volume_patches_op_gpu.cu.pic.o' -DTENSORFLOW_USE_CUSTOM_CONTRACTION_KERNEL -DTENSORFLOW_USE_MKLDNN_CONTRACTION_KERNEL -DEIGEN_MPL2_ONLY '-DEIGEN_MAX_ALIGN_BYTES=64' -iquote . -iquote bazel-out/k8-opt/bin -iquote external/com_google_absl -iquote bazel-out/k8-opt/bin/external/com_google_absl -iquote external/nsync -iquote bazel-out/k8-opt/bin/external/nsync -iquote external/eigen_archive -iquote bazel-out/k8-opt/bin/external/eigen_archive -iquote external/gif -iquote bazel-out/k8-opt/bin/external/gif -iquote external/libjpeg_turbo -iquote bazel-out/k8-opt/bin/external/libjpeg_turbo -iquote external/com_google_protobuf -iquote bazel-out/k8-opt/bin/external/com_google_protobuf -iquote external/com_googlesource_code_re2 -iquote bazel-out/k8-opt/bin/external/com_googlesource_code_re2 -iquote external/farmhash_archive -iquote bazel-out/k8-opt/bin/external/farmhash_archive -iquote external/fft2d -iquote bazel-out/k8-opt/bin/external/fft2d -iquote external/highwayhash -iquote bazel-out/k8-opt/bin/external/highwayhash -iquote external/zlib -iquote bazel-out/k8-opt/bin/external/zlib -iquote external/local_config_cuda -iquote bazel-out/k8-opt/bin/external/local_config_cuda -iquote external/local_config_rocm -iquote bazel-out/k8-opt/bin/external/local_config_rocm -iquote external/local_config_tensorrt -iquote bazel-out/k8-opt/bin/external/local_config_tensorrt -iquote external/mkl_dnn_v1 -iquote bazel-out/k8-opt/bin/external/mkl_dnn_v1 -Ibazel-out/k8-opt/bin/external/local_config_cuda/cuda/_virtual_includes/cuda_headers_virtual -Ibazel-out/k8-opt/bin/external/local_config_tensorrt/_virtual_includes/tensorrt_headers -isystem external/nsync/public -isystem bazel-out/k8-opt/bin/external/nsync/public -isystem third_party/eigen3/mkl_include -isystem bazel-out/k8-opt/bin/third_party/eigen3/mkl_include -isystem external/eigen_archive -isystem bazel-out/k8-opt/bin/external/eigen_archive -isystem external/gif -isystem bazel-out/k8-opt/bin/external/gif -isystem external/com_google_protobuf/src -isystem bazel-out/k8-opt/bin/external/com_google_protobuf/src -isystem external/farmhash_archive/src -isystem bazel-out/k8-opt/bin/external/farmhash_archive/src -isystem external/zlib -isystem bazel-out/k8-opt/bin/external/zlib -isystem external/local_config_cuda/cuda -isystem bazel-out/k8-opt/bin/external/local_config_cuda/cuda -isystem external/local_config_cuda/cuda/cuda/include -isystem bazel-out/k8-opt/bin/external/local_config_cuda/cuda/cuda/include -isystem external/local_config_rocm/rocm -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm -isystem external/local_config_rocm/rocm/rocm/include -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm/rocm/include -isystem external/local_config_rocm/rocm/rocm/include/rocrand -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm/rocm/include/rocrand -isystem external/local_config_rocm/rocm/rocm/include/roctracer -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm/rocm/include/roctracer -isystem external/mkl_dnn_v1/include -isystem bazel-out/k8-opt/bin/external/mkl_dnn_v1/include -isystem external/mkl_dnn_v1/src -isystem bazel-out/k8-opt/bin/external/mkl_dnn_v1/src -isystem external/mkl_dnn_v1/src/common -isystem bazel-out/k8-opt/bin/external/mkl_dnn_v1/src/common -isystem external/mkl_dnn_v1/src/common/ittnotify -isystem bazel-out/k8-opt/bin/external/mkl_dnn_v1/src/common/ittnotify -isystem external/mkl_dnn_v1/src/cpu -isystem bazel-out/k8-opt/bin/external/mkl_dnn_v1/src/cpu -isystem external/mkl_dnn_v1/src/cpu/gemm -isystem bazel-out/k8-opt/bin/external/mkl_dnn_v1/src/cpu/gemm -isystem external/mkl_dnn_v1/src/cpu/x64/xbyak -isystem bazel-out/k8-opt/bin/external/mkl_dnn_v1/src/cpu/x64/xbyak -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fPIC -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -fno-omit-frame-pointer -no-canonical-prefixes -fno-canonical-system-headers -DNDEBUG -g0 -O2 -ffunction-sections -fdata-sections -w -DAUTOLOAD_DYNAMIC_KERNELS -Wno-sign-compare '-std=c++14' -x cuda '-DGOOGLE_CUDA=1' '-Xcuda-fatbinary=--compress-all' '--no-cuda-include-ptx=all' '--cuda-include-ptx=sm_60' '--cuda-gpu-arch=sm_60' '--cuda-include-ptx=sm_61' '--cuda-gpu-arch=sm_61' '--cuda-include-ptx=sm_70' '--cuda-gpu-arch=sm_70' '--cuda-include-ptx=sm_75' '--cuda-gpu-arch=sm_75' -DEIGEN_AVOID_STL_ARRAY -Iexternal/gemmlowp -Wno-sign-compare '-ftemplate-depth=900' -fno-exceptions '-DGOOGLE_CUDA=1' '-DTENSORFLOW_USE_NVCC=1' -DINTEL_MKL -msse3 -pthread '-nvcc_options=relaxed-constexpr' '-nvcc_options=ftz=true' -c tensorflow/core/kernels/image/extract_volume_patches_op_gpu.cu.cc -o bazel-out/k8-opt/bin/tensorflow/core/kernels/image/_objs/extract_volume_patches_op_gpu/extract_volume_patches_op_gpu.cu.pic.o)
Execution platform: @local_execution_config_platform//:platform
INFO: Elapsed time: 1137.195s, Critical Path: 92.10s
INFO: 25995 processes: 11503 internal, 14492 local.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully
Command exited with non-zero status 1

Any other info / logs

Diff from known functional build for 10.2 with 2.4.1 https://github.com/datamachines/cuda_tensorflow_opencv/compare/20210601

Steps to reproduce:

% git clone https://github.com/datamachines/cuda_tensorflow_opencv.git
% cd cuda_tensorflow_opencv
% git checkout 737f860
% make make cudnn_tensorflow_opencv-10.2_2.5.0_3.4.14

Attaching full build log. cudnn_tensorflow_opencv-10.2_2.5.0_3.4.14-20210601.log.txt