tensorflow: nvcc error : 'cudafe++' died with status 0xC0000005 (ACCESS_VIOLATION)

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10 1809 17763.316
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A
TensorFlow installed from (source or binary): source
TensorFlow version: 2.0.0-alpha0
Python version: 3.7.3
Installed using virtualenv? pip? conda?: pip
Bazel version (if compiling from source): 0.23.0
GCC/Compiler version (if compiling from source): Visual Studio 2015 Build tools update 3
CUDA/cuDNN version: CUDA 10.0 / cuDNN 7.5
GPU model and memory: GTX980 4GB

Describe the problem Building from tag 2.0.0-alpha0 results in nvcc error : 'cudafe++' died with status 0xC0000005 (ACCESS_VIOLATION) being thrown on 2 files, repeatably:

tensorflow/core/kernels/bincount_op_gpu.cu.cc
tensorflow/core/kernels/multinomial_op_gpu.cu.cc

Provide the exact sequence of commands / steps that you executed before running into the problem

C:\projects\tensorflow [(v2.0.0-alpha0)]> ./configure
WARNING: Running Bazel server needs to be killed, because the startup options are different.
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
You have bazel 0.23.0 installed.
Please specify the location of python. [Default is C:\Python37\python.exe]:


Found possible Python library paths:
  C:\Python37\lib\site-packages
Please input the desired Python library path to use.  Default is [C:\Python37\lib\site-packages]

Do you wish to build TensorFlow with XLA JIT support? [y/N]: n
No XLA JIT support will be enabled for TensorFlow.

Do you wish to build TensorFlow with ROCm support? [y/N]: n
No ROCm support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: y
CUDA support will be enabled for TensorFlow.

Please specify the CUDA SDK version you want to use. [Leave empty to default to CUDA 10.0]: 10.0


Please specify the location where CUDA 10.0 toolkit is installed. Refer to README.md for more details. [Default is C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.0]:


Please specify the cuDNN version you want to use. [Leave empty to default to cuDNN 7]: 7.5


Please specify the location where cuDNN 7 library is installed. Refer to README.md for more details. [Default is C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.0]:


Please specify a list of comma-separated CUDA compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size, and that TensorFlow only supports compute capabilities >= 3.5 [Default is: 3.5,7.0]: 5.2


Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is /arch:AVX]: /arch:AVX2


Would you like to override eigen strong inline for some C++ compilation to reduce the compilation time? [Y/n]: y
Eigen strong inline overridden.

Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
        --config=mkl            # Build with MKL support.
        --config=monolithic     # Config for mostly static monolithic build.
        --config=gdr            # Build with GDR support.
        --config=verbs          # Build with libverbs support.
        --config=ngraph         # Build with Intel nGraph support.
        --config=numa           # Build with NUMA support.
        --config=dynamic_kernels        # (Experimental) Build kernels into separate shared objects.
Preconfigured Bazel build configs to DISABLE default on features:
        --config=noaws          # Disable AWS S3 filesystem support.
        --config=nogcp          # Disable GCP support.
        --config=nohdfs         # Disable HDFS support.
        --config=noignite       # Disable Apache Ignite support.
        --config=nokafka        # Disable Apache Kafka support.
        --config=nonccl         # Disable NVIDIA NCCL support.

C:\projects\tensorflow [(v2.0.0-alpha0)]> bazel build --config=opt --config=cuda --define=no_tensorflow_py_deps=true //tensorflow/tools/pip_package:build_pip_package

Any other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

ERROR: C:/projects/tensorflow/tensorflow/core/kernels/BUILD:3963:1: C++ compilation of rule '//tensorflow/core/kernels:bincount_op_gpu' failed (Exit 5): python.exe failed: error executing command
  cd C:/users/damlo/_bazel_damlo/fttvoi5m/execroot/org_tensorflow
  SET CUDA_TOOLKIT_PATH=C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.0
    SET CUDNN_INSTALL_PATH=C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.0
    SET INCLUDE=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE;C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared;C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt;
    SET LIB=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\LIB\amd64;C:\Program Files (x86)\Windows Kits\10\lib\10.0.17763.0\ucrt\x64;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64;C:\Program Files (x86)\Windows Kits\10\lib\10.0.17763.0\um\x64;
    SET PATH=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\amd64;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\VCPackages;C:\WINDOWS\Microsoft.NET\Framework64\v4.0.30319;C:\WINDOWS\Microsoft.NET\Framework64\;C:\Program Files (x86)\Windows Kits\10\bin\x64;C:\Program Files (x86)\Windows Kits\10\bin\x86;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools\x64\;;C:\WINDOWS\system32
    SET PWD=/proc/self/cwd
    SET PYTHON_BIN_PATH=C:/Python37/python.exe
    SET PYTHON_LIB_PATH=C:/Python37/lib/site-packages
    SET TEMP=C:\Users\damlo\AppData\Local\Temp
    SET TF_CUDA_CLANG=0
    SET TF_CUDA_COMPUTE_CAPABILITIES=5.2
    SET TF_CUDA_VERSION=10.0
    SET TF_CUDNN_VERSION=7
    SET TF_NEED_CUDA=1
    SET TF_NEED_OPENCL_SYCL=0
    SET TF_NEED_ROCM=0
    SET TMP=C:\Users\damlo\AppData\Local\Temp
  C:/Python37/python.exe -B external/local_config_cuda/crosstool/windows/msvc_wrapper_for_nvcc.py /nologo /DCOMPILER_MSVC /DNOMINMAX /D_WIN32_WINNT=0x0600 /D_CRT_SECURE_NO_DEPRECATE /D_CRT_SECURE_NO_WARNINGS /D_SILENCE_STDEXT_HASH_DEPRECATION_WARNINGS /bigobj /Zm500 /J /Gy /GF /EHsc /wd4351 /wd4291 /wd4250 /wd4996 /I. /Ibazel-out/x64_windows-opt/genfiles /Ibazel-out/x64_windows-opt/bin /Iexternal/com_google_absl /Ibazel-out/x64_windows-opt/genfiles/external/com_google_absl /Ibazel-out/x64_windows-opt/bin/external/com_google_absl /Iexternal/eigen_archive /Ibazel-out/x64_windows-opt/genfiles/external/eigen_archive /Ibazel-out/x64_windows-opt/bin/external/eigen_archive /Iexternal/local_config_sycl /Ibazel-out/x64_windows-opt/genfiles/external/local_config_sycl /Ibazel-out/x64_windows-opt/bin/external/local_config_sycl /Iexternal/nsync /Ibazel-out/x64_windows-opt/genfiles/external/nsync /Ibazel-out/x64_windows-opt/bin/external/nsync /Iexternal/gif_archive /Ibazel-out/x64_windows-opt/genfiles/external/gif_archive /Ibazel-out/x64_windows-opt/bin/external/gif_archive /Iexternal/jpeg /Ibazel-out/x64_windows-opt/genfiles/external/jpeg /Ibazel-out/x64_windows-opt/bin/external/jpeg /Iexternal/protobuf_archive /Ibazel-out/x64_windows-opt/genfiles/external/protobuf_archive /Ibazel-out/x64_windows-opt/bin/external/protobuf_archive /Iexternal/com_googlesource_code_re2 /Ibazel-out/x64_windows-opt/genfiles/external/com_googlesource_code_re2 /Ibazel-out/x64_windows-opt/bin/external/com_googlesource_code_re2 /Iexternal/farmhash_archive /Ibazel-out/x64_windows-opt/genfiles/external/farmhash_archive /Ibazel-out/x64_windows-opt/bin/external/farmhash_archive /Iexternal/fft2d /Ibazel-out/x64_windows-opt/genfiles/external/fft2d /Ibazel-out/x64_windows-opt/bin/external/fft2d /Iexternal/highwayhash /Ibazel-out/x64_windows-opt/genfiles/external/highwayhash /Ibazel-out/x64_windows-opt/bin/external/highwayhash /Iexternal/zlib_archive /Ibazel-out/x64_windows-opt/genfiles/external/zlib_archive /Ibazel-out/x64_windows-opt/bin/external/zlib_archive /Iexternal/double_conversion /Ibazel-out/x64_windows-opt/genfiles/external/double_conversion /Ibazel-out/x64_windows-opt/bin/external/double_conversion /Iexternal/snappy /Ibazel-out/x64_windows-opt/genfiles/external/snappy /Ibazel-out/x64_windows-opt/bin/external/snappy /Iexternal/local_config_cuda /Ibazel-out/x64_windows-opt/genfiles/external/local_config_cuda /Ibazel-out/x64_windows-opt/bin/external/local_config_cuda /Iexternal/cub_archive /Ibazel-out/x64_windows-opt/genfiles/external/cub_archive /Ibazel-out/x64_windows-opt/bin/external/cub_archive /Ibazel-out/x64_windows-opt/bin/external/cub_archive/_virtual_includes/cub /Iexternal/eigen_archive /Ibazel-out/x64_windows-opt/genfiles/external/eigen_archive /Ibazel-out/x64_windows-opt/bin/external/eigen_archive /Iexternal/nsync/public /Ibazel-out/x64_windows-opt/genfiles/external/nsync/public /Ibazel-out/x64_windows-opt/bin/external/nsync/public /Iexternal/gif_archive/lib /Ibazel-out/x64_windows-opt/genfiles/external/gif_archive/lib /Ibazel-out/x64_windows-opt/bin/external/gif_archive/lib /Iexternal/gif_archive/windows /Ibazel-out/x64_windows-opt/genfiles/external/gif_archive/windows /Ibazel-out/x64_windows-opt/bin/external/gif_archive/windows /Iexternal/protobuf_archive/src /Ibazel-out/x64_windows-opt/genfiles/external/protobuf_archive/src /Ibazel-out/x64_windows-opt/bin/external/protobuf_archive/src /Iexternal/farmhash_archive/src /Ibazel-out/x64_windows-opt/genfiles/external/farmhash_archive/src /Ibazel-out/x64_windows-opt/bin/external/farmhash_archive/src /Iexternal/zlib_archive /Ibazel-out/x64_windows-opt/genfiles/external/zlib_archive /Ibazel-out/x64_windows-opt/bin/external/zlib_archive /Iexternal/double_conversion /Ibazel-out/x64_windows-opt/genfiles/external/double_conversion /Ibazel-out/x64_windows-opt/bin/external/double_conversion /Iexternal/local_config_cuda/cuda /Ibazel-out/x64_windows-opt/genfiles/external/local_config_cuda/cuda /Ibazel-out/x64_windows-opt/bin/external/local_config_cuda/cuda /Iexternal/local_config_cuda/cuda/cuda/include /Ibazel-out/x64_windows-opt/genfiles/external/local_config_cuda/cuda/cuda/include /Ibazel-out/x64_windows-opt/bin/external/local_config_cuda/cuda/cuda/include /Iexternal/local_config_cuda/cuda/cuda/include/crt /Ibazel-out/x64_windows-opt/genfiles/external/local_config_cuda/cuda/cuda/include/crt /Ibazel-out/x64_windows-opt/bin/external/local_config_cuda/cuda/cuda/include/crt /D__CLANG_SUPPORT_DYN_ANNOTATION__ /DEIGEN_MPL2_ONLY /DEIGEN_MAX_ALIGN_BYTES=64 /DEIGEN_HAS_TYPE_TRAITS=0 /DTF_USE_SNAPPY /showIncludes /MD /O2 /DNDEBUG -w -DWIN32_LEAN_AND_MEAN /arch:AVX2 -DGOOGLE_CUDA=1 -DTENSORFLOW_MONOLITHIC_BUILD /DPLATFORM_WINDOWS /DEIGEN_HAS_C99_MATH /DTENSORFLOW_USE_EIGEN_THREADPOOL /DEIGEN_AVOID_STL_ARRAY /Iexternal/gemmlowp /wd4018 /wd4577 /DNOGDI /DTF_COMPILE_LIBRARY -x cuda -DGOOGLE_CUDA=1 -nvcc_options=relaxed-constexpr -nvcc_options=ftz=true /Fobazel-out/x64_windows-opt/bin/tensorflow/core/kernels/_objs/bincount_op_gpu/bincount_op_gpu.cu.o /c tensorflow/core/kernels/bincount_op_gpu.cu.cc
Execution platform: @bazel_tools//platforms:host_platform
external/com_google_absl\absl/strings/string_view.h(496): warning: expression has no effect

c:\users\damlo\_bazel_damlo\fttvoi5m\execroot\org_tensorflow\external\eigen_archive\eigen\src/Core/arch/GPU/PacketMathHalf.h(149): warning: missing return statement at end of non-void function "Eigen::internal::ptrue(const Packet &) [with Packet=half2]"

c:\users\damlo\_bazel_damlo\fttvoi5m\execroot\org_tensorflow\external\eigen_archive\eigen\src/Core/products/Parallelizer.h(24): warning: variable "m_maxThreads" was set but never used

external/protobuf_archive/src\google/protobuf/arena_impl.h(55): warning: integer conversion resulted in a change of sign

external/protobuf_archive/src\google/protobuf/arena_impl.h(309): warning: integer conversion resulted in a change of sign

external/protobuf_archive/src\google/protobuf/arena_impl.h(310): warning: integer conversion resulted in a change of sign

external/protobuf_archive/src\google/protobuf/map.h(1025): warning: invalid friend declaration

.\tensorflow/core/lib/gtl/flatmap.h(157): warning: invalid friend declaration

external/com_google_absl\absl/strings/string_view.h(496): warning: expression has no effect

external/protobuf_archive/src\google/protobuf/arena_impl.h(55): warning: integer conversion resulted in a change of sign

external/protobuf_archive/src\google/protobuf/arena_impl.h(309): warning: integer conversion resulted in a change of sign

external/protobuf_archive/src\google/protobuf/arena_impl.h(310): warning: integer conversion resulted in a change of sign

external/protobuf_archive/src\google/protobuf/map.h(1025): warning: invalid friend declaration

.\tensorflow/core/lib/gtl/flatmap.h(157): warning: invalid friend declaration

nvcc error   : 'cudafe++' died with status 0xC0000005 (ACCESS_VIOLATION)
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 359.982s, Critical Path: 193.33s
INFO: 253 processes: 253 local.
FAILED: Build did NOT complete successfully

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 1
Comments: 16 (8 by maintainers)

Most upvoted comments

You can “patch” nvcc yourself and still use cuda 10.0

Download and install CUDA 10.1 update 1, but check only to install nvcc. Rename C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\cudafe++.exe cudafe++.exe.v10.0 Copy C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin\cudafe++.exe to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\cudafe++.exe

This fixed this error. I had a few other errors I had to fix as well before building successfully (building tensorflow branch r2.0 with GPU support, Win 10 19.03, Anaconda 3.7 in a conda virtual environment, Visual Studio 2017, CUDA 10, CUDNN 7.6, bazel 0.26.0, GTX-1080 Ti, RTX-2080 Ti, Compute 6.1,7.5):

Modify the WORKSPACE file to avoid an error on building from bazel:

edit WORKSPACE

Under the line near the top that reads “load(”@bazel_tools…" ADD:

http_archive( name = “io_bazel_rules_docker”, sha256 = “aed1c249d4ec8f703edddf35cbe9dfaca0b5f5ea6e4cd9e83e99f3b0d1136c3d”, strip_prefix = “rules_docker-0.7.0”, urls = [“https://github.com/bazelbuild/rules_docker/archive/v0.7.0.tar.gz”], )

Check this is still the correct lines to use at https://github.com/bazelbuild/rules_docker

Avoid the cudart64_.dll not found error Create a copy of: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\cudart64_100.dll as cudart64_.dll in the same folder i.e. I kept both the original file and the renamed file in the same folder.
Avoid the cudnn64_.dll not found error Create a copy of: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\cudnn64_100.dll as cudnn64_.dll in the same folder

bazel build --config=opt --config=cuda --define=no_tensorflow_py_deps=true --copt=-nvcc_options=disable-warnings //tensorflow/tools/pip_package:build_pip_package

+18

dbonner on Jun 22, 2019

@dbonner Thanks for the sharing these details! I managed to build TF 1.14 for Windows using the CUDA 10.0 + cudafe++ from CUDA 10.1 trick!

The cudart_.dll, cudnn_.dll problems seems to be a problem in this build_info genrule (tensorflow.bzl):

def tf_py_build_info_genrule():
    native.genrule(
        name = "py_build_info_gen",
        outs = ["platform/build_info.py"],
        cmd =
            "$(location //tensorflow/tools/build_info:gen_build_info) --raw_generate \"$@\" --build_config " +
            if_cuda("cuda", "cpu") +
            " --key_value " +
            if_cuda(" cuda_version_number=$${TF_CUDA_VERSION:-} cudnn_version_number=$${TF_CUDNN_VERSION:-} ", "") +
            if_windows(" msvcp_dll_name=msvcp140.dll ", "") +
            if_windows_cuda(" ".join([
                "nvcuda_dll_name=nvcuda.dll",
                "cudart_dll_name=cudart64_$$(echo $${TF_CUDA_VERSION:-} | sed \"s/\\.//\").dll",
                "cudnn_dll_name=cudnn64_$${TF_CUDNN_VERSION:-}.dll",
            ]), ""),
        local = 1,
        tools = [clean_dep("//tensorflow/tools/build_info:gen_build_info")],
    )

AFACT, the shell expansion works only when starting bazel from a MinGW Bash shell, not from the command prompt. Or at least in my case, I was unable to build using cmd.exe but once I got the build configured with “Git Bash”, the cudnn_.dll/cudart_.dll problem went away.

nurpax on Sep 4, 2019

I’m running into this problem on r1.14 + CUDA 10.0, with tensorflow/core/kernels/tile_functor_gpu_int64.cu.cc

Wondering how TensorFlow didn’t run into that for their Windows builds…

Will try the 10.1 nvcc workaround.

reuben on Jun 27, 2019