jax: Failure to build jaxlib v0.1.62 on Windows (Updated)

The original thread #5981 was closed due to a proposed fix. However, even with this in place, using the same build parameters, it still fails to build, but much further into the build process.

     _   _  __  __
    | | / \ \ \/ /
 _  | |/ _ \ \  /
| |_| / ___ \/  \
 \___/_/   \/_/\_\


Starting local Bazel server and connecting to it...
Bazel binary path: C:\bazel\bazel.EXE
Python binary path: C:/Users/Adam/anaconda3/python.exe
Python version: 3.8
MKL-DNN enabled: yes
Target CPU features: release
CUDA enabled: yes
CUDA toolkit path: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.0
CUDNN library path: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.0
CUDA compute capabilities: 7.5
CUDA version: 11.0
CUDNN version: 8.0.5
TPU enabled: no
ROCm enabled: no

Building XLA and installing it in the jaxlib source tree...
C:\bazel\bazel.EXE run --verbose_failures=true --config=short_logs --config=mkl_open_source_only --config=cuda --define=xla_python_enable_gpu=true :build_wheel -- --output_path=C:\sdks\jax-jaxlib-v0.1.62\dist
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=80
INFO: Reading rc options for 'run' from c:\sdks\jax-jaxlib-v0.1.62\.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Options provided by the client:
  Inherited 'build' options: --python_path=C:/Users/Adam/anaconda3/python.exe
INFO: Reading rc options for 'run' from c:\sdks\jax-jaxlib-v0.1.62\.bazelrc:
  Inherited 'build' options: --repo_env PYTHON_BIN_PATH=C:/Users/Adam/anaconda3/python.exe --action_env=PYENV_ROOT --python_path=C:/Users/Adam/anaconda3/python.exe --repo_env TF_NEED_CUDA=1 --action_env TF_CUDA_COMPUTE_CAPABILITIES=7.5 --repo_env TF_NEED_ROCM=0 --action_env TF_ROCM_AMDGPU_TARGETS=gfx803,gfx900,gfx906,gfx1010 --distinct_host_configuration=false -c opt --apple_platform_type=macos --macos_minimum_os=10.9 --announce_rc --define open_source_build=true --define=no_kafka_support=true --define=no_ignite_support=true --define=grpc_no_ares=true --spawn_strategy=standalone --strategy=Genrule=standalone --enable_platform_specific_config --action_env CUDA_TOOLKIT_PATH=C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.0 --action_env CUDNN_INSTALL_PATH=C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.0 --action_env TF_CUDA_VERSION=11.0 --action_env TF_CUDNN_VERSION=8.0.5
INFO: Found applicable config definition build:short_logs in file c:\sdks\jax-jaxlib-v0.1.62\.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:mkl_open_source_only in file c:\sdks\jax-jaxlib-v0.1.62\.bazelrc: --define=tensorflow_mkldnn_contraction_kernel=1
INFO: Found applicable config definition build:cuda in file c:\sdks\jax-jaxlib-v0.1.62\.bazelrc: --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:windows in file c:\sdks\jax-jaxlib-v0.1.62\.bazelrc: --copt=/D_USE_MATH_DEFINES --host_copt=/D_USE_MATH_DEFINES --copt=-DWIN32_LEAN_AND_MEAN --host_copt=-DWIN32_LEAN_AND_MEAN --copt=-DNOGDI --host_copt=-DNOGDI --copt=/Zc:preprocessor --cxxopt=/std:c++14 --host_cxxopt=/std:c++14 --linkopt=/DEBUG --host_linkopt=/DEBUG --linkopt=/OPT:REF --host_linkopt=/OPT:REF --linkopt=/OPT:ICF --host_linkopt=/OPT:ICF --experimental_strict_action_env=true
DEBUG: Rule 'io_bazel_rules_docker' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1556410077 -0400"
DEBUG: Repository io_bazel_rules_docker instantiated at:
  C:/sdks/jax-jaxlib-v0.1.62/WORKSPACE:34:10: in <toplevel>
  C:/users/adam/_bazel_adam/nzquhzn2/external/org_tensorflow/tensorflow/workspace0.bzl:105:34: in workspace
  C:/users/adam/_bazel_adam/nzquhzn2/external/bazel_toolchains/repositories/repositories.bzl:37:23: in repositories
Repository rule git_repository defined at:
  C:/users/adam/_bazel_adam/nzquhzn2/external/bazel_tools/tools/build_defs/repo/git.bzl:199:33: in <toplevel>
INFO: Analyzed target //build:build_wheel (182 packages loaded, 15809 targets configured).
INFO: Found 1 target...
ERROR: C:/users/adam/_bazel_adam/nzquhzn2/external/org_tensorflow/tensorflow/core/tpu/BUILD:105:11: C++ compilation of rule '@org_tensorflow//tensorflow/core/tpu:tpu_initializer_helper' failed (Exit 2): python.exe failed: error executing command
  cd C:/users/adam/_bazel_adam/nzquhzn2/execroot/__main__
  SET CUDA_TOOLKIT_PATH=C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.0
    SET CUDNN_INSTALL_PATH=C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.0
    SET INCLUDE=C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.28.29333\ATLMFC\include;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.28.29333\include;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt;C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt
    SET LIB=C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.28.29333\ATLMFC\lib\x64;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.28.29333\lib\x64;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64;C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64;C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64
    SET PATH=C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\\Extensions\Microsoft\IntelliCode\CLI;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.28.29333\bin\HostX64\x64;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\VC\VCPackages;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\bin\Roslyn;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Team Tools\Performance Tools\x64;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Team Tools\Performance Tools;C:\Program Files (x86)\Microsoft Visual Studio\Shared\Common\VSPerfCollectionTools\vs2019\\x64;C:\Program Files (x86)\Microsoft Visual Studio\Shared\Common\VSPerfCollectionTools\vs2019\;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\x64\;C:\Program Files (x86)\Windows Kits\10\bin\10.0.18362.0\x64;C:\Program Files (x86)\Windows Kits\10\bin\x64;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\\MSBuild\Current\Bin;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\Tools\;;C:\WINDOWS\system32;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja
    SET PWD=/proc/self/cwd
    SET RUNFILES_MANIFEST_ONLY=1
    SET TEMP=C:\Users\Adam\AppData\Local\Temp
    SET TF_CUDA_COMPUTE_CAPABILITIES=7.5
    SET TF_CUDA_VERSION=11.0
    SET TF_CUDNN_VERSION=8.0.5
    SET TF_ROCM_AMDGPU_TARGETS=gfx803,gfx900,gfx906,gfx1010
    SET TMP=C:\Users\Adam\AppData\Local\Temp
  C:/Users/Adam/anaconda3/python.exe -B external/local_config_cuda/crosstool/windows/msvc_wrapper_for_nvcc.py /nologo /DCOMPILER_MSVC /DNOMINMAX /D_WIN32_WINNT=0x0600 /D_CRT_SECURE_NO_DEPRECATE /D_CRT_SECURE_NO_WARNINGS /D_SILENCE_STDEXT_HASH_DEPRECATION_WARNINGS /bigobj /Zm500 /J /Gy /GF /EHsc /wd4351 /wd4291 /wd4250 /wd4996 /Iexternal/org_tensorflow /Ibazel-out/x64_windows-opt/bin/external/org_tensorflow /Iexternal/eigen_archive /Ibazel-out/x64_windows-opt/bin/external/eigen_archive /Iexternal/com_google_absl /Ibazel-out/x64_windows-opt/bin/external/com_google_absl /Iexternal/nsync /Ibazel-out/x64_windows-opt/bin/external/nsync /Iexternal/eigen_archive /Ibazel-out/x64_windows-opt/bin/external/eigen_archive /Iexternal/nsync/public /Ibazel-out/x64_windows-opt/bin/external/nsync/public /DEIGEN_MPL2_ONLY /DEIGEN_MAX_ALIGN_BYTES=64 /D__CLANG_SUPPORT_DYN_ANNOTATION__ /showIncludes /MD /O2 /DNDEBUG /D_USE_MATH_DEFINES -DWIN32_LEAN_AND_MEAN -DNOGDI /Zc:preprocessor /std:c++14 /Fobazel-out/x64_windows-opt/bin/external/org_tensorflow/tensorflow/core/tpu/_objs/tpu_initializer_helper/tpu_initializer_helper.obj /c external/org_tensorflow/tensorflow/core/tpu/tpu_initializer_helper.cc
Execution platform: @local_execution_config_platform//:platform
Target //build:build_wheel failed to build
INFO: Elapsed time: 1450.560s, Critical Path: 375.03s
INFO: 5002 processes: 2127 internal, 2875 local.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully
Traceback (most recent call last):
  File ".\build\build.py", line 521, in <module>
    main()
  File ".\build\build.py", line 516, in main
    shell(command)
  File ".\build\build.py", line 51, in shell
    output = subprocess.check_output(cmd)
  File "C:\Users\Adam\anaconda3\lib\subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "C:\Users\Adam\anaconda3\lib\subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['C:\\bazel\\bazel.EXE', 'run', '--verbose_failures=true', '--config=short_logs', '--config=mkl_open_source_only', '--config=cuda', '--define=xla_python_enable_gpu=true', ':build_wheel', '--', '--output_path=C:\\sdks\\jax-jaxlib-v0.1.62\\dist']' returned non-zero exit status 1.

_Originally posted by @oracle3001 in https://github.com/google/jax/issues/5981#issuecomment-793318112_

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 41 (11 by maintainers)

Commits related to this issue

Most upvoted comments

jaxlib-v.0.1.65 with CUDA 11.0 on Windows 10 using Bazel 3.7.0 has built successfully for me after following the instructions of from Skye above to alter the WORKSPACE file.

@erwincoumans https://github.com/google/jax/pull/6427 bumps the XLA version. I think that’s all you should need.

Sounds like we can close this issue once that PR is merged?

No, sorry for the delay. I have to do some refactoring to make linking in this file optional (so you can exclude it on Windows) and haven’t had time yet.

It appears this is related to https://github.com/tensorflow/tensorflow/issues/15423 and MSYS is required to run Bazel

I seems to have the same compile failure.

     _   _  __  __
    | | / \ \ \/ /
 _  | |/ _ \ \  /
| |_| / ___ \/  \
 \___/_/   \/_/\_\
Starting local Bazel server and connecting to it...
Bazel binary path: C:\ProgramData\chocoportable\lib\bazel\bazel.EXE
Python binary path: C:/Users/userName/AppData/Local/Programs/Python/Python38/python.exe
Python version: 3.8
MKL-DNN enabled: yes
Target CPU features: release
CUDA enabled: no
TPU enabled: no
ROCm enabled: no
Building XLA and installing it in the jaxlib source tree...
C:\ProgramData\chocoportable\lib\bazel\bazel.EXE run --verbose_failures=true --config=short_logs --config=mkl_open_source_only :build_wheel -- --output_path=C:\Users\userName\Documents\Developer\Python-Workspace\jax\dist
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=80
INFO: Reading rc options for 'run' from c:\users\userName\documents\developer\python-workspace\jax\.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Options provided by the client:
  Inherited 'build' options: --python_path=C:/Users/userName/AppData/Local/Programs/Python/Python38/python.exe
INFO: Reading rc options for 'run' from c:\users\userName\documents\developer\python-workspace\jax\.bazelrc:
  Inherited 'build' options: --repo_env PYTHON_BIN_PATH=C:/Users/userName/AppData/Local/Programs/Python/Python38/python.exe --action_env=PYENV_ROOT --python_path=C:/Users/userName/AppData/Local/Programs/Python/Python38/python.exe --repo_env TF_NEED_CUDA=0 --action_env TF_CUDA_COMPUTE_CAPABILITIES=3.5,5.2,6.0,6.1,7.0 --repo_env TF_NEED_ROCM=0 --action_env TF_ROCM_AMDGPU_TARGETS=gfx803,gfx900,gfx906,gfx1010 --distinct_host_configuration=false -c opt --apple_platform_type=macos --macos_minimum_os=10.9 --announce_rc --define open_source_build=true --define=no_kafka_support=true --define=no_ignite_support=true --define=grpc_no_ares=true --spawn_strategy=standalone --strategy=Genrule=standalone --enable_platform_specific_config
INFO: Found applicable config definition build:short_logs in file c:\users\userName\documents\developer\python-workspace\jax\.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:mkl_open_source_only in file c:\users\userName\documents\developer\python-workspace\jax\.bazelrc: --define=tensorflow_mkldnn_contraction_kernel=1
INFO: Found applicable config definition build:windows in file c:\users\userName\documents\developer\python-workspace\jax\.bazelrc: --copt=/D_USE_MATH_DEFINES --host_copt=/D_USE_MATH_DEFINES --copt=-DWIN32_LEAN_AND_MEAN --host_copt=-DWIN32_LEAN_AND_MEAN --copt=-DNOGDI --host_copt=-DNOGDI --copt=/Zc:preprocessor --cxxopt=/std:c++14 --host_cxxopt=/std:c++14 --linkopt=/DEBUG --host_linkopt=/DEBUG --linkopt=/OPT:REF --host_linkopt=/OPT:REF --linkopt=/OPT:ICF --host_linkopt=/OPT:ICF --experimental_strict_action_env=true
DEBUG: Rule 'io_bazel_rules_docker' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1556410077 -0400"
DEBUG: Repository io_bazel_rules_docker instantiated at:
  C:/users/userName/documents/developer/python-workspace/jax/WORKSPACE:34:10: in <toplevel>
  C:/users/userName/_bazel_userName/msjayd3l/external/org_tensorflow/tensorflow/workspace0.bzl:105:34: in workspace
  C:/users/userName/_bazel_userName/msjayd3l/external/bazel_toolchains/repositories/repositories.bzl:37:23: in repositories
Repository rule git_repository defined at:
  C:/users/userName/_bazel_userName/msjayd3l/external/bazel_tools/tools/build_defs/repo/git.bzl:199:33: in <toplevel>
INFO: Analyzed target //build:build_wheel (158 packages loaded, 13149 targets configured).
INFO: Found 1 target...
ERROR: C:/users/userName/_bazel_userName/msjayd3l/external/org_tensorflow/tensorflow/core/tpu/BUILD:105:11: Compiling tensorflow/core/tpu/tpu_initializer_helper.cc failed: (Exit 2): cl.exe failed: error executing command
  cd C:/users/userName/_bazel_userName/msjayd3l/execroot/__main__
  SET INCLUDE=C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29910\include;C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt;C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared;C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt;C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt
    SET PATH=C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29910\bin\HostX64\x64;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\VC\VCPackages;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Current\bin\Roslyn;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\Tools\devinit;C:\Program Files (x86)\Windows Kits\10\bin\10.0.19041.0\x64;C:\Program Files (x86)\Windows Kits\10\bin\x64;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\\MSBuild\Current\Bin;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\Tools\;;C:\WINDOWS\system32;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja
    SET PWD=/proc/self/cwd
    SET RUNFILES_MANIFEST_ONLY=1
    SET TEMP=C:\Users\USERNAME~1.KEL\AppData\Local\Temp
    SET TF_CUDA_COMPUTE_CAPABILITIES=3.5,5.2,6.0,6.1,7.0
    SET TF_ROCM_AMDGPU_TARGETS=gfx803,gfx900,gfx906,gfx1010
    SET TMP=C:\Users\USERNAME~1.KEL\AppData\Local\Temp
  C:/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Tools/MSVC/14.28.29910/bin/HostX64/x64/cl.exe /nologo /DCOMPILER_MSVC /DNOMINMAX /D_WIN32_WINNT=0x0601 /D_CRT_SECURE_NO_DEPRECATE /D_CRT_SECURE_NO_WARNINGS /bigobj /Zm500 /EHsc /wd4351 /wd4291 /wd4250 /wd4996 /Iexternal/org_tensorflow /Ibazel-out/x64_windows-opt/bin/external/org_tensorflow /Iexternal/eigen_archive /Ibazel-out/x64_windows-opt/bin/external/eigen_archive /Iexternal/com_google_absl /Ibazel-out/x64_windows-opt/bin/external/com_google_absl /Iexternal/nsync /Ibazel-out/x64_windows-opt/bin/external/nsync /Iexternal/eigen_archive /Ibazel-out/x64_windows-opt/bin/external/eigen_archive /Iexternal/nsync/public /Ibazel-out/x64_windows-opt/bin/external/nsync/public /DEIGEN_MPL2_ONLY /DEIGEN_MAX_ALIGN_BYTES=64 /D__CLANG_SUPPORT_DYN_ANNOTATION__ /showIncludes /MD /O2 /Oy- /DNDEBUG /wd4117 -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" /Gy /Gw /D_USE_MATH_DEFINES -DWIN32_LEAN_AND_MEAN -DNOGDI /Zc:preprocessor /std:c++14 /Fobazel-out/x64_windows-opt/bin/external/org_tensorflow/tensorflow/core/tpu/_objs/tpu_initializer_helper/tpu_initializer_helper.obj /c external/org_tensorflow/tensorflow/core/tpu/tpu_initializer_helper.cc
Execution platform: @local_execution_config_platform//:platform
Target //build:build_wheel failed to build
INFO: Elapsed time: 4474.828s, Critical Path: 156.40s
INFO: 5414 processes: 1095 internal, 4319 local.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully
Traceback (most recent call last):
  File ".\build\build.py", line 521, in <module>
    main()
  File ".\build\build.py", line 516, in main
    shell(command)
  File ".\build\build.py", line 51, in shell
    output = subprocess.check_output(cmd)
  File "C:\Users\userName\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "C:\Users\userName\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['C:\\ProgramData\\chocoportable\\lib\\bazel\\bazel.EXE', 'run', '--verbose_failures=true', '--config=short_logs', '--config=mkl_open_source_only', ':build_wheel', '--', '--output_path=C:\\Users\\userName\\Documents\\Developer\\Python-Workspace\\jax\\dist']' returned non-zero exit status 1.
PS C:\Users\userName\Documents\Developer\Python-Workspace\jax>```