TheRock: [Issue]: Error when building ROCm/PyTorch on Windows (7800 XT)

Problem Description

I followed all the instructions in here, using chocolatey for all the tools.

I get this error that cuts the build about 30/40 mins in. Python version is 3.12.3.

[200/235] Building sub-project rocBLAS (in background)
FAILED: math-libs/BLAS/rocBLAS/stamp/build.stamp I:/TheRock/build/math-libs/BLAS/rocBLAS/stamp/build.stamp
C:\WINDOWS\system32\cmd.exe /C "cd /D I:\TheRock\build\math-libs\BLAS\rocBLAS\build && I:\TheRock\.venv\Scripts\python.exe I:/TheRock/build_tools/teatime.py --log-timestamps --label rocBLAS --interactive I:/TheRock/build/logs/rocBLAS_build.log -- "C:/Program Files/CMake/bin/cmake.exe" -E env -- "C:/Program Files/CMake/bin/cmake.exe" --build I:/TheRock/build/math-libs/BLAS/rocBLAS/build && "C:\Program Files\CMake\bin\cmake.exe" -E touch I:/TheRock/build/math-libs/BLAS/rocBLAS/stamp/build.stamp"
[rocBLAS] [0/2] Re-checking globbed directories...
[rocBLAS] [0/628] Generating libraries with TensileCreateLibrary
[rocBLAS] > UserWarning: Found `clang++.exe` but in a non-default ROCm location:
[rocBLAS] I:\TheRock\build\core\clr\dist\lib\llvm\bin\clang++.exe
[rocBLAS] > UserWarning: Did not detect SupportedISA: [(8, 0, 3), (9, 0, 0), (9, 0, 6),
[rocBLAS] (9, 0, 8), (9, 0, 10), (9, 4, 2), (9, 5, 0), (10, 1, 0), (10, 1, 1), (10, 1,
[rocBLAS] 2), (10, 3, 0), (10, 3, 1), (11, 0, 0), (11, 0, 1), (11, 0, 2), (11, 5, 1),
[rocBLAS] (12, 0, 0), (12, 0, 1)]; cannot benchmark assembly kernels.
[rocBLAS] > UserWarning: Failed to detect ISA so forcing (gfx906) on windows
[rocBLAS] Traceback (most recent call last):
[rocBLAS]   File "I:\TheRock\build\math-libs\BLAS\rocBLAS\build\virtualenv\Lib\site-packages\Tensile\bin\TensileCreateLibrary", line 43, in <module>
[rocBLAS]     TensileCreateLibrary()
[rocBLAS]   File "I:\TheRock\build\math-libs\BLAS\rocBLAS\build\virtualenv\Lib\site-packages\Tensile\TensileCreateLibrary.py", line 1379, in TensileCreateLibrary
[rocBLAS]     assignGlobalParameters(args)
[rocBLAS]   File "I:\TheRock\build\math-libs\BLAS\rocBLAS\build\virtualenv\Lib\site-packages\Tensile\Common.py", line 2482, in assignGlobalParameters
[rocBLAS]     *[int(c.split("-")[0]) for c in globalParameters["HipClangVersion"].split(".")[:3]]
[rocBLAS]       ^^^^^^^^^^^^^^^^^^^^
[rocBLAS] ValueError: invalid literal for int() with base 10: 'hipconfig'
[rocBLAS] [11/628] Generating prototypes from I:/TheRock/math-libs/BLAS/rocBLAS/library/src.
[rocBLAS] FAILED: Tensile/library I:/TheRock/build/math-libs/BLAS/rocBLAS/build/Tensile/library
[rocBLAS] C:\WINDOWS\system32\cmd.exe /C "cd /D I:\TheRock\build\math-libs\BLAS\rocBLAS\build\library\src && "C:\Program Files\CMake\bin\cmake.exe" -E env "PATH=I:/TheRock/build/core/clr/dist/bin;I:/TheRock/build/core/clr/dist/lib/llvm/bin;I:\TheRock\.venv\Scripts;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\HostX64\x64;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Current\bin\Roslyn;C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\\x64;C:\Program Files (x86)\Windows Kits\10\bin\\x64;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\\MSBuild\Current\Bin\amd64;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\Tools\;C:\Python312\Scripts\;C:\Python312\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\dotnet\;C:\Program Files\Apollo;C:\Program Files\Apollo\tools;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\chocolatey\bin;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Program Files\CMake\bin;C:\Strawberry\c\bin;C:\Strawberry\perl\site\bin;C:\Strawberry\perl\bin;C:\Users\Max\AppData\Local\Microsoft\WindowsApps;C:\Users\Max\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\Max\AppData\Local\GitHubDesktop\bin;C:\Users\Max\AppData\Local\Google\Cloud SDK\google-cloud-sdk\bin;C:\Program Files\AMD\ROCm\6.2\bin;C:\ZLUDA\zluda;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja" -- I:/TheRock/build/math-libs/BLAS/rocBLAS/build/virtualenv/Scripts/python.exe I:/TheRock/build/math-libs/BLAS/rocBLAS/build/virtualenv/Lib/site-packages/Tensile/bin/TensileCreateLibrary --merge-files --separate-architectures --lazy-library-loading --no-short-file-names --verbose=0 --code-object-version=default --cxx-compiler=I:/TheRock/build/core/clr/dist/lib/llvm/bin/clang++.exe --library-format=msgpack --architecture=gfx1100_gfx1101_gfx1102 --no-enumerate I:/TheRock/math-libs/BLAS/rocBLAS/library/src/blas3/Tensile/Logic/asm_full I:/TheRock/build/math-libs/BLAS/rocBLAS/build/Tensile HIP"

Operating System

Windows 11

CPU

7700x

GPU

7800 xt

ROCm Version

latest

ROCm Component

rocBLAS

Steps to Reproduce

No response

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

PS C:\Users\Max> (Get-WmiObject Win32_OperatingSystem).Version
10.0.26100
PS C:\Users\Max>   (Get-WmiObject win32_Processor).Name
AMD Ryzen 7 7700X 8-Core Processor
PS C:\Users\Max>   (Get-WmiObject win32_VideoController).Name
AMD  Radeon RX 7800 XT

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 52 (40 by maintainers)

Commits related to this issue

Most upvoted comments

Btw AMD has announced that ROCm would officially support running PyTorch on Windows at 2025Q3.

Now that we’re saying that out loud, I can provide some additional details.

There will be nightlies of it on here long before that, and there is still the question of what the intercept with what you can get from pytorch.org for which architectures is. How we message what is “supported” is still a pretty evolving topic. The model we’re moving to is just continuous releases of latest “core” rocm and pytorch on Linux and Windows with an expanding set of gfx architectures that we keep building out.

There will also be regular “cuts” of that which are given a new version number and distributed via all channels with a larger set of rocm components (and likely a smaller set of supported gfx arches, since we are capacity limited on how much we can support in an “everything” build). These are what you currently see as ROCm releases. These are what we will continue to talk about in terms of official support, but actual support will be available both more broadly and sooner through our community release channels (which are the precursors to the official releases).

We still don’t quite know how to talk about release plans like this, which is how you get coarse dates like “Q3” and fuzziness on what “supported” means. I expect the way we communicate this will evolve over time. In the meantime, the approach is roughly what I describe above.

That worked!

Image

Image

looks like a previously installed version of ROCm could be conflicting?

I think we’ve fixed those issues now:

I got a little further by hardcoding the version ("6.5.25201" in lieu of globalParameters["HipClangVersion"]) in Tensile.

Might have incidentally fixed that too, but I’m not sure. I haven’t seen that in my own testing.

I’m adding a few notes to the documentation for now: https://github.com/ROCm/TheRock/pull/660. I’ll have development time later in the week to help land fixes.

BTW, thanks for filing this issue and taking the time to debug!

@ScottTodd Thank you!

Might have incidentally fixed that too, but I’m not sure. I haven’t seen that in my own testing.

I was also experiencing the same hipconfig issue reported in this thread, but it’s resolved now after updating my repository to the latest version.

the build worked!

@ScottTodd I have identified the conditions for reproduction. I will write the details in #670.

@7shi Thanks for the report. I’ve re-opened https://github.com/ROCm/TheRock/issues/670 and we can follow up on HIP_PATH env var or C:\Program Files\AMD\ROCm file path issues there.

@7shi san, @jammm and I have produced a self-contained Pytorch wheel that should support the 7600 XT (gfx1102) here: https://github.com/scottt/rocm-TheRock/releases/download/v6.5.0rc-pytorch-gfx110x/torch-2.7.0a0+gitafb201d-cp312-cp312-win_amd64.whl

Hope that it’s useful to you.

Btw AMD has announced that ROCm would officially support running PyTorch on Windows at 2025Q3.

@MaximilienLC, once you have aotriton_v2.dll built on Windows.

here’s the wheel: https://drive.proton.me/urls/4QRV647ZXC#RPyLKsYAOHIv works in a new venv, onto aotriton and torch friends

@MaximilienLC , I suggest building torch-{vision,audio} without ffmpeg first

  • That’s enough to run stable diffusion and ComfyUI etc
  • The build time options to enable ffmpeg and hardware accelerated decoding is the main finicky part of the build

For building torchvision, make sure that your ROCM_HOME is set to the rocm directory e.g., <TheRock_dir>\build\dist\rocm And set CC=clang-cl and CXX=clang-cl

Then it should just be a matter of python setup.py bdist_wheel -v and check if it’s building device libs in there.

@MaximilienLC, congrats on the successful native Pytorch build! I was just writing to tell you about aotriton on Windows.

Note:

  • The flash attention and memory efficient attention operations in Pytorch, torch.nn.functional.scaled_dot_product_attention, would be slow in your current build
    • This will greatly improve once this work is merged in TheRock
  • huggingface/transformers needs Pytorch to be rebuilt with libuv installed and USE_DISTRIBUTED=ON enabled to work

torchvision builds and runs fine. Don’t know about torchaudio

how should I go about building torchaudio & torchvision though?

@lamikr has been setting torchaudio and torchvision up on Linux and I think he was planning on looking at Windows too. Maybe @scottt or @jammm have tried also. I don’t know what the delta between Linux and Windows looks like for those projects myself yet.

@scottt going to try to follow your aotritron tutorial now

@ScottTodd I’ll try building the fat wheel and share it. how should I go about building torchaudio & torchvision though?

@ScottTodd Disabling WSL worked!

Build completed! Wheels should be at /i/TheRock/external-builds/pytorch/pytorch/dist:
torch-2.7.0a0+gitd7625d2-cp312-cp312-win_amd64.whl

On to the next error:

>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "I:\TheRock\external-builds\pytorch\.venv\Lib\site-packages\torch\__init__.py", line 270, in <module>
    _load_dll_libraries()
  File "I:\TheRock\external-builds\pytorch\.venv\Lib\site-packages\torch\__init__.py", line 266, in _load_dll_libraries
    raise err
OSError: [WinError 126] The specified module could not be found. Error loading "I:\TheRock\external-builds\pytorch\.venv\Lib\site-packages\torch\lib\caffe2_nvrtc.dll" or one of its dependencies.

ptbuild.py will still use patches from https://github.com/ROCm/TheRock/tree/main/external-builds/pytorch/patches/v2.7.0/pytorch/hipified. Need https://github.com/ROCm/TheRock/blob/main/external-builds/pytorch/patches/pytorch/v2.7.0/pytorch/hipified/0013-ROCm-Windows-Enable-USE_ROCM-and-fix-assorted-issues.patch, which I only placed in the new location 🙈

(The plan is to get all patches landed upstream in PyTorch then built from tip of tree instead of v2.7.0 + patches… we’ll get there eventually)

@MaximilienLC , if you’re still using ptbuild.py see: python ptbuild.py checkout --pytorch-ref v2.7.0 in https://gist.github.com/ScottTodd/cb4f9f5a697cce91084977ed3674a4bd

(or wait a few days and let us sort out these last few problems of building Pytorch on Windows)

cool will try

lol i’ve been waiting for months can’t wait a few more days hhhh

@MaximilienLC , if you’re still using ptbuild.py see: python ptbuild.py checkout --pytorch-ref v2.7.0 in https://gist.github.com/ScottTodd/cb4f9f5a697cce91084977ed3674a4bd

(or wait a few days and let us sort out these last few problems of building Pytorch on Windows)

Argh. I need to drop offline now, but that should be using 2.7.0, not 2.6.0. The ptbuild script defaults to checking 2.6.0 and is missing some Windows patches. I’ve been testing with checkout from pytorch_torch_repo.py, which defaults to 2.7.0 and has the latest patches merged in. Need to delete the old patches and set up better defaults.

the build worked!

Did you have to use the hardcoded version in Tensile? I’m getting this issue without having installed hipSDK or another build of rocm.

yep I hardcoded the version that I found

Cool I’ll uninstall it and report back.

I verbatim had the exact same error earlier today. I too have the 7800 XT. Sorry I cant give anymore info as it happened this morning and I never thought to save the compilation outputs.

(.venv) F:\TheRock>cmake --build build --target therock-dist
[0/2] Re-checking globbed directories...
[3/30] Building sub-project rocBLAS (in background)
FAILED: math-libs/BLAS/rocBLAS/stamp/build.stamp F:/TheRock/build/math-libs/BLAS/rocBLAS/stamp/build.stamp
C:\Windows\system32\cmd.exe /C "cd /D F:\TheRock\build\math-libs\BLAS\rocBLAS\build && F:\TheRock\.venv\Scripts\python.exe F:/TheRock/build_tools/teatime.py --log-timestamps --label rocBLAS --interactive F:/TheRock/build/logs/rocBLAS_build.log -- "C:/Program Files/CMake/bin/cmake.exe" -E env -- "C:/Program Files/CMake/bin/cmake.exe" --build F:/TheRock/build/math-libs/BLAS/rocBLAS/build && "C:\Program Files\CMake\bin\cmake.exe" -E touch F:/TheRock/build/math-libs/BLAS/rocBLAS/stamp/build.stamp"
[rocBLAS] [0/2] Re-checking globbed directories...
[rocBLAS] [0/616] Generating libraries with TensileCreateLibrary
[rocBLAS] > UserWarning: Found `clang++.exe` but in a non-default ROCm location:
[rocBLAS] F:\TheRock\build\core\clr\dist\lib\llvm\bin\clang++.exe
[rocBLAS] > UserWarning: Did not detect SupportedISA: [(8, 0, 3), (9, 0, 0), (9, 0, 6),
[rocBLAS] (9, 0, 8), (9, 0, 10), (9, 4, 2), (9, 5, 0), (10, 1, 0), (10, 1, 1), (10, 1,
[rocBLAS] 2), (10, 3, 0), (10, 3, 1), (11, 0, 0), (11, 0, 1), (11, 0, 2), (11, 5, 1),
[rocBLAS] (12, 0, 0), (12, 0, 1)]; cannot benchmark assembly kernels.
[rocBLAS] > UserWarning: Failed to detect ISA so forcing (gfx906) on windows
[rocBLAS] Traceback (most recent call last):
[rocBLAS]   File "F:\TheRock\build\math-libs\BLAS\rocBLAS\build\virtualenv\Lib\site-packages\Tensile\bin\TensileCreateLibrary", line 43, in <module>
[rocBLAS]     TensileCreateLibrary()
[rocBLAS]     ~~~~~~~~~~~~~~~~~~~~^^
[rocBLAS]   File "F:\TheRock\build\math-libs\BLAS\rocBLAS\build\virtualenv\Lib\site-packages\Tensile\TensileCreateLibrary.py", line 1379, in TensileCreateLibrary
[rocBLAS]     assignGlobalParameters(args)
[rocBLAS]     ~~~~~~~~~~~~~~~~~~~~~~^^^^^^
[rocBLAS]   File "F:\TheRock\build\math-libs\BLAS\rocBLAS\build\virtualenv\Lib\site-packages\Tensile\Common.py", line 2482, in assignGlobalParameters
[rocBLAS]     *[int(c.split("-")[0]) for c in globalParameters["HipClangVersion"].split(".")[:3]]
[rocBLAS]       ~~~^^^^^^^^^^^^^^^^^
[rocBLAS] ValueError: invalid literal for int() with base 10: 'hipconfig'
[rocBLAS] FAILED: Tensile/library F:/TheRock/build/math-libs/BLAS/rocBLAS/build/Tensile/library
[rocBLAS] C:\Windows\system32\cmd.exe /C "cd /D F:\TheRock\build\math-libs\BLAS\rocBLAS\build\library\src && "C:\Program Files\CMake\bin\cmake.exe" -E env "PATH=F:/TheRock/build/core/clr/dist/bin;F:/TheRock/build/core/clr/dist/lib/llvm/bin;F:\TheRock\.venv\Scripts;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\HostX64\x64;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Current\bin\Roslyn;C:\Program Files (x86)\Windows Kits\10\bin\10.0.26100.0\\x64;C:\Program Files (x86)\Windows Kits\10\bin\\x64;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\\MSBuild\Current\Bin\amd64;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\Tools\;C:\Python313\Scripts\;C:\Python313\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\VSCodium - Insiders\bin;C:\Program Files (x86)\Incredibuild;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\ProgramData\chocolatey\bin;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Program Files\CMake\bin;C:\Strawberry\c\bin;C:\Strawberry\perl\site\bin;C:\Strawberry\perl\bin;C:\Program Files\Microsoft Visual Studio\2022\Preview;C:\Users\Barry\AppData\Local\Microsoft\WindowsApps;C:\Users\Barry\AppData\Local\Microsoft\WinGet\Packages\GlennDelahoy.SnappyDriverInstallerOrigin_Microsoft.Winget.Source_8wekyb3d8bbwe;;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja" -- F:/TheRock/build/math-libs/BLAS/rocBLAS/build/virtualenv/Scripts/python.exe F:/TheRock/build/math-libs/BLAS/rocBLAS/build/virtualenv/Lib/site-packages/Tensile/bin/TensileCreateLibrary --merge-files --separate-architectures --lazy-library-loading --no-short-file-names --verbose=0 --code-object-version=default --cxx-compiler=F:/TheRock/build/core/clr/dist/lib/llvm/bin/clang++.exe --library-format=msgpack --architecture=gfx1101 --no-enumerate F:/TheRock/math-libs/BLAS/rocBLAS/library/src/blas3/Tensile/Logic/asm_full F:/TheRock/build/math-libs/BLAS/rocBLAS/build/Tensile HIP"
[rocBLAS] ninja: build stopped: subcommand failed.
[rocBLAS completed in 3 seconds]
ninja: build stopped: subcommand failed.

OS: Windows 11 Pro x86_64 Kernel: WIN32_NT 10.0.26100.4061 (24H2) Uptime: 3 hours, 26 mins Packages: 27 (choco) Shell: PowerShell 7.6.0 Display (G27QC): 2560x1440 @ 120 Hz (as 2048x1152) in 27" [External] * Display (U28D590): 3840x2160 @ 60 Hz (as 2560x1440) in 27" [External] Terminal: Windows Terminal Preview 1.23.11132.0 Locale: en-GB Terminal Font: Cascadia Mono (12pt) CPU: Intel® Core™ i9-9900KF (16) @ 3.60 GHz GPU: AMD Radeon RX 7800 XT (15.78 GiB) [Discrete] Memory: 16.85 GiB / 31.92 GiB (53%) Swap: 219.36 MiB / 5.00 GiB (4%) Disk (C:): 250.46 GiB / 953.12 GiB (26%) - NTFS Disk (D:): 5.43 GiB / 27.22 GiB (20%) - exFAT Disk (E:): 127.48 GiB / 238.44 GiB (53%) - exFAT Disk (F:): 2.99 GiB / 216.94 GiB (1%) - ReFS Disk (M:): 348.20 MiB / 21.48 GiB (2%) - NTFS