TheRock: [Issue]: Error when building ROCm/PyTorch on Windows (7800 XT)
Problem Description
I followed all the instructions in here, using chocolatey for all the tools.
I get this error that cuts the build about 30/40 mins in. Python version is 3.12.3.
[200/235] Building sub-project rocBLAS (in background)
FAILED: math-libs/BLAS/rocBLAS/stamp/build.stamp I:/TheRock/build/math-libs/BLAS/rocBLAS/stamp/build.stamp
C:\WINDOWS\system32\cmd.exe /C "cd /D I:\TheRock\build\math-libs\BLAS\rocBLAS\build && I:\TheRock\.venv\Scripts\python.exe I:/TheRock/build_tools/teatime.py --log-timestamps --label rocBLAS --interactive I:/TheRock/build/logs/rocBLAS_build.log -- "C:/Program Files/CMake/bin/cmake.exe" -E env -- "C:/Program Files/CMake/bin/cmake.exe" --build I:/TheRock/build/math-libs/BLAS/rocBLAS/build && "C:\Program Files\CMake\bin\cmake.exe" -E touch I:/TheRock/build/math-libs/BLAS/rocBLAS/stamp/build.stamp"
[rocBLAS] [0/2] Re-checking globbed directories...
[rocBLAS] [0/628] Generating libraries with TensileCreateLibrary
[rocBLAS] > UserWarning: Found `clang++.exe` but in a non-default ROCm location:
[rocBLAS] I:\TheRock\build\core\clr\dist\lib\llvm\bin\clang++.exe
[rocBLAS] > UserWarning: Did not detect SupportedISA: [(8, 0, 3), (9, 0, 0), (9, 0, 6),
[rocBLAS] (9, 0, 8), (9, 0, 10), (9, 4, 2), (9, 5, 0), (10, 1, 0), (10, 1, 1), (10, 1,
[rocBLAS] 2), (10, 3, 0), (10, 3, 1), (11, 0, 0), (11, 0, 1), (11, 0, 2), (11, 5, 1),
[rocBLAS] (12, 0, 0), (12, 0, 1)]; cannot benchmark assembly kernels.
[rocBLAS] > UserWarning: Failed to detect ISA so forcing (gfx906) on windows
[rocBLAS] Traceback (most recent call last):
[rocBLAS] File "I:\TheRock\build\math-libs\BLAS\rocBLAS\build\virtualenv\Lib\site-packages\Tensile\bin\TensileCreateLibrary", line 43, in <module>
[rocBLAS] TensileCreateLibrary()
[rocBLAS] File "I:\TheRock\build\math-libs\BLAS\rocBLAS\build\virtualenv\Lib\site-packages\Tensile\TensileCreateLibrary.py", line 1379, in TensileCreateLibrary
[rocBLAS] assignGlobalParameters(args)
[rocBLAS] File "I:\TheRock\build\math-libs\BLAS\rocBLAS\build\virtualenv\Lib\site-packages\Tensile\Common.py", line 2482, in assignGlobalParameters
[rocBLAS] *[int(c.split("-")[0]) for c in globalParameters["HipClangVersion"].split(".")[:3]]
[rocBLAS] ^^^^^^^^^^^^^^^^^^^^
[rocBLAS] ValueError: invalid literal for int() with base 10: 'hipconfig'
[rocBLAS] [11/628] Generating prototypes from I:/TheRock/math-libs/BLAS/rocBLAS/library/src.
[rocBLAS] FAILED: Tensile/library I:/TheRock/build/math-libs/BLAS/rocBLAS/build/Tensile/library
[rocBLAS] C:\WINDOWS\system32\cmd.exe /C "cd /D I:\TheRock\build\math-libs\BLAS\rocBLAS\build\library\src && "C:\Program Files\CMake\bin\cmake.exe" -E env "PATH=I:/TheRock/build/core/clr/dist/bin;I:/TheRock/build/core/clr/dist/lib/llvm/bin;I:\TheRock\.venv\Scripts;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.44.35207\bin\HostX64\x64;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Current\bin\Roslyn;C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\\x64;C:\Program Files (x86)\Windows Kits\10\bin\\x64;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\\MSBuild\Current\Bin\amd64;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\Tools\;C:\Python312\Scripts\;C:\Python312\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\dotnet\;C:\Program Files\Apollo;C:\Program Files\Apollo\tools;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\chocolatey\bin;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Program Files\CMake\bin;C:\Strawberry\c\bin;C:\Strawberry\perl\site\bin;C:\Strawberry\perl\bin;C:\Users\Max\AppData\Local\Microsoft\WindowsApps;C:\Users\Max\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\Max\AppData\Local\GitHubDesktop\bin;C:\Users\Max\AppData\Local\Google\Cloud SDK\google-cloud-sdk\bin;C:\Program Files\AMD\ROCm\6.2\bin;C:\ZLUDA\zluda;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja" -- I:/TheRock/build/math-libs/BLAS/rocBLAS/build/virtualenv/Scripts/python.exe I:/TheRock/build/math-libs/BLAS/rocBLAS/build/virtualenv/Lib/site-packages/Tensile/bin/TensileCreateLibrary --merge-files --separate-architectures --lazy-library-loading --no-short-file-names --verbose=0 --code-object-version=default --cxx-compiler=I:/TheRock/build/core/clr/dist/lib/llvm/bin/clang++.exe --library-format=msgpack --architecture=gfx1100_gfx1101_gfx1102 --no-enumerate I:/TheRock/math-libs/BLAS/rocBLAS/library/src/blas3/Tensile/Logic/asm_full I:/TheRock/build/math-libs/BLAS/rocBLAS/build/Tensile HIP"
Operating System
Windows 11
CPU
7700x
GPU
7800 xt
ROCm Version
latest
ROCm Component
rocBLAS
Steps to Reproduce
No response
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
No response
Additional Information
PS C:\Users\Max> (Get-WmiObject Win32_OperatingSystem).Version
10.0.26100
PS C:\Users\Max> (Get-WmiObject win32_Processor).Name
AMD Ryzen 7 7700X 8-Core Processor
PS C:\Users\Max> (Get-WmiObject win32_VideoController).Name
AMD Radeon RX 7800 XT
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 52 (40 by maintainers)
Now that we’re saying that out loud, I can provide some additional details.
There will be nightlies of it on here long before that, and there is still the question of what the intercept with what you can get from pytorch.org for which architectures is. How we message what is “supported” is still a pretty evolving topic. The model we’re moving to is just continuous releases of latest “core” rocm and pytorch on Linux and Windows with an expanding set of gfx architectures that we keep building out.
There will also be regular “cuts” of that which are given a new version number and distributed via all channels with a larger set of rocm components (and likely a smaller set of supported gfx arches, since we are capacity limited on how much we can support in an “everything” build). These are what you currently see as ROCm releases. These are what we will continue to talk about in terms of official support, but actual support will be available both more broadly and sooner through our community release channels (which are the precursors to the official releases).
We still don’t quite know how to talk about release plans like this, which is how you get coarse dates like “Q3” and fuzziness on what “supported” means. I expect the way we communicate this will evolve over time. In the meantime, the approach is roughly what I describe above.
That worked!
I think we’ve fixed those issues now:
main) https://github.com/ROCm/TheRock/issues/670Might have incidentally fixed that too, but I’m not sure. I haven’t seen that in my own testing.
I’m adding a few notes to the documentation for now: https://github.com/ROCm/TheRock/pull/660. I’ll have development time later in the week to help land fixes.
BTW, thanks for filing this issue and taking the time to debug!
@ScottTodd Thank you!
I was also experiencing the same
hipconfigissue reported in this thread, but it’s resolved now after updating my repository to the latest version.the build worked!
@ScottTodd I have identified the conditions for reproduction. I will write the details in #670.
@7shi Thanks for the report. I’ve re-opened https://github.com/ROCm/TheRock/issues/670 and we can follow up on
HIP_PATHenv var orC:\Program Files\AMD\ROCmfile path issues there.@7shi san, @jammm and I have produced a self-contained Pytorch wheel that should support the 7600 XT (gfx1102) here: https://github.com/scottt/rocm-TheRock/releases/download/v6.5.0rc-pytorch-gfx110x/torch-2.7.0a0+gitafb201d-cp312-cp312-win_amd64.whl
Hope that it’s useful to you.
Btw AMD has announced that ROCm would officially support running PyTorch on Windows at 2025Q3.
@MaximilienLC, once you have
aotriton_v2.dllbuilt on Windows.aotriton.imagesdirectory fromaotriton/libinside the tarballaotriton.imagesnext toaotriton_v2.dllin the install directory (You want aotriton kernels for gfx1101 while my other instructions are for gfx1151)here’s the wheel: https://drive.proton.me/urls/4QRV647ZXC#RPyLKsYAOHIv works in a new venv, onto
aotritonandtorchfriends@MaximilienLC , I suggest building
torch-{vision,audio}withoutffmpegfirstFor building torchvision, make sure that your ROCM_HOME is set to the rocm directory e.g.,
<TheRock_dir>\build\dist\rocmAnd setCC=clang-clandCXX=clang-clThen it should just be a matter of
python setup.py bdist_wheel -vand check if it’s building device libs in there.@MaximilienLC, congrats on the successful native Pytorch build! I was just writing to tell you about
aotritonon Windows.Note:
torch.nn.functional.scaled_dot_product_attention, would be slow in your current buildhuggingface/transformersneeds Pytorch to be rebuilt withlibuvinstalled andUSE_DISTRIBUTED=ONenabled to worktorchvision builds and runs fine. Don’t know about torchaudio
@lamikr has been setting torchaudio and torchvision up on Linux and I think he was planning on looking at Windows too. Maybe @scottt or @jammm have tried also. I don’t know what the delta between Linux and Windows looks like for those projects myself yet.
@scottt going to try to follow your
aotritrontutorial now@ScottTodd I’ll try building the fat wheel and share it. how should I go about building
torchaudio&torchvisionthough?@ScottTodd Disabling WSL worked!
On to the next error:
ptbuild.py will still use patches from https://github.com/ROCm/TheRock/tree/main/external-builds/pytorch/patches/v2.7.0/pytorch/hipified. Need https://github.com/ROCm/TheRock/blob/main/external-builds/pytorch/patches/pytorch/v2.7.0/pytorch/hipified/0013-ROCm-Windows-Enable-USE_ROCM-and-fix-assorted-issues.patch, which I only placed in the new location 🙈
(The plan is to get all patches landed upstream in PyTorch then built from tip of tree instead of v2.7.0 + patches… we’ll get there eventually)
cool will try
lol i’ve been waiting for months can’t wait a few more days hhhh
@MaximilienLC , if you’re still using
ptbuild.pysee:python ptbuild.py checkout --pytorch-ref v2.7.0in https://gist.github.com/ScottTodd/cb4f9f5a697cce91084977ed3674a4bd(or wait a few days and let us sort out these last few problems of building Pytorch on Windows)
Argh. I need to drop offline now, but that should be using 2.7.0, not 2.6.0. The ptbuild script defaults to checking 2.6.0 and is missing some Windows patches. I’ve been testing with checkout from
pytorch_torch_repo.py, which defaults to 2.7.0 and has the latest patches merged in. Need to delete the old patches and set up better defaults.yep I hardcoded the version that I found
Cool I’ll uninstall it and report back.
I verbatim had the exact same error earlier today. I too have the 7800 XT. Sorry I cant give anymore info as it happened this morning and I never thought to save the compilation outputs.