ROCm: Regression in rocm 5.3 and newer for gfx1010
Since when pytorch 2 was officially released, i wasn’t able to run it on my 5700XT, while i was previously able to use it just fine on pytorch 1.13.1 by setting “export HSA_OVERRIDE_GFX_VERSION=10.3.0” There are many reporting the same issue on the 5000 series, like for example https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/6420
–precison-full and --no-half are also needed because the card seems like can’t use fp16 on linux/rocm, as already reported here https://github.com/RadeonOpenCompute/ROCm/issues/1857
i also read about the PCI atomics requirement, following this issue https://github.com/pytorch/pytorch/issues/103973 …But that doesn’t seems to be my case. the command “grep flags /sys/class/kfd/kfd/topology/nodes/*/io_links/0/properties” returns:
/sys/class/kfd/kfd/topology/nodes/0/io_links/0/properties:flags 3
/sys/class/kfd/kfd/topology/nodes/1/io_links/0/properties:flags 1
Also, i tried to compile pytorch using the new “-mprintf-kind=buffered” flag, but it didn’t change anything.
Finally, i recently found out that pytorch 2 works just fine on gfx1010 if that’s compiled by rocm 5.2, as suggested here https://github.com/pytorch/pytorch/issues/106728
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 3
- Comments: 66 (1 by maintainers)
ok, we will tackle this issue next @kmsedu @DGdev91
Good news, ROCm/Tensile#1897 will be included in the ROCm 6.2 release judging from the
release-staging/rocm-rel-6.2branch at ROCm/Tensile. It seems like no additional ROCm 6.1 releases are planned, so this meansgfx101*GPUs will likely be fixed in the next (minor) ROCm release.OK after a lot of trial and error I’ve managed to get a consistent set of steps.
Steps:
cmake -B build -S . -DCMAKE_INSTALL_PREFIX=/opt/rocm-6.1.0 -DAMDGPU_TARGETS="gfx1010"comgrlibrary with the code from here: https://github.com/GZGavinZhao/ROCm-CompilerSupport/commit/3419d519fca9b03ba82dde037cc4600348a9d71dcomgrand install it toROCM_PATHgfx1010targetROCM_PATH=/opt/rocm USE_ROCM=1 PYTORCH_ROCM_ARCH=gfx1010 python3 setup.py developpytorch/External/aotriton.cmakedidn’t work for me, so I manually edited the fileUSE_FLASH_ATTENTIONwill still cause it to be built so might as well keep it enabled. Stopping and starting the build process will cause all of those HSA command objects / HIP kernels to recompile, so be prepared for pytorch to take over an hour to buildHSA_OVERRIDE_GFX_VERSIONvariable. Also gfx1010 will now work in both fp16 and fp32 modes, and it should be a little more stable without random SDMA queue timeouts and GPU resetsCaveats:
The key takeaway from this exercise is that I would have probably been better off had I not nuked my Windows install and just used DirectML. Needless to say it was both educational and frustrating. Anyway, hopefully this should help others until AMD releases ROCM 6.2 with the fallback libraries in the official rocBLAS packages and the docker images used to build pytorch, and pytorch themselves build pytorch wheels against those
Does anyone have an ETA for ROCm 6.2?
To be clear, on Ubuntu were you using
libhipblas-dev(which installs to/usr/lib/x86_64-linux-gnu) or were you usinghipblas-dev(which installs to/opt/rocm/lib)? If you were usinglibhipblas-dev, I’m very interested in learning more. Could you provide some instructions on how to reproduce the problem?Using
HSA_OVERRIDE_GFX_VERSION=10.3.0on RDNA 2 GPUs is fundamentally different from using it on RDNA 1 GPUs. All RDNA 2 GPUs use the exact same instructions, but there’s a bunch of differences between the instructions used on RDNA 1 and RDNA 2 GPUs. The only way to undo this ‘regression’ withHSA_OVERRIDE_GFX_VERSIONwould be to change LLVM so that the compiler only uses instructions available on RDNA 1, even when asked to compile for RDNA 2. That’s not going to happen.A better path to getting gfx1010 enabled in PyTorch would be to build the ROCm math and AI libraries for gfx1010 (or gfx10.1-generic). That is probably not going to happen in AMD’s official packages, but there are other groups building and distributing ROCm packages. I can’t speak for other distributions, but I expect to have it enabled later this year on Debian. With that said, my work with Debian is strictly volunteer work (on top of my full-time job), so don’t expect it to happen quickly.
Well, for example to be able to use the official pytorch builds instead of using old nighties or compliling from source.
I was able to get the latest pytorch main code to build successfully. Also, no that aotriton is pre-built, the compilation time is much faster.
Ok in all seriousness, this issue should be resolved if ROCm/Tensile#1897 is cherry-picked into a release. I’ll open an issue there (ROCm/Tensile#1916). @Zakhrov if you really want to test gfx1010 support, you can try on Solus (the distro that I’m a maintainer for) with Docker. Note that the docker image is experimental/community-maintained so this shouldn’t be used for anything serious, just for testing purposes:
Inside the container, run the following to install PyTorch with ROCm support:
Now, you can proceed to run ComfyUI as normal. We didn’t test against ComfyUI but we did test against Fooocus, so the process shoudl be similar. I assume you would need to create a
venvto run ComfyUI. The only change you would have to make is that when creating thevenv(e.g.python3 -m venv <rest-of-your-arguments>, you should add the flag--system-site-packages. With Fooocus, it looks something like this:If you’re familiar with Nix, after NixOS/nixpkgs#298388 is merged, you should also be able to use
gfx101*GPUs with Nix. With PyTorch in particular, this may be a bit difficult due to how Nix handles Python packages, but stuff that are just binary executables likellama-cppshould work seamlessly with Nix.Regarding
composable_kernel,gfx1010is not supported. At Solus, we have a patch that enables some sort of support for it, so you may get a little performance boost, though nobody on the Solus team has a GPU to verify this claim 😅Building the pytorch wheels via docker/podman worked: https://rocm.docs.amd.com/projects/install-on-linux/en/docs-6.2.0/install/3rd-party/pytorch-install.html#using-the-pytorch-upstream-docker-file
git clone https://github.com/pytorch/pytorch.git && git submodule update --init --recursivecd .ci/dockerexport PYTORCH_ROCM_ARCH=gfx1010./build.sh pytorch-linux-ubuntu22.04-rocm6.2-py3.10 -t rocm/pytorch:build_from_dockerfile(had to change a lot of package names in the install shell scripts, the ones prefixed with 6.2.0 worked)docker/podman run -it --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --user root --device=/dev/kfd --device=/dev/dri --group-add video --ipc=host --shm-size 8G -v ~/pytorch:/pytorch rocm/pytorch:build_from_dockerfileexport PYTORCH_ROCM_ARCH=gfx1010.ci/pytorch/build.shSome tests fail when running. I’ve installed ComfyUI in the container and run a basic example. All the basic samplers except SDE ones seem to work fine. You also don’t need to force fp32 anymore (saves some vram)
The official pytorch wheels don’t work. I suppose they need to be built against rocm 6.2. I am rebuilding pytorch locally against my new rocm 6.2 installation
I have two 5500 XTs (the 8GB model to be specific, not the 4GB one) that I regularly run machine learning applications on both cards. This bug meant I had to compile affected ROCm versions and the latest PyTorch from source to target the
gfx1012architecture to restore functionality.I wasted many long hours patching pieces of code and tweaking build flags to make the process successful from start to finish. I was relieved to find the PyTorch test suites were working without the need to set the
HSA_OVERRIDE_GFX_VERSIONenvironment variable.I look forward to an official upstream fix sooner or later. Hopefully, this will save users time by not having to rebuild everything from scratch when a future problem occurs. Better a late fix than no fix.
Pytorch wasn’t building because of https://github.com/ROCm/aotriton/issues/18 so I hacked it out and made pytorch compile without aotriton. This actually works and I’m able to run ComfyUI!
Notes:
--use-pytorch-cross-attentioncan lead to HIP out of memory errors, subquadratic attention seems to work better.python main.pyI’m trying again with aotriton patched with the fix @xinyazhang suggested to see if that helps with easier/more straightforward building and possibly with performance.
Well, that was a colossal waste of time, looks like rocBLAS doesn’t ship with https://github.com/ROCm/Tensile/pull/1862 at least not for the SLES/OpenSUSE packages
Going to try to build rocBLAS and Tensile now
Now that ROCM 6.1 is out, I tried it with the latest pytorch nightly (which still is built with Rocm 6.0) and this is the error I get when trying to run ComfyUI:
setting HSA_OVERRIDE_GFX_VERSION to 10.3.0 still doesn’t work it just maxes out the GPU clock and graphics pipeline but doesn’t actually do anything.
Ok, now it’s more clear. I can confirm i used hipblas-dev
I was also thinking that the hsa override flag was needed for rocblas too, because i couldn’t use it on native 1010 since the libs for 1010 were missing in the official packages.
I also just found this PR wich has been merged just 5 days ago wich makes life a bit more simple for compiling the tensile libs for 1010 https://github.com/ROCm/Tensile/pull/1862
Ok, then you have just the same problem. We know that anything compiled using Rocm 5.2 or older works just fine on that card. If you try to force it to a newer version in webui_user.sh probably it’s not going to work. Also, usually HSA_OVERRIDE_GFX_VERSION=10.3.0 is used for the override. Automatic1111’s webui should force it automatically for older gpus, so maybe you were actually using that.
I know. But it’s still wierd that a gpu wich worked perfectly fine with pytorch compiled for an older romc version + HSA_OVERRIDE_GFX_VERSION=10.3.0 (even if not officially supported) suddently stops working on everything compiled using something newer. I also tried to pick up an old docker image with rocm 5.2 but it doesn’t seems able to compile it.
This is also true for other softwares wich rely on rocm, like llama-cpp with hipblas support
Hey there. No, I’m not running on the latest version of pytorch. And yes, you are correct, I’ve run this on/witth automatic1111’s webui with (your 🙏 ❤️) workaround in place. The env HSA_OVERRIDE_GFX_VERSION=9.4.0 made the solution to automatically downgrade the version of pytorch to the version built with rocm 5.2.
I made this work with HSA_OVERRIDE_GFX_VERSION=9.4.0 (and I had to find this out purely by trial & error…). It is not fully stable, but it worked with SD within at least two separate system boots…