ROCm: Regression in rocm 5.3 and newer for gfx1010

Since when pytorch 2 was officially released, i wasn’t able to run it on my 5700XT, while i was previously able to use it just fine on pytorch 1.13.1 by setting “export HSA_OVERRIDE_GFX_VERSION=10.3.0” There are many reporting the same issue on the 5000 series, like for example https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/6420

–precison-full and --no-half are also needed because the card seems like can’t use fp16 on linux/rocm, as already reported here https://github.com/RadeonOpenCompute/ROCm/issues/1857

i also read about the PCI atomics requirement, following this issue https://github.com/pytorch/pytorch/issues/103973 …But that doesn’t seems to be my case. the command “grep flags /sys/class/kfd/kfd/topology/nodes/*/io_links/0/properties” returns:

/sys/class/kfd/kfd/topology/nodes/0/io_links/0/properties:flags 3
/sys/class/kfd/kfd/topology/nodes/1/io_links/0/properties:flags 1

Also, i tried to compile pytorch using the new “-mprintf-kind=buffered” flag, but it didn’t change anything.

Finally, i recently found out that pytorch 2 works just fine on gfx1010 if that’s compiled by rocm 5.2, as suggested here https://github.com/pytorch/pytorch/issues/106728

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 3
  • Comments: 66 (1 by maintainers)

Most upvoted comments

ok, we will tackle this issue next @kmsedu @DGdev91

Good news, ROCm/Tensile#1897 will be included in the ROCm 6.2 release judging from the release-staging/rocm-rel-6.2 branch at ROCm/Tensile. It seems like no additional ROCm 6.1 releases are planned, so this means gfx101* GPUs will likely be fixed in the next (minor) ROCm release.

OK after a lot of trial and error I’ve managed to get a consistent set of steps.

Steps:

  • Build and install rocBLAS to get the Tensile library to build against gfx1010 (can skip if your distribution already builds it, the official SLES package doesnt) use cmake -B build -S . -DCMAKE_INSTALL_PREFIX=/opt/rocm-6.1.0 -DAMDGPU_TARGETS="gfx1010"
  • Patch the comgr library with the code from here: https://github.com/GZGavinZhao/ROCm-CompilerSupport/commit/3419d519fca9b03ba82dde037cc4600348a9d71d
  • Build comgr and install it to ROCM_PATH
  • Build and install rocSPARSE for the gfx1010 target
  • Start the build process for pytorch from source against your system rocm. I used ROCM_PATH=/opt/rocm USE_ROCM=1 PYTORCH_ROCM_ARCH=gfx1010 python3 setup.py develop
  • Abort the pytorch build process once it completes configuring aotriton, and then patch aotriton with this: https://github.com/ROCm/aotriton/commit/0873896ab690d5767975f2bb9ab850b1a103b26e Changing the commit hash in pytorch/External/aotriton.cmake didn’t work for me, so I manually edited the file
  • Start the build process again. This can take an insanely long time, thanks to aotriton building triton from source, and compiling a whole bunch of HSA command objects for the MI200 and MI300X. Disabling USE_FLASH_ATTENTION will still cause it to be built so might as well keep it enabled. Stopping and starting the build process will cause all of those HSA command objects / HIP kernels to recompile, so be prepared for pytorch to take over an hour to build
  • Once pytorch has been successfully built and installed, build and install torchvision from source. This won’t take as long, but it will need to be done every time you update the pytorch build
  • With pytorch and torchvision built and installed, you can run ComfyUI or any other pytorch workload, without needing the HSA_OVERRIDE_GFX_VERSION variable. Also gfx1010 will now work in both fp16 and fp32 modes, and it should be a little more stable without random SDMA queue timeouts and GPU resets

Caveats:

  • Pytorch’s SDP backend for cross attention does not work and fails with the following error:
torch.OutOfMemoryError: HIP out of memory. Tried to allocate 14.00 MiB. GPU 0 has a total capacity of 5.98 GiB of which 6.00 MiB is free. Of the allocated memory 5.63 GiB is allocated by PyTorch, and 190.05 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_HIP_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

  • The expandable memory segments that pytorch suggests you set, also does not work and fails with this error:
UserWarning: expandable_segments not supported on this platform (Triggered internally at ../c10/hip/HIPAllocatorConfig.h:29.)

  • These steps are not something you can easily add into an automated build system or script, as of the time of writing, it requires you to manually edit files in the pytorch build directory after a build has partially completed. Furthermore, this only targets gfx1010

The key takeaway from this exercise is that I would have probably been better off had I not nuked my Windows install and just used DirectML. Needless to say it was both educational and frustrating. Anyway, hopefully this should help others until AMD releases ROCM 6.2 with the fallback libraries in the official rocBLAS packages and the docker images used to build pytorch, and pytorch themselves build pytorch wheels against those

Does anyone have an ETA for ROCm 6.2?

Last time i tried i had a memory access error […] when trying to load a model in lama.cpp with both hipblas and clblast offloading, while the second worked fine on Windows. I had the same problem in both ArchLinux and Ubuntu.

To be clear, on Ubuntu were you using libhipblas-dev (which installs to /usr/lib/x86_64-linux-gnu) or were you using hipblas-dev (which installs to /opt/rocm/lib)? If you were using libhipblas-dev, I’m very interested in learning more. Could you provide some instructions on how to reproduce the problem?

That’s also true for every consumer-grade amd gpu other than the 7900xt and 7900xtx, wich many users are still using thanks to the same workaround.

Using HSA_OVERRIDE_GFX_VERSION=10.3.0 on RDNA 2 GPUs is fundamentally different from using it on RDNA 1 GPUs. All RDNA 2 GPUs use the exact same instructions, but there’s a bunch of differences between the instructions used on RDNA 1 and RDNA 2 GPUs. The only way to undo this ‘regression’ with HSA_OVERRIDE_GFX_VERSION would be to change LLVM so that the compiler only uses instructions available on RDNA 1, even when asked to compile for RDNA 2. That’s not going to happen.

A better path to getting gfx1010 enabled in PyTorch would be to build the ROCm math and AI libraries for gfx1010 (or gfx10.1-generic). That is probably not going to happen in AMD’s official packages, but there are other groups building and distributing ROCm packages. I can’t speak for other distributions, but I expect to have it enabled later this year on Debian. With that said, my work with Debian is strictly volunteer work (on top of my full-time job), so don’t expect it to happen quickly.

What’s your motivation to use newer ROCm? Expect better performance?

Well, for example to be able to use the official pytorch builds instead of using old nighties or compliling from source.

I was able to get the latest pytorch main code to build successfully. Also, no that aotriton is pre-built, the compilation time is much faster.

Who pinged me

Ok in all seriousness, this issue should be resolved if ROCm/Tensile#1897 is cherry-picked into a release. I’ll open an issue there (ROCm/Tensile#1916). @Zakhrov if you really want to test gfx1010 support, you can try on Solus (the distro that I’m a maintainer for) with Docker. Note that the docker image is experimental/community-maintained so this shouldn’t be used for anything serious, just for testing purposes:

# I personally use podman because I don't need to deal with sudo permission issues,
# but if you're more comfortable with Docker, replace `podman` with `sudo docker`
podman run -it --device=/dev/kfd --device=/dev/dri --group-add=video --group-add=render --group-add=nobody --ipc=host --cap-add=SYS_PTRACE --security-opt seccomp=unconfined silkeh/solus:devel bash

Inside the container, run the following to install PyTorch with ROCm support:

sudo eopkg ur && sudo eopkg up --ignore-comar -y # similar to sudo apt-get update && sudo apt-get upgrade
sudo eopkg it --ignore-comar pytorch python-torchaudio python-torchvision python-torchtext rocm-info -y

Now, you can proceed to run ComfyUI as normal. We didn’t test against ComfyUI but we did test against Fooocus, so the process shoudl be similar. I assume you would need to create a venv to run ComfyUI. The only change you would have to make is that when creating the venv (e.g. python3 -m venv <rest-of-your-arguments>, you should add the flag --system-site-packages. With Fooocus, it looks something like this:

# Before everything, check that ROCm is not completely broken
rocminfo
# Assume you're already inside the Fooocus directory 
python3 -m venv venv --system-site-packages
source venv/bin/activate
# You may get warnings about dependencies failed to be uninstalled/updated, but in practice we haven't found this to be problematic
pip3 install -r requirements.txt 
# Now run Fooocus!

If you’re familiar with Nix, after NixOS/nixpkgs#298388 is merged, you should also be able to use gfx101* GPUs with Nix. With PyTorch in particular, this may be a bit difficult due to how Nix handles Python packages, but stuff that are just binary executables like llama-cpp should work seamlessly with Nix.

Regarding composable_kernel, gfx1010 is not supported. At Solus, we have a patch that enables some sort of support for it, so you may get a little performance boost, though nobody on the Solus team has a GPU to verify this claim 😅

Building the pytorch wheels via docker/podman worked: https://rocm.docs.amd.com/projects/install-on-linux/en/docs-6.2.0/install/3rd-party/pytorch-install.html#using-the-pytorch-upstream-docker-file

  1. git clone https://github.com/pytorch/pytorch.git && git submodule update --init --recursive
  2. apply the patches https://github.com/pytorch/pytorch/pull/132555/commits/0a53e6396902c075f547b8187e6457f82d6dc682
  3. cd .ci/docker
  4. export PYTORCH_ROCM_ARCH=gfx1010
  5. ./build.sh pytorch-linux-ubuntu22.04-rocm6.2-py3.10 -t rocm/pytorch:build_from_dockerfile (had to change a lot of package names in the install shell scripts, the ones prefixed with 6.2.0 worked)
  6. docker/podman run -it --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --user root --device=/dev/kfd --device=/dev/dri --group-add video --ipc=host --shm-size 8G -v ~/pytorch:/pytorch rocm/pytorch:build_from_dockerfile
  7. export PYTORCH_ROCM_ARCH=gfx1010
  8. .ci/pytorch/build.sh

Some tests fail when running. I’ve installed ComfyUI in the container and run a basic example. All the basic samplers except SDE ones seem to work fine. You also don’t need to force fp32 anymore (saves some vram)

The official pytorch wheels don’t work. I suppose they need to be built against rocm 6.2. I am rebuilding pytorch locally against my new rocm 6.2 installation

I have two 5500 XTs (the 8GB model to be specific, not the 4GB one) that I regularly run machine learning applications on both cards. This bug meant I had to compile affected ROCm versions and the latest PyTorch from source to target the gfx1012 architecture to restore functionality.

I wasted many long hours patching pieces of code and tweaking build flags to make the process successful from start to finish. I was relieved to find the PyTorch test suites were working without the need to set the HSA_OVERRIDE_GFX_VERSION environment variable.

I look forward to an official upstream fix sooner or later. Hopefully, this will save users time by not having to rebuild everything from scratch when a future problem occurs. Better a late fix than no fix.

Pytorch wasn’t building because of https://github.com/ROCm/aotriton/issues/18 so I hacked it out and made pytorch compile without aotriton. This actually works and I’m able to run ComfyUI!

Notes:

  • rocBLAS must be built with gfx1010 Tensile libraries and installed into ROCM_PATH, the SLES package does not include them
  • Because of my dirty hacks, pytorch does not have flash or memory efficient attention, so --use-pytorch-cross-attention can lead to HIP out of memory errors, subquadratic attention seems to work better.
  • Forcing fp16 works and gives proper images
  • Forcing fp32 also works
  • You don’t need the HSA_OVERRIDE_GFX_VERSION anymore, it works with python main.py
  • Performance is slower than pytorch built against ROCM 5.2 (although this could be because of thermal throttling) but seems to be much more stable (no random GPU resets despite both the CPU and GPU running near the thermal throttle limit - so far)

I’m trying again with aotriton patched with the fix @xinyazhang suggested to see if that helps with easier/more straightforward building and possibly with performance.

Well, that was a colossal waste of time, looks like rocBLAS doesn’t ship with https://github.com/ROCm/Tensile/pull/1862 at least not for the SLES/OpenSUSE packages

rocBLAS error: Cannot read /opt/rocm/lib/rocblas/library/TensileLibrary.dat: No such file or directory for GPU arch : gfx1010
 List of available TensileLibrary Files : 
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx900.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx906.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx908.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx90a.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx940.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx941.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx942.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1102.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1101.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1030.dat"
"/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1100.dat"
Aborted (core dumped)

Going to try to build rocBLAS and Tensile now

Now that ROCM 6.1 is out, I tried it with the latest pytorch nightly (which still is built with Rocm 6.0) and this is the error I get when trying to run ComfyUI:

:3:hip_platform.cpp         :211 : 14629861941 us: [pid:31113 tid:0x7fa5a37fe700] __hipPopCallConfiguration: Returned hipSuccess : 
:3:hip_module.cpp           :668 : 14629861947 us: [pid:31113 tid:0x7fa5a37fe700]  hipLaunchKernel ( 0x7fa9706d5550, {57,1,1}, {256,1,1}, 0x7fa5a37fc9a0, 0, stream:<null> ) 
:3:hip_module.cpp           :669 : 14629861952 us: [pid:31113 tid:0x7fa5a37fe700] hipLaunchKernel: Returned hipErrorInvalidDeviceFunction : 
:3:hip_error.cpp            :35  : 14629861955 us: [pid:31113 tid:0x7fa5a37fe700]  hipGetLastError (  ) 
:3:hip_error.cpp            :35  : 14629861957 us: [pid:31113 tid:0x7fa5a37fe700]  hipGetLastError (  ) 
:3:hip_device_runtime.cpp   :652 : 14629866892 us: [pid:31113 tid:0x7fa5a37fe700]  hipSetDevice ( 0 ) 
:3:hip_device_runtime.cpp   :656 : 14629866904 us: [pid:31113 tid:0x7fa5a37fe700] hipSetDevice: Returned hipSuccess : 
:3:hip_device_runtime.cpp   :652 : 14629866934 us: [pid:31113 tid:0x7fa5a37fe700]  hipSetDevice ( 0 ) 
:3:hip_device_runtime.cpp   :656 : 14629866937 us: [pid:31113 tid:0x7fa5a37fe700] hipSetDevice: Returned hipSuccess : 
!!! Exception during processing !!!
Traceback (most recent call last):
  File "/home/aaron/Projects/personal/ComfyUI/execution.py", line 151, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aaron/Projects/personal/ComfyUI/execution.py", line 81, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aaron/Projects/personal/ComfyUI/execution.py", line 74, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aaron/Projects/personal/ComfyUI/nodes.py", line 1378, in sample
    return common_ksampler(model, noise_seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise, disable_noise=disable_noise, start_step=start_at_step, last_step=end_at_step, force_full_denoise=force_full_denoise)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aaron/Projects/personal/ComfyUI/nodes.py", line 1314, in common_ksampler
    samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aaron/Projects/personal/ComfyUI/comfy/sample.py", line 37, in sample
    samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aaron/Projects/personal/ComfyUI/comfy/samplers.py", line 755, in sample
    return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aaron/Projects/personal/ComfyUI/comfy/samplers.py", line 657, in sample
    return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aaron/Projects/personal/ComfyUI/comfy/samplers.py", line 644, in sample
    output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aaron/Projects/personal/ComfyUI/comfy/samplers.py", line 616, in inner_sample
    if latent_image is not None and torch.count_nonzero(latent_image) > 0: #Don't shift the empty latent image.
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: HIP error: invalid device function
Compile with TORCH_USE_HIP_DSA to enable device-side assertions.


Prompt executed in 6.49 seconds

setting HSA_OVERRIDE_GFX_VERSION to 10.3.0 still doesn’t work it just maxes out the GPU clock and graphics pipeline but doesn’t actually do anything.

Last time i tried i had a memory access error […] when trying to load a model in lama.cpp with both hipblas and clblast offloading, while the second worked fine on Windows. I had the same problem in both ArchLinux and Ubuntu.

To be clear, on Ubuntu were you using libhipblas-dev (which installs to /usr/lib/x86_64-linux-gnu) or were you using hipblas-dev (which installs to /opt/rocm/lib)? If you were using libhipblas-dev, I’m very interested in learning more. Could you provide some instructions on how to reproduce the problem?

That’s also true for every consumer-grade amd gpu other than the 7900xt and 7900xtx, wich many users are still using thanks to the same workaround.

Using HSA_OVERRIDE_GFX_VERSION=10.3.0 on RDNA 2 GPUs is fundamentally different from using it on RDNA 1 GPUs. All RDNA 2 GPUs use the exact same instructions, but there’s a bunch of differences between the instructions used on RDNA 1 and RDNA 2 GPUs. The only way to undo this ‘regression’ with HSA_OVERRIDE_GFX_VERSION would be to change LLVM so that the compiler only uses instructions available on RDNA 1, even when asked to compile for RDNA 2. That’s not going to happen.

A better path to getting gfx1010 enabled in PyTorch would be to build the ROCm math and AI libraries for gfx1010 (or gfx10.1-generic). That is probably not going to happen in AMD’s official packages, but there are other groups building and distributing ROCm packages. I can’t speak for other distributions, but I expect to have it enabled later this year on Debian. With that said, my work with Debian is strictly volunteer work (on top of my full-time job), so don’t expect it to happen quickly.

Ok, now it’s more clear. I can confirm i used hipblas-dev

I was also thinking that the hsa override flag was needed for rocblas too, because i couldn’t use it on native 1010 since the libs for 1010 were missing in the official packages.

I also just found this PR wich has been merged just 5 days ago wich makes life a bit more simple for compiling the tensile libs for 1010 https://github.com/ROCm/Tensile/pull/1862

Hey there. No, I’m not running on the latest version of pytorch. And yes, you are correct, I’ve run this on/witth automatic1111’s webui with (your 🙏 ❤️) workaround in place. The env HSA_OVERRIDE_GFX_VERSION=9.4.0 made the solution to automatically downgrade the version of pytorch to the version built with rocm 5.2.

Ok, then you have just the same problem. We know that anything compiled using Rocm 5.2 or older works just fine on that card. If you try to force it to a newer version in webui_user.sh probably it’s not going to work. Also, usually HSA_OVERRIDE_GFX_VERSION=10.3.0 is used for the override. Automatic1111’s webui should force it automatically for older gpus, so maybe you were actually using that.

Since gfx1010 is not in the support gfx target list ( https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html#supported-gpus), the latest versions may not work for your gpu.

I know. But it’s still wierd that a gpu wich worked perfectly fine with pytorch compiled for an older romc version + HSA_OVERRIDE_GFX_VERSION=10.3.0 (even if not officially supported) suddently stops working on everything compiled using something newer. I also tried to pick up an old docker image with rocm 5.2 but it doesn’t seems able to compile it.

This is also true for other softwares wich rely on rocm, like llama-cpp with hipblas support

Aldo, are you sure you were actually running on the last pytorch version? In automatic1111’s webui there’s a workaround for making it use by default an older pytorch version compiled on ROCm 5.2 on older navi cards like the RX5700xt (i know that well because… I wrote that workaround).

Hey there. No, I’m not running on the latest version of pytorch. And yes, you are correct, I’ve run this on/witth automatic1111’s webui with (your 🙏 ❤️) workaround in place. The env HSA_OVERRIDE_GFX_VERSION=9.4.0 made the solution to automatically downgrade the version of pytorch to the version built with rocm 5.2.

Memory access fault by GPU node-1 (Agent handle: 0x968d080) on address 0x7fa860641000. Reason: Page not present or supervisor privilege. happens with both HSA_OVERRIDE_GFX_VERSION=10.3.0 and without.

I made this work with HSA_OVERRIDE_GFX_VERSION=9.4.0 (and I had to find this out purely by trial & error…). It is not fully stable, but it worked with SD within at least two separate system boots…