ROCm: [Issue]: ROCm 6.x doesn't work with space in the path

Problem Description

I’m trying to use ROCm with ComfyUI and RX 6700 XT (gfx1031) on Debian 13. It works fine when I use ROCm 5.7 (pytorch 2.3.1+rocm5.7 with HSA_OVERRIDE_GFX_VERSION=10.3.0), but I can’t any newer version to work. I get this error when running ROCm 6.2 (pytorch 2.5.1+rocm6.2):

RuntimeError: HIP error: invalid device function
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

Is there any way to make it work or am I stuck with ROCm 5.7? I’ve also tried running Ollama and the error is a bit different there:

ROCm error: no kernel image is available for execution on the device

Operating System

Debian 13

CPU

GPU

RX 6700 XT

ROCm Version

ROCm 6.2

ROCm Component

No response

Steps to Reproduce

No response

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 24

Most upvoted comments

Hi @pkrasicki @kaiusdepaula, please try out ROCm 7 Alpha, the space issue is fixed in the Alpha build and will be available in ROCm 7.0. Tested with pytorch nightly build on 7900XTX:

(venv with space) ~$  pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm6.4/
 ...
(venv with space) ~$ python3 -c "import torch;print(torch.cuda.is_available())"
True
(venv with space) ~$ python3 -c "import torch; a=torch.randn(3).to('cuda'); print(a)"
tensor([-1.1108, -1.3592,  0.4885], device='cuda:0')

@YyLlBbYy based on the error message, have you tried execstack -c venv/lib/python3.10/site-packages/torch/lib/libamdhip64.so to clear the flag?

returns execstack: venv/lib/python3.10/site-packages/torch/lib/libamdhip64.so: section file offsets not monotonically increasing

using patchelf --clear-execstack instead clears the flag, and I also needed to clear the flag for libhiprtc.so in order to import torch.

oh happy day.

Good to hear that it works now for both of you! I’ll try to trace down the space issue.

As @zichguan-amd mentioned https://github.com/ROCm/ROCm/issues/3680 that was very similar to the situation

@kaiusdepaula I think you referenced the wrong issue?

Unfortunately, I don’t have control over that. ROCm 7 is the next major release and should be coming up later this year. If you want it in ROCm 6.x you can always build from source.

Commit 84a867fb735f69299c45c0b30a085b8e8400cd15 should address the space issue. If you want to test it out now, you can build clr from source on the amd-staging branch or cherry pick this commit and rebuild clr. Will update again when the fix lands in a future release.

I’ve been unable to use automatic1111 WebUI and Forge ever since February 4 and I don’t have a space in my path. a1111 and Forge insist on installing rocm 5.7 Between the time that everything was functional and I discovered everything broke, I updated EndeavourOS and installed calligra and Huiontablet packages, and edited ownership of the files in the Huiontablet installation directory. system packages (which are up to date at what the package managers push to me; rocm 6.2.4 and whatever pytorch) are functional and respond properly to rudimentary tensor calculation troubleshooting tests, once I’ve exported HSA_OVERRIDE_GFX_VERSION=10.3.0

I’ll have to bounce between different operating systems to provide more information, which I will edit into this post.

with automatic1111 activating the venv and calling HSA_OVERRIDE_GFX_VERSION=10.3.0 python3.10 -c "import torch; a=torch.randn(3).to('cuda');print(a)" throws me ImportError: libamdhip64.so: cannot enable executable stack as shared object requires: Invalid argument and ldd venv/lib/python3.10/site-packages/torch/lib/libamdhip64.so returns this

I’ve downgraded to python 3.10.13 (reasoning that 3.10.16 has an install date and time right before my problems began) with no improvement to my predicament, but I’ve installed ComfyUI now, and everything works as long as I manually activate the venv prior to running ComfyUI. That either points to me installing the dependencies wrong or there’s something that’s wrong elsewhere but halfway serviceable. With this in mind there’s probably nothing wrong with rocm 6.2 itself for my own situation, and something went wrong somewhere else that’s preventing automatic1111/Forge /torch-2.3.1+rocm5.7 from working. and at that point it’s probably more sane to nudge those projects into using rocm 6.x than troubleshoot a rocm stack that’s 18 months past EOS.

I’ll keep the issue open for visibility.

Hi @pkrasicki @kaiusdepaula, a fix for the space issue is under development. I’m closing this issue now since a workaround (remove space in path) exists. I’ll update once the fix lands in a future ROCm release.

@pkrasicki it could definitely be a space in the path! I’ve renamed it with a _ and it worked. This was definitely the issue.

@zichguan-amd the issue I referred to was wrong, I edited it to be right now. Thank you all very much!

Hi @pkrasicki. Internal ticket has been created to investigate this issue. Thanks!