ROCm: [Issue]: `RuntimeError: HIP error: invalid device function` with MI250X and ROCm 6.2.1

Problem Description

I am trying to use ROCm with Pytorch for model inference and have setup a small Resnet18 script to see if things work. I am running a Singularity container on a single node in a Slurm cluster, where each node is equipped with a MI250X GPU. When running, I get a RuntimeError: HIP error: invalid device function:

Traceback (most recent call last):
  File "<string>", line 22, in <module>
  File "/opt/aispecs-venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/aispecs-venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/aispecs-venv/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 465, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/aispecs-venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/aispecs-venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/aispecs-venv/lib/python3.12/site-packages/torchvision/models/resnet.py", line 285, in forward
    return self._forward_impl(x)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/aispecs-venv/lib/python3.12/site-packages/torchvision/models/resnet.py", line 269, in _forward_impl
    x = self.bn1(x)
        ^^^^^^^^^^^
  File "/opt/aispecs-venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/aispecs-venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/aispecs-venv/lib/python3.12/site-packages/torch/nn/modules/batchnorm.py", line 173, in forward
    self.num_batches_tracked.add_(1)  # type: ignore[has-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: HIP error: invalid device function
Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

srun: Received task exit notification for 1 task of StepId=8934739.0 (status=0x0100).
srun: error: nid005414: task 0: Exited with exit code 1
srun: Terminating StepId=8934739.0

I have tried the proposed solutions in https://github.com/ROCm/ROCm/issues/2536 and export HSA_OVERRIDE_GFX_VERSION=9.0.10, but still get the same error. Is there another solution, or some way to debug this error?

Operating System

Ubuntu 24.04.1 LTS (Noble Numbat)

CPU

AMD EPYC 7742 64-Core Processor

GPU

1x AMD Instinct MI250X

ROCm Version

ROCm 6.2.1

ROCm Component

No response

Steps to Reproduce

This is the script I used for running Pytorch and ROCm:

#!/bin/bash
#SBATCH --job-name=AISPECS
#SBATCH --output=results/output%j
#SBATCH --error=results/error%j
#SBATCH --account=project_XXX
#SBATCH --time=08:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --gpus-per-task=1
#SBATCH --mem=64G
#SBATCH --partition=standard-g

srun --verbose \
  singularity exec \
  --env PYTORCH_ROCM_ARCH="gfx90a" \
  --bind "./wheelhouse:/opt/wheelhouse" \
  aispecs-docker-eval-amd.sif \
  bash -c "/opt/aispecs-venv/bin/python3 -c \"

import torch
import amdsmi

# Check installation
amdsmi.amdsmi_init()
print('amdsmi:          ', amdsmi.amdsmi_get_processor_handles())
print('CUDA built:      ', torch.backends.cuda.is_built())
print('CUDA available:  ', torch.cuda.is_available())
print('cuDNN enabled:   ', torch.backends.cudnn.enabled)
print('cuDNN available: ', torch.backends.cudnn.is_available())
print('cuDNN version:   ', torch.backends.cudnn.version())
print('Torch Compiler Backends: ', torch.compiler.list_backends())
print('Flash Attention available: ', torch.backends.cuda.flash_sdp_enabled())
import torchvision.models as models
import torch._dynamo
torch._dynamo.config.suppress_errors = True

model = models.resnet18().cuda()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
compiled_model = torch.compile(model, backend='inductor')
x = torch.randn(16, 3, 224, 224).cuda()
optimizer.zero_grad()
out = compiled_model(x)
out.sum().backward()
optimizer.step()
\""

I get this stdout-output:

amdsmi:           [c_void_p(12949680)]
CUDA built:       True
CUDA available:   True
cuDNN enabled:    True
cuDNN available:  True
cuDNN version:    3002000
Torch Compiler Backends:  ['cudagraphs', 'inductor', 'onnxrt', 'openxla', 'tvm']
Flash Attention available:  True

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

^[[37mROCk module version 6.3.6 is loaded^[[0m
=====================
HSA System Attributes
=====================
Runtime Version:         1.14
Runtime Ext Version:     1.6
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE
System Endianness:       LITTLE
Mwaitx:                  DISABLED
DMAbuf Support:          YES

==========
HSA Agents
==========
*******
Agent 1
*******
  Name:                    AMD EPYC 7A53 64-Core Processor
  Uuid:                    CPU-XX
  Marketing Name:          AMD EPYC 7A53 64-Core Processor
  Vendor Name:             CPU
  Feature:                 None specified
  Profile:                 FULL_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        0(0x0)
  Queue Min Size:          0(0x0)
  Queue Max Size:          0(0x0)
  Queue Type:              MULTI
  Node:                    0
  Device Type:             CPU
  Cache Info:
    L1:                      32768(0x8000) KB
  Chip ID:                 0(0x0)
  ASIC Revision:           0(0x0)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   2000
  BDFID:                   0
  Internal Node ID:        0
  Compute Unit:            32
  SIMDs per CU:            0
  Shader Engines:          0
  Shader Arrs. per Eng.:   0
  WatchPts on Addr. Ranges:1
  Memory Properties:
  Features:                None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    131295136(0x7d367a0) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 2
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    131295136(0x7d367a0) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 3
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    131295136(0x7d367a0) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
  ISA Info:
*******
Agent 2
*******
  Name:                    AMD EPYC 7A53 64-Core Processor
  Uuid:                    CPU-XX
  Marketing Name:          AMD EPYC 7A53 64-Core Processor
  Vendor Name:             CPU
  Feature:                 None specified
  Profile:                 FULL_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        0(0x0)
  Queue Min Size:          0(0x0)
  Queue Max Size:          0(0x0)
  Queue Type:              MULTI
  Node:                    1
  Device Type:             CPU
  Cache Info:
    L1:                      32768(0x8000) KB
  Chip ID:                 0(0x0)
  ASIC Revision:           0(0x0)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   2000
  BDFID:                   0
  Internal Node ID:        1
  Compute Unit:            32
  SIMDs per CU:            0
  Shader Engines:          0
  Shader Arrs. per Eng.:   0
  WatchPts on Addr. Ranges:1
  Memory Properties:
  Features:                None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    132111288(0x7dfdbb8) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 2
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    132111288(0x7dfdbb8) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 3
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    132111288(0x7dfdbb8) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
  ISA Info:
*******
Agent 3
*******
  Name:                    AMD EPYC 7A53 64-Core Processor
  Uuid:                    CPU-XX
  Marketing Name:          AMD EPYC 7A53 64-Core Processor
  Vendor Name:             CPU
  Feature:                 None specified
  Profile:                 FULL_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        0(0x0)
  Queue Min Size:          0(0x0)
  Queue Max Size:          0(0x0)
  Queue Type:              MULTI
  Node:                    2
  Device Type:             CPU
  Cache Info:
    L1:                      32768(0x8000) KB
  Chip ID:                 0(0x0)
  ASIC Revision:           0(0x0)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   2000
  BDFID:                   0
  Internal Node ID:        2
  Compute Unit:            32
  SIMDs per CU:            0
  Shader Engines:          0
  Shader Arrs. per Eng.:   0
  WatchPts on Addr. Ranges:1
  Memory Properties:
  Features:                None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    132111292(0x7dfdbbc) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 2
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    132111292(0x7dfdbbc) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 3
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    132111292(0x7dfdbbc) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
  ISA Info:
*******
Agent 4
*******
  Name:                    AMD EPYC 7A53 64-Core Processor
  Uuid:                    CPU-XX
  Marketing Name:          AMD EPYC 7A53 64-Core Processor
  Vendor Name:             CPU
  Feature:                 None specified
  Profile:                 FULL_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        0(0x0)
  Queue Min Size:          0(0x0)
  Queue Max Size:          0(0x0)
  Queue Type:              MULTI
  Node:                    3
  Device Type:             CPU
  Cache Info:
    L1:                      32768(0x8000) KB
  Chip ID:                 0(0x0)
  ASIC Revision:           0(0x0)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   2000
  BDFID:                   0
  Internal Node ID:        3
  Compute Unit:            32
  SIMDs per CU:            0
  Shader Engines:          0
  Shader Arrs. per Eng.:   0
  WatchPts on Addr. Ranges:1
  Memory Properties:
  Features:                None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    132056244(0x7df04b4) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 2
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    132056244(0x7df04b4) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 3
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    132056244(0x7df04b4) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
  ISA Info:
*******
Agent 5
*******
  Name:                    gfx90a
  Uuid:                    GPU-a543f05567cbe736
  Marketing Name:          AMD Instinct MI250X
  Vendor Name:             AMD
  Feature:                 KERNEL_DISPATCH
  Profile:                 BASE_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        128(0x80)
  Queue Min Size:          64(0x40)
  Queue Max Size:          131072(0x20000)
  Queue Type:              MULTI
  Node:                    4
  Device Type:             GPU
  Cache Info:
    L1:                      16(0x10) KB
    L2:                      8192(0x2000) KB
  Chip ID:                 29704(0x7408)
  ASIC Revision:           1(0x1)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   1700
  BDFID:                   53504
  Internal Node ID:        4
  Compute Unit:            110
  SIMDs per CU:            4
  Shader Engines:          8
  Shader Arrs. per Eng.:   1
  WatchPts on Addr. Ranges:4
  Coherent Host Access:    TRUE
  Memory Properties:
  Features:                KERNEL_DISPATCH
  Fast F16 Operation:      TRUE
  Wavefront Size:          64(0x40)
  Workgroup Max Size:      1024(0x400)
  Workgroup Max Size per Dimension:
    x                        1024(0x400)
    y                        1024(0x400)
    z                        1024(0x400)
  Max Waves Per CU:        32(0x20)
  Max Work-item Per CU:    2048(0x800)
  Grid Max Size:           4294967295(0xffffffff)
  Grid Max Size per Dimension:
    x                        4294967295(0xffffffff)
    y                        4294967295(0xffffffff)
    z                        4294967295(0xffffffff)
  Max fbarriers/Workgrp:   32
  Packet Processor uCode:: 78
  SDMA engine uCode::      8
  IOMMU Support::          None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    67092480(0x3ffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 2
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    67092480(0x3ffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 3
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    67092480(0x3ffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 4
      Segment:                 GROUP
      Size:                    64(0x40) KB
      Allocatable:             FALSE
      Alloc Granule:           0KB
      Alloc Recommended Granule:0KB
      Alloc Alignment:         0KB
      Accessible by all:       FALSE
  ISA Info:
    ISA 1
      Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:           4294967295(0xffffffff)
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)
        y                        4294967295(0xffffffff)
        z                        4294967295(0xffffffff)
      FBarrier Max Size:       32
*** Done ***

Additional Information

This gist https://gist.github.com/segeljakt/d854cf6cc8f11a8e268d69542167a6f4 contains:

  • The output when running with AMD_SERIALIZE_KERNEL=3
  • The Dockerfile I used to build the wheels for Pytorch:
  • The Dockerfile that I later converted to Singularity to run Pytorch with ROCm.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19

Most upvoted comments

I rebuilt without the -e flag, and it started working! Thank you for the help. Also, I noticed that the HPC cluster was running Singularity with version 4.1.3, but I had 4.2.1, so I downgraded. I am not sure which one of these things made it work, but I might try to reproduce it later. Feel free to close this issue.

@segeljakt, just noticed in the line you are using to build wheel, you have

PYTORCH_ROCM_ARCH=$GPU_VERSION USE_KINETO=0 USE_ROCTRACER=0 pip3 --verbose wheel --use-pep517 --no-deps -w "$AISPECS_WHEEL_DIR" --no-build-isolation -e .

The -e makes pip build in editable mode, which results in it creating symbolic links instead of properly saving the libraries to site packages. Typically this is not used in conjunction with building wheels I think. I would try removing this flag.

When we compile Torch, we have previously specified PYTORCH_ROCM_ARCH=gfx90a. We are considering to re-compile our image. We are considering if our choice of PYTORCH_ROCM_ARCH caused the problem. Should we specify PYTORCH_ROCM_ARCH=gfx90a:sramecc+:xnack- instead, or leave the flag unset?

I think PYTORCH_ROCM_ARCH=gfx90a is the right way to set the flag, and the flag should be set for compilation. I will try to investigate a bit more to see if I can find anything else that might be breaking your compilation. Thanks!

@segeljakt Thanks for the update!

Also, where can we find more information about HSA_OVERRIDE_GFX_VERSION and its relation to the AMD GPU architecture (e.g., gfx90a)?

Unfortunately, we do not have official document for HSA_OVERRIDE_GFX_VERSION (since it should be needed for officially supported devices). That’s being said, gfx90a is well tested and you shouldn’t really need HSA_OVERRIDE_GFX_VERSION in this case, especially if hip already works.

This is very interesting. So it was originally working on Singularity without any additional commands, but when you tried to manually bind devices it stopped working. I think this confirms that Singularity was doing something automatically under the hood specifically for ROCm, which might be what is causing the issue.

Hmm, I think it could also be Slurm doing something. Slurm provides the GPUs to the host, and then the host runs Singularity. For example, with:

$ cat script.sh

#!/bin/bash
#SBATCH --job-name=AISPECS
#SBATCH --output=/users/segeljak/results/AISPECS.o%j
#SBATCH --error=/users/segeljak/results/AISPECS.e%j
#SBATCH --account=project_xxx
#SBATCH --time=08:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --gpus-per-task=1
#SBATCH --mem=64G
#SBATCH --partition=standard-g

srun --verbose \
  singularity exec \
  /users/segeljak/aispecs-docker-eval-amd.sif \
  bash -c "source /opt/aispecs-venv/bin/activate;
           AMD_SERIALIZE_KERNEL=3 AMD_LOG_LEVEL=5 python -c \"import torch; a=torch.randn(3).to('cuda'); print(a)\""

$ sbatch script.sh

The gpus become accessible in the command passed to srun. I don’t know exactly how singularity binds them though. It could be that Singularity requires passing the --rocm flag, but then I get the wrong ROCm version.

I just found out that the HPC cluster also supports podman. I’ll see if I can get the container running on it.