ROCm: [Issue]: Intermittent GPU Hang HW Exception by GPU on MI300X when training with axolotl

Problem Description

When running axolotl runs, I get intermittent GPU hangs:

{'loss': 0.4589, 'grad_norm': 1.0493940198290594, 'learning_rate': 5.284132841328413e-06, 'epoch': 1.22}
 41%|██████████████████████████████████████████▌                                                              | 366/903 [5:15:39<6:53:09, 46.16s/it]HW Exception by GPU node-9 (Agent handle: 0x631351bb5a20) reason :GPU Hang
HW Exception by GPU node-9 (Agent handle: 0x64ff272d8ab0) reason :GPU Hang
HW Exception by GPU node-9 (Agent handle: 0x565d634d8610) reason :GPU Hang
HW Exception by GPU node-9 (Agent handle: 0x5ea50312c910) reason :GPU Hang
HW Exception by GPU node-9 (Agent handle: 0x619acaf10aa0) reason :GPU Hang
HW Exception by GPU node-9 (Agent handle: 0x5ab551f9f310) reason :GPU Hang
HW Exception by GPU node-9 (Agent handle: 0x5e2772f37790) reason :GPU Hang
HW Exception by GPU node-9 (Agent handle: 0x5a6c6e4a4680) reason :GPU Hang
HW Exception by GPU node-9 (Agent handle: 0x582b7268cac0) reason :GPU Hang
HW Exception by GPU node-9 (Agent handle: 0x5b5e27441f40) reason :GPU Hang
HW Exception by GPU node-9 (Agent handle: 0x5c6c69e57ff0) reason :GPU Hang
Aborted (core dumped)
(axolotl) 134 hotaisle@ENC1-CLS01-SVR14:~/shisa-v2/train/ablations$ [rank2]:[E1109 16:13:20.222288244 ProcessGroupNCCL.cpp:616] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=500045, OpType=_REDUCE_SCATTER_BASE, NumelIn=67108864, NumelOut=8388608, Timeout(ms)=600000) ran for 600038 milliseconds before timing out.
[rank2]:[E1109 16:13:20.222705810 ProcessGroupNCCL.cpp:1785] [PG ID 1 PG GUID 1 Rank 2] Exception (either an error or timeout) detected by watchdog at work: 500045, last enqueued NCCL work: 500047, last completed NCCL work: 500044.
[rank7]:[E1109 16:13:20.230357202 ProcessGroupNCCL.cpp:616] [Rank 7] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=500045, OpType=_REDUCE_SCATTER_BASE, NumelIn=67108864, NumelOut=8388608, Timeout(ms)=600000) ran for 600046 milliseconds before timing out.
[rank7]:[E1109 16:13:20.230674458 ProcessGroupNCCL.cpp:1785] [PG ID 1 PG GUID 1 Rank 7] Exception (either an error or timeout) detected by watchdog at work: 500045, last enqueued NCCL work: 500047, last completed NCCL work: 500044.
[rank3]:[E1109 16:13:20.249114105 ProcessGroupNCCL.cpp:616] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=500045, OpType=_REDUCE_SCATTER_BASE, NumelIn=67108864, NumelOut=8388608, Timeout(ms)=600000) ran for 600065 milliseconds before timing out.
[rank3]:[E1109 16:13:20.249396998 ProcessGroupNCCL.cpp:1785] [PG ID 1 PG GUID 1 Rank 3] Exception (either an error or timeout) detected by watchdog at work: 500045, last enqueued NCCL work: 500045, last completed NCCL work: 500044.
[rank4]:[E1109 16:13:20.252514562 ProcessGroupNCCL.cpp:616] [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=500045, OpType=_REDUCE_SCATTER_BASE, NumelIn=67108864, NumelOut=8388608, Timeout(ms)=600000) ran for 600068 milliseconds before timing out.
[rank4]:[E1109 16:13:20.252770852 ProcessGroupNCCL.cpp:1785] [PG ID 1 PG GUID 1 Rank 4] Exception (either an error or timeout) detected by watchdog at work: 500045, last enqueued NCCL work: 500045, last completed NCCL work: 500044.
[rank6]:[E1109 16:13:20.262103416 ProcessGroupNCCL.cpp:616] [Rank 6] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=500045, OpType=_REDUCE_SCATTER_BASE, NumelIn=67108864, NumelOut=8388608, Timeout(ms)=600000) ran for 600078 milliseconds before timing out.
[rank6]:[E1109 16:13:20.262386071 ProcessGroupNCCL.cpp:1785] [PG ID 1 PG GUID 1 Rank 6] Exception (either an error or timeout) detected by watchdog at work: 500045, last enqueued NCCL work: 500045, last completed NCCL work: 500044.
[rank1]:[E1109 16:13:20.273208899 ProcessGroupNCCL.cpp:616] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=500045, OpType=_REDUCE_SCATTER_BASE, NumelIn=67108864, NumelOut=8388608, Timeout(ms)=600000) ran for 600089 milliseconds before timing out.
[rank1]:[E1109 16:13:20.273362088 ProcessGroupNCCL.cpp:1785] [PG ID 1 PG GUID 1 Rank 1] Exception (either an error or timeout) detected by watchdog at work: 500045, last enqueued NCCL work: 500047, last completed NCCL work: 500044.

Note, the GPUs are left in a bad state (despite a reset being reported in dmesg, logs included below):

============================================ ROCm System Management Interface ============================================
====================================================== Concise Info ======================================================
Device  Node  IDs              Temp        Power     Partitions          SCLK    MCLK    Fan  Perf  PwrCap  VRAM%  GPU%
^[3m              (DID,     GUID)  (Junction)  (Socket)  (Mem, Compute, ID)                                                  ^[0m
==========================================================================================================================
0       2     0x74a1,   55354  48.0°C      145.0W    NPS1, SPX, 0        132Mhz  900Mhz  0%   auto  750.0W  1%     0%
1       3     0x74a1,   41632  40.0°C      138.0W    NPS1, SPX, 0        131Mhz  900Mhz  0%   auto  750.0W  75%    0%
2       4     0x74a1,   47045  42.0°C      135.0W    NPS1, SPX, 0        132Mhz  900Mhz  0%   auto  750.0W  74%    0%
3       5     0x74a1,   60169  49.0°C      150.0W    NPS1, SPX, 0        132Mhz  900Mhz  0%   auto  750.0W  74%    0%
4       6     0x74a1,   56024  47.0°C      146.0W    NPS1, SPX, 0        132Mhz  900Mhz  0%   auto  750.0W  75%    0%
5       7     0x74a1,   705    42.0°C      139.0W    NPS1, SPX, 0        132Mhz  900Mhz  0%   auto  750.0W  1%     0%
6       8     0x74a1,   59108  48.0°C      147.0W    NPS1, SPX, 0        132Mhz  900Mhz  0%   auto  750.0W  75%    0%
7       9     0x74a1,   10985  40.0°C      138.0W    NPS1, SPX, 0        132Mhz  900Mhz  0%   auto  750.0W  74%    0%
==========================================================================================================================
================================================== End of ROCm SMI Log ===================================================

VRAM remains in use. Running an additional GPU reset does not work: sudo amd-smi reset -g 0 -G, a full system reboot is required to clear this up.

This seems to happen on about half of my runs (that are greater than a few hours).

Operating System

Ubuntu 22.04.5 LTS (Jammy Jellyfish)

CPU

Intel® Xeon® Platinum 8470

GPU

AMD Instinct MI300X

ROCm Version

ROCm 6.2.3

ROCm Component

No response

Steps to Reproduce

Install and run axolotl, eg:

accelerate launch -m axolotl.cli.train mi300x-llama3.1-70b-fft.dsz3.yaml

You can see https://github.com/AUGMXNT/MI300-testing/ or https://github.com/shisa-ai/shisa-v2/tree/main/train/ablations for some sample yaml files (I am running llama3-70b ffts that take about 12-20h to do)

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

ROCk module version 6.8.5 is loaded
=====================
HSA System Attributes
=====================
Runtime Version:         1.14
Runtime Ext Version:     1.6
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE
System Endianness:       LITTLE
Mwaitx:                  DISABLED
DMAbuf Support:          YES

==========
HSA Agents
==========
*******
Agent 1
*******
  Name:                    Intel(R) Xeon(R) Platinum 8470
  Uuid:                    CPU-XX
  Marketing Name:          Intel(R) Xeon(R) Platinum 8470
  Vendor Name:             CPU
  Feature:                 None specified
  Profile:                 FULL_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        0(0x0)
  Queue Min Size:          0(0x0)
  Queue Max Size:          0(0x0)
  Queue Type:              MULTI
  Node:                    0
  Device Type:             CPU
  Cache Info:
    L1:                      49152(0xc000) KB
  Chip ID:                 0(0x0)
  ASIC Revision:           0(0x0)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   3800
  BDFID:                   0
  Internal Node ID:        0
  Compute Unit:            104
  SIMDs per CU:            0
  Shader Engines:          0
  Shader Arrs. per Eng.:   0
  WatchPts on Addr. Ranges:1
  Memory Properties:
  Features:                None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    1056364836(0x3ef6d924) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 2
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    1056364836(0x3ef6d924) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 3
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    1056364836(0x3ef6d924) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
  ISA Info:
*******
Agent 2
*******
  Name:                    Intel(R) Xeon(R) Platinum 8470
  Uuid:                    CPU-XX
  Marketing Name:          Intel(R) Xeon(R) Platinum 8470
  Vendor Name:             CPU
  Feature:                 None specified
  Profile:                 FULL_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        0(0x0)
  Queue Min Size:          0(0x0)
  Queue Max Size:          0(0x0)
  Queue Type:              MULTI
  Node:                    1
  Device Type:             CPU
  Cache Info:
    L1:                      49152(0xc000) KB
  Chip ID:                 0(0x0)
  ASIC Revision:           0(0x0)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   3800
  BDFID:                   0
  Internal Node ID:        1
  Compute Unit:            104
  SIMDs per CU:            0
  Shader Engines:          0
  Shader Arrs. per Eng.:   0
  WatchPts on Addr. Ranges:1
  Memory Properties:
  Features:                None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    1056871544(0x3efe9478) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 2
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    1056871544(0x3efe9478) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 3
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    1056871544(0x3efe9478) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
  ISA Info:
*******
Agent 3
*******
  Name:                    gfx942
  Uuid:                    GPU-07386a321fa45d1e
  Marketing Name:          AMD Instinct MI300X
  Vendor Name:             AMD
  Feature:                 KERNEL_DISPATCH
  Profile:                 BASE_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        128(0x80)
  Queue Min Size:          64(0x40)
  Queue Max Size:          131072(0x20000)
  Queue Type:              MULTI
  Node:                    2
  Device Type:             GPU
  Cache Info:
    L1:                      32(0x20) KB
    L2:                      4096(0x1000) KB
    L3:                      262144(0x40000) KB
  Chip ID:                 29857(0x74a1)
  ASIC Revision:           1(0x1)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   2100
  BDFID:                   6912
  Internal Node ID:        2
  Compute Unit:            304
  SIMDs per CU:            4
  Shader Engines:          32
  Shader Arrs. per Eng.:   1
  WatchPts on Addr. Ranges:4
  Coherent Host Access:    FALSE
  Memory Properties:
  Features:                KERNEL_DISPATCH
  Fast F16 Operation:      TRUE
  Wavefront Size:          64(0x40)
  Workgroup Max Size:      1024(0x400)
  Workgroup Max Size per Dimension:
    x                        1024(0x400)
    y                        1024(0x400)
    z                        1024(0x400)
  Max Waves Per CU:        32(0x20)
  Max Work-item Per CU:    2048(0x800)
  Grid Max Size:           4294967295(0xffffffff)
  Grid Max Size per Dimension:
    x                        4294967295(0xffffffff)
    y                        4294967295(0xffffffff)
    z                        4294967295(0xffffffff)
  Max fbarriers/Workgrp:   32
  Packet Processor uCode:: 150
  SDMA engine uCode::      21
  IOMMU Support::          None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 2
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 3
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 4
      Segment:                 GROUP
      Size:                    64(0x40) KB
      Allocatable:             FALSE
      Alloc Granule:           0KB
      Alloc Recommended Granule:0KB
      Alloc Alignment:         0KB
      Accessible by all:       FALSE
  ISA Info:
    ISA 1
      Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:           4294967295(0xffffffff)
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)
        y                        4294967295(0xffffffff)
        z                        4294967295(0xffffffff)
      FBarrier Max Size:       32
*******
Agent 4
*******
  Name:                    gfx942
  Uuid:                    GPU-c14abb436c69353f
  Marketing Name:          AMD Instinct MI300X
  Vendor Name:             AMD
  Feature:                 KERNEL_DISPATCH
  Profile:                 BASE_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        128(0x80)
  Queue Min Size:          64(0x40)
  Queue Max Size:          131072(0x20000)
  Queue Type:              MULTI
  Node:                    3
  Device Type:             GPU
  Cache Info:
    L1:                      32(0x20) KB
    L2:                      4096(0x1000) KB
    L3:                      262144(0x40000) KB
  Chip ID:                 29857(0x74a1)
  ASIC Revision:           1(0x1)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   2100
  BDFID:                   15616
  Internal Node ID:        3
  Compute Unit:            304
  SIMDs per CU:            4
  Shader Engines:          32
  Shader Arrs. per Eng.:   1
  WatchPts on Addr. Ranges:4
  Coherent Host Access:    FALSE
  Memory Properties:
  Features:                KERNEL_DISPATCH
  Fast F16 Operation:      TRUE
  Wavefront Size:          64(0x40)
  Workgroup Max Size:      1024(0x400)
  Workgroup Max Size per Dimension:
    x                        1024(0x400)
    y                        1024(0x400)
    z                        1024(0x400)
  Max Waves Per CU:        32(0x20)
  Max Work-item Per CU:    2048(0x800)
  Grid Max Size:           4294967295(0xffffffff)
  Grid Max Size per Dimension:
    x                        4294967295(0xffffffff)
    y                        4294967295(0xffffffff)
    z                        4294967295(0xffffffff)
  Max fbarriers/Workgrp:   32
  Packet Processor uCode:: 150
  SDMA engine uCode::      21
  IOMMU Support::          None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 2
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 3
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 4
      Segment:                 GROUP
      Size:                    64(0x40) KB
      Allocatable:             FALSE
      Alloc Granule:           0KB
      Alloc Recommended Granule:0KB
      Alloc Alignment:         0KB
      Accessible by all:       FALSE
  ISA Info:
    ISA 1
      Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:           4294967295(0xffffffff)
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)
        y                        4294967295(0xffffffff)
        z                        4294967295(0xffffffff)
      FBarrier Max Size:       32
*******
Agent 5
*******
  Name:                    gfx942
  Uuid:                    GPU-4be0e20b12ab7fa1
  Marketing Name:          AMD Instinct MI300X
  Vendor Name:             AMD
  Feature:                 KERNEL_DISPATCH
  Profile:                 BASE_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        128(0x80)
  Queue Min Size:          64(0x40)
  Queue Max Size:          131072(0x20000)
  Queue Type:              MULTI
  Node:                    4
  Device Type:             GPU
  Cache Info:
    L1:                      32(0x20) KB
    L2:                      4096(0x1000) KB
    L3:                      262144(0x40000) KB
  Chip ID:                 29857(0x74a1)
  ASIC Revision:           1(0x1)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   2100
  BDFID:                   19968
  Internal Node ID:        4
  Compute Unit:            304
  SIMDs per CU:            4
  Shader Engines:          32
  Shader Arrs. per Eng.:   1
  WatchPts on Addr. Ranges:4
  Coherent Host Access:    FALSE
  Memory Properties:
  Features:                KERNEL_DISPATCH
  Fast F16 Operation:      TRUE
  Wavefront Size:          64(0x40)
  Workgroup Max Size:      1024(0x400)
  Workgroup Max Size per Dimension:
    x                        1024(0x400)
    y                        1024(0x400)
    z                        1024(0x400)
  Max Waves Per CU:        32(0x20)
  Max Work-item Per CU:    2048(0x800)
  Grid Max Size:           4294967295(0xffffffff)
  Grid Max Size per Dimension:
    x                        4294967295(0xffffffff)
    y                        4294967295(0xffffffff)
    z                        4294967295(0xffffffff)
  Max fbarriers/Workgrp:   32
  Packet Processor uCode:: 150
  SDMA engine uCode::      21
  IOMMU Support::          None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 2
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 3
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 4
      Segment:                 GROUP
      Size:                    64(0x40) KB
      Allocatable:             FALSE
      Alloc Granule:           0KB
      Alloc Recommended Granule:0KB
      Alloc Alignment:         0KB
      Accessible by all:       FALSE
  ISA Info:
    ISA 1
      Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:           4294967295(0xffffffff)
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)
        y                        4294967295(0xffffffff)
        z                        4294967295(0xffffffff)
      FBarrier Max Size:       32
*******
Agent 6
*******
  Name:                    gfx942
  Uuid:                    GPU-18c80aec63536c85
  Marketing Name:          AMD Instinct MI300X
  Vendor Name:             AMD
  Feature:                 KERNEL_DISPATCH
  Profile:                 BASE_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        128(0x80)
  Queue Min Size:          64(0x40)
  Queue Max Size:          131072(0x20000)
  Queue Type:              MULTI
  Node:                    5
  Device Type:             GPU
  Cache Info:
    L1:                      32(0x20) KB
    L2:                      4096(0x1000) KB
    L3:                      262144(0x40000) KB
  Chip ID:                 29857(0x74a1)
  ASIC Revision:           1(0x1)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   2100
  BDFID:                   24320
  Internal Node ID:        5
  Compute Unit:            304
  SIMDs per CU:            4
  Shader Engines:          32
  Shader Arrs. per Eng.:   1
  WatchPts on Addr. Ranges:4
  Coherent Host Access:    FALSE
  Memory Properties:
  Features:                KERNEL_DISPATCH
  Fast F16 Operation:      TRUE
  Wavefront Size:          64(0x40)
  Workgroup Max Size:      1024(0x400)
  Workgroup Max Size per Dimension:
    x                        1024(0x400)
    y                        1024(0x400)
    z                        1024(0x400)
  Max Waves Per CU:        32(0x20)
  Max Work-item Per CU:    2048(0x800)
  Grid Max Size:           4294967295(0xffffffff)
  Grid Max Size per Dimension:
    x                        4294967295(0xffffffff)
    y                        4294967295(0xffffffff)
    z                        4294967295(0xffffffff)
  Max fbarriers/Workgrp:   32
  Packet Processor uCode:: 150
  SDMA engine uCode::      21
  IOMMU Support::          None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 2
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 3
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 4
      Segment:                 GROUP
      Size:                    64(0x40) KB
      Allocatable:             FALSE
      Alloc Granule:           0KB
      Alloc Recommended Granule:0KB
      Alloc Alignment:         0KB
      Accessible by all:       FALSE
  ISA Info:
    ISA 1
      Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:           4294967295(0xffffffff)
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)
        y                        4294967295(0xffffffff)
        z                        4294967295(0xffffffff)
      FBarrier Max Size:       32
*******
Agent 7
*******
  Name:                    gfx942
  Uuid:                    GPU-98c78750297b3198
  Marketing Name:          AMD Instinct MI300X
  Vendor Name:             AMD
  Feature:                 KERNEL_DISPATCH
  Profile:                 BASE_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        128(0x80)
  Queue Min Size:          64(0x40)
  Queue Max Size:          131072(0x20000)
  Queue Type:              MULTI
  Node:                    6
  Device Type:             GPU
  Cache Info:
    L1:                      32(0x20) KB
    L2:                      4096(0x1000) KB
    L3:                      262144(0x40000) KB
  Chip ID:                 29857(0x74a1)
  ASIC Revision:           1(0x1)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   2100
  BDFID:                   40192
  Internal Node ID:        6
  Compute Unit:            304
  SIMDs per CU:            4
  Shader Engines:          32
  Shader Arrs. per Eng.:   1
  WatchPts on Addr. Ranges:4
  Coherent Host Access:    FALSE
  Memory Properties:
  Features:                KERNEL_DISPATCH
  Fast F16 Operation:      TRUE
  Wavefront Size:          64(0x40)
  Workgroup Max Size:      1024(0x400)
  Workgroup Max Size per Dimension:
    x                        1024(0x400)
    y                        1024(0x400)
    z                        1024(0x400)
  Max Waves Per CU:        32(0x20)
  Max Work-item Per CU:    2048(0x800)
  Grid Max Size:           4294967295(0xffffffff)
  Grid Max Size per Dimension:
    x                        4294967295(0xffffffff)
    y                        4294967295(0xffffffff)
    z                        4294967295(0xffffffff)
  Max fbarriers/Workgrp:   32
  Packet Processor uCode:: 150
  SDMA engine uCode::      21
  IOMMU Support::          None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 2
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 3
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 4
      Segment:                 GROUP
      Size:                    64(0x40) KB
      Allocatable:             FALSE
      Alloc Granule:           0KB
      Alloc Recommended Granule:0KB
      Alloc Alignment:         0KB
      Accessible by all:       FALSE
  ISA Info:
    ISA 1
      Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:           4294967295(0xffffffff)
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)
        y                        4294967295(0xffffffff)
        z                        4294967295(0xffffffff)
      FBarrier Max Size:       32
*******
Agent 8
*******
  Name:                    gfx942
  Uuid:                    GPU-c8d0ce1363e0130e
  Marketing Name:          AMD Instinct MI300X
  Vendor Name:             AMD
  Feature:                 KERNEL_DISPATCH
  Profile:                 BASE_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        128(0x80)
  Queue Min Size:          64(0x40)
  Queue Max Size:          131072(0x20000)
  Queue Type:              MULTI
  Node:                    7
  Device Type:             GPU
  Cache Info:
    L1:                      32(0x20) KB
    L2:                      4096(0x1000) KB
    L3:                      262144(0x40000) KB
  Chip ID:                 29857(0x74a1)
  ASIC Revision:           1(0x1)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   2100
  BDFID:                   48384
  Internal Node ID:        7
  Compute Unit:            304
  SIMDs per CU:            4
  Shader Engines:          32
  Shader Arrs. per Eng.:   1
  WatchPts on Addr. Ranges:4
  Coherent Host Access:    FALSE
  Memory Properties:
  Features:                KERNEL_DISPATCH
  Fast F16 Operation:      TRUE
  Wavefront Size:          64(0x40)
  Workgroup Max Size:      1024(0x400)
  Workgroup Max Size per Dimension:
    x                        1024(0x400)
    y                        1024(0x400)
    z                        1024(0x400)
  Max Waves Per CU:        32(0x20)
  Max Work-item Per CU:    2048(0x800)
  Grid Max Size:           4294967295(0xffffffff)
  Grid Max Size per Dimension:
    x                        4294967295(0xffffffff)
    y                        4294967295(0xffffffff)
    z                        4294967295(0xffffffff)
  Max fbarriers/Workgrp:   32
  Packet Processor uCode:: 150
  SDMA engine uCode::      21
  IOMMU Support::          None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 2
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 3
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 4
      Segment:                 GROUP
      Size:                    64(0x40) KB
      Allocatable:             FALSE
      Alloc Granule:           0KB
      Alloc Recommended Granule:0KB
      Alloc Alignment:         0KB
      Accessible by all:       FALSE
  ISA Info:
    ISA 1
      Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:           4294967295(0xffffffff)
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)
        y                        4294967295(0xffffffff)
        z                        4294967295(0xffffffff)
      FBarrier Max Size:       32
*******
Agent 9
*******
  Name:                    gfx942
  Uuid:                    GPU-fdee8fae2de9f2b6
  Marketing Name:          AMD Instinct MI300X
  Vendor Name:             AMD
  Feature:                 KERNEL_DISPATCH
  Profile:                 BASE_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        128(0x80)
  Queue Min Size:          64(0x40)
  Queue Max Size:          131072(0x20000)
  Queue Type:              MULTI
  Node:                    8
  Device Type:             GPU
  Cache Info:
    L1:                      32(0x20) KB
    L2:                      4096(0x1000) KB
    L3:                      262144(0x40000) KB
  Chip ID:                 29857(0x74a1)
  ASIC Revision:           1(0x1)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   2100
  BDFID:                   52480
  Internal Node ID:        8
  Compute Unit:            304
  SIMDs per CU:            4
  Shader Engines:          32
  Shader Arrs. per Eng.:   1
  WatchPts on Addr. Ranges:4
  Coherent Host Access:    FALSE
  Memory Properties:
  Features:                KERNEL_DISPATCH
  Fast F16 Operation:      TRUE
  Wavefront Size:          64(0x40)
  Workgroup Max Size:      1024(0x400)
  Workgroup Max Size per Dimension:
    x                        1024(0x400)
    y                        1024(0x400)
    z                        1024(0x400)
  Max Waves Per CU:        32(0x20)
  Max Work-item Per CU:    2048(0x800)
  Grid Max Size:           4294967295(0xffffffff)
  Grid Max Size per Dimension:
    x                        4294967295(0xffffffff)
    y                        4294967295(0xffffffff)
    z                        4294967295(0xffffffff)
  Max fbarriers/Workgrp:   32
  Packet Processor uCode:: 150
  SDMA engine uCode::      21
  IOMMU Support::          None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 2
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 3
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 4
      Segment:                 GROUP
      Size:                    64(0x40) KB
      Allocatable:             FALSE
      Alloc Granule:           0KB
      Alloc Recommended Granule:0KB
      Alloc Alignment:         0KB
      Accessible by all:       FALSE
  ISA Info:
    ISA 1
      Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:           4294967295(0xffffffff)
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)
        y                        4294967295(0xffffffff)
        z                        4294967295(0xffffffff)
      FBarrier Max Size:       32
*******
Agent 10
*******
  Name:                    gfx942
  Uuid:                    GPU-6f890acb8eb0492b
  Marketing Name:          AMD Instinct MI300X
  Vendor Name:             AMD
  Feature:                 KERNEL_DISPATCH
  Profile:                 BASE_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        128(0x80)
  Queue Min Size:          64(0x40)
  Queue Max Size:          131072(0x20000)
  Queue Type:              MULTI
  Node:                    9
  Device Type:             GPU
  Cache Info:
    L1:                      32(0x20) KB
    L2:                      4096(0x1000) KB
    L3:                      262144(0x40000) KB
  Chip ID:                 29857(0x74a1)
  ASIC Revision:           1(0x1)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   2100
  BDFID:                   56576
  Internal Node ID:        9
  Compute Unit:            304
  SIMDs per CU:            4
  Shader Engines:          32
  Shader Arrs. per Eng.:   1
  WatchPts on Addr. Ranges:4
  Coherent Host Access:    FALSE
  Memory Properties:
  Features:                KERNEL_DISPATCH
  Fast F16 Operation:      TRUE
  Wavefront Size:          64(0x40)
  Workgroup Max Size:      1024(0x400)
  Workgroup Max Size per Dimension:
    x                        1024(0x400)
    y                        1024(0x400)
    z                        1024(0x400)
  Max Waves Per CU:        32(0x20)
  Max Work-item Per CU:    2048(0x800)
  Grid Max Size:           4294967295(0xffffffff)
  Grid Max Size per Dimension:
    x                        4294967295(0xffffffff)
    y                        4294967295(0xffffffff)
    z                        4294967295(0xffffffff)
  Max fbarriers/Workgrp:   32
  Packet Processor uCode:: 150
  SDMA engine uCode::      21
  IOMMU Support::          None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 2
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 3
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    201310208(0xbffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 4
      Segment:                 GROUP
      Size:                    64(0x40) KB
      Allocatable:             FALSE
      Alloc Granule:           0KB
      Alloc Recommended Granule:0KB
      Alloc Alignment:         0KB
      Accessible by all:       FALSE
  ISA Info:
    ISA 1
      Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:           4294967295(0xffffffff)
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)
        y                        4294967295(0xffffffff)
        z                        4294967295(0xffffffff)
      FBarrier Max Size:       32
*** Done ***

Additional Information

Here’s what the dmesg looks like:

[76914.152141] amdgpu 0000:dd:00.0: {1}uncorrectable hardware error(ERREVENT_ATHUB_INTERRUPT) detected!
[76914.152174] amdgpu 0000:3d:00.0: amdgpu: The cp might be in an unrecoverable state due to an unsuccessful queues preemption
[76914.152926] amdgpu 0000:1b:00.0: {1}[Hardware Error]: Accelerator Check Architecture events logged
[76914.152940] amdgpu 0000:1b:00.0: {1}[Hardware Error]: aca entry[00].STATUS=0xbaa000000000080b
[76914.152948] amdgpu 0000:1b:00.0: {1}[Hardware Error]: aca entry[00].ADDR=0x0000000000000000
[76914.152953] amdgpu 0000:1b:00.0: {1}[Hardware Error]: aca entry[00].MISC0=0xd008000100000000
[76914.152957] amdgpu 0000:1b:00.0: {1}[Hardware Error]: aca entry[00].IPID=0x0001d02e1f007900
[76914.152961] amdgpu 0000:1b:00.0: {1}[Hardware Error]: aca entry[00].SYND=0x000000005d003e1b
[76914.156884] amdgpu: Failed to restore process queues
[76914.156901] amdgpu: Failed to restore process queues
[76914.156914] amdgpu: Failed to restore process queues
[76914.156925] amdgpu: Failed to restore process queues
[76914.156935] amdgpu: Failed to restore process queues
[76914.156945] amdgpu: Failed to restore process queues
[76914.156956] amdgpu: Failed to restore process queues
[76914.156965] amdgpu: Failed to restore process queues
[76914.184889] amdgpu 0000:9d:00.0: {1}[Hardware Error]: Accelerator Check Architecture events logged
[76914.184894] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[00].STATUS=0xd820000000060150
[76914.184898] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[00].ADDR=0x0000000000000000
[76914.184902] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[00].MISC0=0xd008005100000000
[76914.184906] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[00].IPID=0x0000605011a09201
[76914.184910] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[00].SYND=0x009c732d4a000000
[76914.185363] amdgpu 0000:9d:00.0: {1}[Hardware Error]: Accelerator Check Architecture events logged
[76914.185366] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[01].STATUS=0xd820000000060150
[76914.185370] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[01].ADDR=0x0000000000000000
[76914.185373] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[01].MISC0=0xd008003100000000
[76914.185377] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[01].IPID=0x0000605011a09201
[76914.185381] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[01].SYND=0x009c752d4a000000
[76914.185838] amdgpu 0000:9d:00.0: {1}[Hardware Error]: Accelerator Check Architecture events logged
[76914.185842] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[02].STATUS=0xd820000000060400
[76914.185846] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[02].ADDR=0x0000000000000000
[76914.185850] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[02].MISC0=0xd008008100000000
[76914.185853] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[02].IPID=0x0000605011a09201
[76914.185857] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[02].SYND=0x00000000809c812d
[76914.186309] amdgpu 0000:9d:00.0: {1}[Hardware Error]: Accelerator Check Architecture events logged
[76914.186313] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[03].STATUS=0xd820000000060150
[76914.186316] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[03].ADDR=0x0000000000000000
[76914.186320] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[03].MISC0=0xd008000400000000
[76914.186324] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[03].IPID=0x0000605011a09201
[76914.186327] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[03].SYND=0x009c812d4a000000
[76914.186783] amdgpu 0000:9d:00.0: {1}[Hardware Error]: Accelerator Check Architecture events logged
[76914.186786] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[04].STATUS=0xd820000000060150
[76914.186790] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[04].ADDR=0x0000000000000000
[76914.186794] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[04].MISC0=0xd00800f800000000
[76914.186797] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[04].IPID=0x0000605011a09201
[76914.186801] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[04].SYND=0x009c812d4a000000
[76914.187259] amdgpu 0000:9d:00.0: {1}[Hardware Error]: Accelerator Check Architecture events logged
[76914.187262] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[05].STATUS=0xd820000000060150
[76914.187266] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[05].ADDR=0x0000000000000000
[76914.187269] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[05].MISC0=0xd008005100000000
[76914.187273] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[05].IPID=0x0000605011a09201
[76914.187277] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[05].SYND=0x009c752d4a000000
[76914.187740] amdgpu 0000:9d:00.0: {1}[Hardware Error]: Accelerator Check Architecture events logged
[76914.187744] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[06].STATUS=0xd820000000060150
[76914.187748] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[06].ADDR=0x0000000000000000
[76914.187751] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[06].MISC0=0xd008004000000000
[76914.187755] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[06].IPID=0x0000605011a09201
[76914.187759] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[06].SYND=0x009c752d4a000000
[76914.188214] amdgpu 0000:9d:00.0: {1}[Hardware Error]: Accelerator Check Architecture events logged
[76914.188217] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[07].STATUS=0xd820000000060150
[76914.188221] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[07].ADDR=0x0000000000000000
[76914.188225] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[07].MISC0=0xd008002000000000
[76914.188228] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[07].IPID=0x0000605011a09201
[76914.188232] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[07].SYND=0x009c752d4a000000
[76914.188696] amdgpu 0000:9d:00.0: {1}[Hardware Error]: Accelerator Check Architecture events logged
[76914.188700] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[08].STATUS=0xd820000000060150
[76914.188704] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[08].ADDR=0x0000000000000000
[76914.188708] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[08].MISC0=0xd00800a400000000
[76914.188711] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[08].IPID=0x0000605011a09201
[76914.188715] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[08].SYND=0x009c752d4a000000
[76914.189163] amdgpu 0000:9d:00.0: {1}[Hardware Error]: Accelerator Check Architecture events logged
[76914.189166] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[09].STATUS=0xd820000000060150
[76914.189169] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[09].ADDR=0x0000000000000000
[76914.189172] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[09].MISC0=0xd008005000000000
[76914.189175] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[09].IPID=0x0000605011a09201
[76914.189178] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[09].SYND=0x009c752d4a000000
[76914.189644] amdgpu 0000:9d:00.0: {1}[Hardware Error]: Accelerator Check Architecture events logged
[76914.189647] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[10].STATUS=0xd820000000060150
[76914.189651] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[10].ADDR=0x0000000000000000
[76914.189655] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[10].MISC0=0xd008012400000000
[76914.189658] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[10].IPID=0x0000605011a09201
[76914.189662] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[10].SYND=0x009c812d4a000000
[76914.190110] amdgpu 0000:9d:00.0: {1}[Hardware Error]: Accelerator Check Architecture events logged
[76914.190112] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[11].STATUS=0xd820000000060150
[76914.190116] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[11].ADDR=0x0000000000000000
[76914.190119] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[11].MISC0=0xd008013500000000
[76914.190122] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[11].IPID=0x0000605011a09201
[76914.190126] amdgpu 0000:9d:00.0: {1}[Hardware Error]: aca entry[11].SYND=0x009c812d4a000000
[76914.190522] amdgpu 0000:9d:00.0: {1}socket: 6, die: 1, 1533 new correctable hardware errors detected in xgmi_wafl block
[76914.190529] amdgpu 0000:9d:00.0: {1}socket: 6, die: 1, 1533 correctable hardware errors detected in total in xgmi_wafl block
[76914.214195] amdgpu 0000:dd:00.0: amdgpu: GPU reset begin!
[76914.216568] Pid 1230312(python3.12) over core_pipe_limit
[76914.216571] Skipping core dump
[76914.237823] amdgpu 0000:dd:00.0: amdgpu: Dumping IP State
[76914.237826] amdgpu 0000:dd:00.0: amdgpu: Dumping IP State Completed
[76914.238315] amdgpu 0000:1b:00.0: amdgpu: MODE1 reset
[76914.238316] amdgpu 0000:dd:00.0: amdgpu: MODE1 reset
[76914.238321] amdgpu 0000:dd:00.0: amdgpu: GPU mode1 reset
[76914.238320] amdgpu 0000:3d:00.0: amdgpu: MODE1 reset
[76914.238322] amdgpu 0000:1b:00.0: amdgpu: GPU mode1 reset
[76914.238321] amdgpu 0000:4e:00.0: amdgpu: MODE1 reset
[76914.238322] amdgpu 0000:bd:00.0: amdgpu: MODE1 reset
[76914.238323] amdgpu 0000:9d:00.0: amdgpu: MODE1 reset
[76914.238323] amdgpu 0000:5f:00.0: amdgpu: MODE1 reset
[76914.238324] amdgpu 0000:cd:00.0: amdgpu: MODE1 reset
[76914.238329] amdgpu 0000:9d:00.0: amdgpu: GPU mode1 reset
[76914.238330] amdgpu 0000:5f:00.0: amdgpu: GPU mode1 reset
[76914.238332] amdgpu 0000:bd:00.0: amdgpu: GPU mode1 reset
[76914.238334] amdgpu 0000:4e:00.0: amdgpu: GPU mode1 reset
[76914.238336] amdgpu 0000:cd:00.0: amdgpu: GPU mode1 reset
[76914.238337] amdgpu 0000:3d:00.0: amdgpu: GPU mode1 reset
[76914.240668] amdgpu 0000:dd:00.0: amdgpu: GPU smu mode1 reset
[76914.241079] amdgpu 0000:cd:00.0: amdgpu: GPU smu mode1 reset
[76914.241104] amdgpu 0000:1b:00.0: amdgpu: GPU smu mode1 reset
[76914.241152] amdgpu 0000:5f:00.0: amdgpu: GPU smu mode1 reset
[76914.241257] amdgpu 0000:3d:00.0: amdgpu: GPU smu mode1 reset
[76914.241282] amdgpu 0000:4e:00.0: amdgpu: GPU smu mode1 reset
[76914.243199] amdgpu 0000:9d:00.0: amdgpu: GPU smu mode1 reset
[76914.243283] amdgpu 0000:bd:00.0: amdgpu: GPU smu mode1 reset
[76919.010450] amdgpu 0000:dd:00.0: amdgpu: GPU reset succeeded, trying to resume
[76919.011774] [drm] PCIE GART of 512M enabled.
[76919.011779] [drm] PTB located at 0x000002EFED700000
[76919.011910] [drm] VRAM is lost due to GPU reset!

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 38 (9 by maintainers)

Most upvoted comments

Hi @jamesxu2, I’ve seen a GPU Hang error a couple times on a different training stack on ROCm 6.3.1. I don’t know if it’s the same issue, but is the fix for this out yet? Should I expect the issue to be resolved by upgrading to 6.3.2?

Update: AMD is looking into this internally. Hopefully we know more soon.

This issue has been assessed with the help of HotAisle, @lhl and @anthonix, and we have resolved it internally.

Here is the most recent failure just checking on my latest traiing run, it was only a single node hang and didn’t leave memory resident. This was 4:51:05<8:54:01 (5h into a ~14h run). Error:

 33%|███████████████████Memory access fault by GPU node-3 (Agent handle: 0x5a5c587a51c0) on address 0x709843600000. Reason: Write access to a read-only page.:54:01, 26.24s/it]

from dmesg (if there’s a better place to look at more useful errors let me know:

[19277.975559] amdgpu 0000:3d:00.0: amdgpu: [mmhub0] retry page fault (src_id:0 ring:0 vmid:3 pasid:32790)
[19277.975569] amdgpu 0000:3d:00.0: amdgpu: [mmhub0] no-retry page fault (src_id:0 ring:160 vmid:3 pasid:32790)
[19277.975611] amdgpu 0000:3d:00.0: amdgpu:  for process python3.12 pid 6071 thread python3.12 pid 6071)
[19277.975625] amdgpu 0000:3d:00.0: amdgpu:   in page starting at address 0x0000709843600000 from IH client 0x12 (VMC)
[19277.975640] amdgpu 0000:3d:00.0: amdgpu:   cookie node_id 0 fault from die AID0
[19277.975655] amdgpu 0000:3d:00.0: amdgpu: VM_L2_PROTECTION_FAULT_STATUS:0x00360251
[19277.975667] amdgpu 0000:3d:00.0: amdgpu:      Faulty UTCL2 client ID: SDMA1 (0x101)
[19277.975679] amdgpu 0000:3d:00.0: amdgpu:      MORE_FAULTS: 0x1
[19277.975694] amdgpu 0000:3d:00.0: amdgpu:      WALKER_ERROR: 0x0
[19277.975702] amdgpu 0000:3d:00.0: amdgpu:      PERMISSION_FAULTS: 0x5
[19277.975711] amdgpu 0000:3d:00.0: amdgpu:      MAPPING_ERROR: 0x0
[19277.975722] amdgpu 0000:3d:00.0: amdgpu:      RW: 0x1
[19277.975743] amdgpu 0000:3d:00.0: amdgpu: [mmhub0] no-retry page fault (src_id:0 ring:160 vmid:3 pasid:32790)
[19277.975762] amdgpu 0000:3d:00.0: amdgpu:  for process python3.12 pid 6071 thread python3.12 pid 6071)
[19277.975782] amdgpu 0000:3d:00.0: amdgpu:   in page starting at address 0x0000709843600000 from IH client 0x12 (VMC)
[19277.975802] amdgpu 0000:3d:00.0: amdgpu:   cookie node_id 0 fault from die AID0
[19277.975819] amdgpu 0000:3d:00.0: amdgpu: VM_L2_PROTECTION_FAULT_STATUS:0x00360341
[19277.975834] amdgpu 0000:3d:00.0: amdgpu:      Faulty UTCL2 client ID: SDMA1 (0x101)
[19277.975848] amdgpu 0000:3d:00.0: amdgpu:      MORE_FAULTS: 0x1
[19277.975859] amdgpu 0000:3d:00.0: amdgpu:      WALKER_ERROR: 0x0
[19277.975871] amdgpu 0000:3d:00.0: amdgpu:      PERMISSION_FAULTS: 0x4
[19277.975882] amdgpu 0000:3d:00.0: amdgpu:      MAPPING_ERROR: 0x1
[19277.975893] amdgpu 0000:3d:00.0: amdgpu:      RW: 0x1
[19277.975903] amdgpu 0000:3d:00.0: amdgpu: [mmhub0] no-retry page fault (src_id:0 ring:160 vmid:3 pasid:32790)
[19277.975920] amdgpu 0000:3d:00.0: amdgpu:  for process python3.12 pid 6071 thread python3.12 pid 6071)
[19277.975933] amdgpu 0000:3d:00.0: amdgpu:   in page starting at address 0x0000709843600000 from IH client 0x12 (VMC)
[19277.975948] amdgpu 0000:3d:00.0: amdgpu:   cookie node_id 0 fault from die AID0
[19277.975964] amdgpu 0000:3d:00.0: amdgpu: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[19277.975975] amdgpu 0000:3d:00.0: amdgpu:      Faulty UTCL2 client ID: unknown (0x0)
[19277.975987] amdgpu 0000:3d:00.0: amdgpu:      MORE_FAULTS: 0x0
[19277.975996] amdgpu 0000:3d:00.0: amdgpu:      WALKER_ERROR: 0x0
[19277.976005] amdgpu 0000:3d:00.0: amdgpu:      PERMISSION_FAULTS: 0x0
[19277.976015] amdgpu 0000:3d:00.0: amdgpu:      MAPPING_ERROR: 0x0
[19277.976025] amdgpu 0000:3d:00.0: amdgpu:      RW: 0x0
[19277.976034] amdgpu 0000:3d:00.0: amdgpu: [mmhub0] no-retry page fault (src_id:0 ring:160 vmid:3 pasid:32790)
[19277.976051] amdgpu 0000:3d:00.0: amdgpu:  for process python3.12 pid 6071 thread python3.12 pid 6071)
[19277.976064] amdgpu 0000:3d:00.0: amdgpu:   in page starting at address 0x0000709843600000 from IH client 0x12 (VMC)
[19277.976079] amdgpu 0000:3d:00.0: amdgpu:   cookie node_id 0 fault from die AID0
[19277.976095] amdgpu 0000:3d:00.0: amdgpu: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[19277.976106] amdgpu 0000:3d:00.0: amdgpu:      Faulty UTCL2 client ID: unknown (0x0)
[19277.976118] amdgpu 0000:3d:00.0: amdgpu:      MORE_FAULTS: 0x0
[19277.976128] amdgpu 0000:3d:00.0: amdgpu:      WALKER_ERROR: 0x0
[19277.976137] amdgpu 0000:3d:00.0: amdgpu:      PERMISSION_FAULTS: 0x0
[19277.976147] amdgpu 0000:3d:00.0: amdgpu:      MAPPING_ERROR: 0x0
[19277.976425] amdgpu 0000:3d:00.0: amdgpu:      RW: 0x0
[19277.976679] amdgpu 0000:3d:00.0: amdgpu: [mmhub0] no-retry page fault (src_id:0 ring:160 vmid:3 pasid:32790)
[19277.976942] amdgpu 0000:3d:00.0: amdgpu:  for process python3.12 pid 6071 thread python3.12 pid 6071)
[19277.977192] amdgpu 0000:3d:00.0: amdgpu:   in page starting at address 0x0000709843600000 from IH client 0x12 (VMC)
[19277.977437] amdgpu 0000:3d:00.0: amdgpu:   cookie node_id 0 fault from die AID0
[19277.977680] amdgpu 0000:3d:00.0: amdgpu: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[19277.977919] amdgpu 0000:3d:00.0: amdgpu:      Faulty UTCL2 client ID: unknown (0x0)
[19277.978154] amdgpu 0000:3d:00.0: amdgpu:      MORE_FAULTS: 0x0
[19277.978383] amdgpu 0000:3d:00.0: amdgpu:      WALKER_ERROR: 0x0
[19277.978610] amdgpu 0000:3d:00.0: amdgpu:      PERMISSION_FAULTS: 0x0
[19277.978837] amdgpu 0000:3d:00.0: amdgpu:      MAPPING_ERROR: 0x0
[19277.979061] amdgpu 0000:3d:00.0: amdgpu:      RW: 0x0
[19277.979305] amdgpu 0000:3d:00.0: amdgpu:  for process python3.12 pid 6071 thread python3.12 pid 6071)
[19277.979477] amdgpu 0000:3d:00.0: amdgpu:   in page starting at address 0x0000709843600000 from IH client 0x12 (VMC)
[19277.979646] amdgpu 0000:3d:00.0: amdgpu:   cookie node_id 0 fault from die AID0
[19277.979890] amdgpu 0000:3d:00.0: amdgpu: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[19277.980057] amdgpu 0000:3d:00.0: amdgpu:      Faulty UTCL2 client ID: unknown (0x0)
[19277.980222] amdgpu 0000:3d:00.0: amdgpu:      MORE_FAULTS: 0x0
[19277.980384] amdgpu 0000:3d:00.0: amdgpu:      WALKER_ERROR: 0x0
[19277.980541] amdgpu 0000:3d:00.0: amdgpu:      PERMISSION_FAULTS: 0x0
[19277.980700] amdgpu 0000:3d:00.0: amdgpu:      MAPPING_ERROR: 0x0
[19277.980922] amdgpu 0000:3d:00.0: amdgpu:      RW: 0x0
[19277.981083] amdgpu 0000:3d:00.0: amdgpu: [mmhub0] retry page fault (src_id:0 ring:0 vmid:3 pasid:32790)
[19277.981250] amdgpu 0000:3d:00.0: amdgpu:  for process python3.12 pid 6071 thread python3.12 pid 6071)
[19277.981413] amdgpu 0000:3d:00.0: amdgpu:   in page starting at address 0x0000709843601000 from IH client 0x12 (VMC)
[19277.981581] amdgpu 0000:3d:00.0: amdgpu:   cookie node_id 0 fault from die AID0
[19277.981774] amdgpu 0000:3d:00.0: amdgpu: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[19277.981942] amdgpu 0000:3d:00.0: amdgpu:      Faulty UTCL2 client ID: unknown (0x0)
[19277.982108] amdgpu 0000:3d:00.0: amdgpu:      MORE_FAULTS: 0x0
[19277.982269] amdgpu 0000:3d:00.0: amdgpu:      WALKER_ERROR: 0x0
[19277.982428] amdgpu 0000:3d:00.0: amdgpu:      PERMISSION_FAULTS: 0x0
[19277.982585] amdgpu 0000:3d:00.0: amdgpu:      MAPPING_ERROR: 0x0
[19277.982785] amdgpu 0000:3d:00.0: amdgpu:      RW: 0x0
[19277.982945] amdgpu 0000:3d:00.0: amdgpu: [mmhub0] retry page fault (src_id:0 ring:0 vmid:3 pasid:32790)
[19277.983119] amdgpu 0000:3d:00.0: amdgpu:  for process python3.12 pid 6071 thread python3.12 pid 6071)
[19277.983282] amdgpu 0000:3d:00.0: amdgpu:   in page starting at address 0x0000709843602000 from IH client 0x12 (VMC)
[19277.983449] amdgpu 0000:3d:00.0: amdgpu:   cookie node_id 0 fault from die AID0
[19277.983621] amdgpu 0000:3d:00.0: amdgpu: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[19277.983823] amdgpu 0000:3d:00.0: amdgpu:      Faulty UTCL2 client ID: unknown (0x0)
[19277.983989] amdgpu 0000:3d:00.0: amdgpu:      MORE_FAULTS: 0x0
[19277.984151] amdgpu 0000:3d:00.0: amdgpu:      WALKER_ERROR: 0x0
[19277.984309] amdgpu 0000:3d:00.0: amdgpu:      PERMISSION_FAULTS: 0x0
[19277.984466] amdgpu 0000:3d:00.0: amdgpu:      MAPPING_ERROR: 0x0
[19277.984621] amdgpu 0000:3d:00.0: amdgpu:      RW: 0x0
[19277.984810] amdgpu 0000:3d:00.0: amdgpu: [mmhub0] retry page fault (src_id:0 ring:0 vmid:3 pasid:32790)
[19277.984984] amdgpu 0000:3d:00.0: amdgpu:  for process python3.12 pid 6071 thread python3.12 pid 6071)
[19277.985148] amdgpu 0000:3d:00.0: amdgpu:   in page starting at address 0x0000709843603000 from IH client 0x12 (VMC)
[19277.985316] amdgpu 0000:3d:00.0: amdgpu:   cookie node_id 0 fault from die AID0
[19277.985487] amdgpu 0000:3d:00.0: amdgpu: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[19277.985652] amdgpu 0000:3d:00.0: amdgpu:      Faulty UTCL2 client ID: unknown (0x0)
[19277.985866] amdgpu 0000:3d:00.0: amdgpu:      MORE_FAULTS: 0x0
[19277.986029] amdgpu 0000:3d:00.0: amdgpu:      WALKER_ERROR: 0x0
[19277.986187] amdgpu 0000:3d:00.0: amdgpu:      PERMISSION_FAULTS: 0x0
[19277.986343] amdgpu 0000:3d:00.0: amdgpu:      MAPPING_ERROR: 0x0
[19277.986499] amdgpu 0000:3d:00.0: amdgpu:      RW: 0x0
[19278.003288] amdgpu 0000:3d:00.0: amdgpu: [mmhub0] retry page fault (src_id:0 ring:0 vmid:3 pasid:32790)
[19278.003623] amdgpu 0000:3d:00.0: amdgpu:  for process python3.12 pid 6071 thread python3.12 pid 6071)
[19278.003924] amdgpu 0000:3d:00.0: amdgpu:   in page starting at address 0x0000709843604000 from IH client 0x12 (VMC)
[19278.004213] amdgpu 0000:3d:00.0: amdgpu:   cookie node_id 0 fault from die AID0
[19278.004501] amdgpu 0000:3d:00.0: amdgpu: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[19278.004919] amdgpu 0000:3d:00.0: amdgpu:      Faulty UTCL2 client ID: unknown (0x0)
[19278.005274] amdgpu 0000:3d:00.0: amdgpu:      MORE_FAULTS: 0x0
[19278.005533] amdgpu 0000:3d:00.0: amdgpu:      WALKER_ERROR: 0x0
[19278.005955] amdgpu 0000:3d:00.0: amdgpu:      PERMISSION_FAULTS: 0x0
[19278.006294] amdgpu 0000:3d:00.0: amdgpu:      MAPPING_ERROR: 0x0
[19278.006555] amdgpu 0000:3d:00.0: amdgpu:      RW: 0x0

This is the more standard mult-GPU hang:

[500967.571516] amdgpu 0000:dd:00.0: {1}uncorrectable hardware error(ERREVENT_ATHUB_INTERRUPT) detected!
[500967.587601] amdgpu 0000:4e:00.0: {1}[Hardware Error]: Accelerator Check Architecture events logged
[500967.587615] amdgpu 0000:4e:00.0: {1}[Hardware Error]: aca entry[00].STATUS=0xfaa00000000d080b
[500967.587623] amdgpu 0000:4e:00.0: {1}[Hardware Error]: aca entry[00].ADDR=0x0000000000000000
[500967.587628] amdgpu 0000:4e:00.0: {1}[Hardware Error]: aca entry[00].MISC0=0xd008000600000000
[500967.587633] amdgpu 0000:4e:00.0: {1}[Hardware Error]: aca entry[00].IPID=0x0002702e1f000300
[500967.587637] amdgpu 0000:4e:00.0: {1}[Hardware Error]: aca entry[00].SYND=0x000000005d000000
[500967.587920] amdgpu 0000:4e:00.0: {1}[Hardware Error]: Accelerator Check Architecture events logged
[500967.587924] amdgpu 0000:4e:00.0: {1}[Hardware Error]: aca entry[01].STATUS=0xbaa000000000080b
[500967.587928] amdgpu 0000:4e:00.0: {1}[Hardware Error]: aca entry[01].ADDR=0x0000000000000000
[500967.587932] amdgpu 0000:4e:00.0: {1}[Hardware Error]: aca entry[01].MISC0=0xd008000100000000
[500967.587936] amdgpu 0000:4e:00.0: {1}[Hardware Error]: aca entry[01].IPID=0x0001b02e1f007900
[500967.587940] amdgpu 0000:4e:00.0: {1}[Hardware Error]: aca entry[01].SYND=0x000000005d00241b
[500967.607060] amdgpu 0000:1b:00.0: amdgpu: Couldn't get PPT limit
[500967.610557] amdgpu 0000:3d:00.0: amdgpu: Couldn't get PPT limit
[500967.613844] amdgpu 0000:4e:00.0: amdgpu: Couldn't get PPT limit
[500967.617094] amdgpu 0000:5f:00.0: amdgpu: Couldn't get PPT limit
[500967.620317] amdgpu 0000:9d:00.0: amdgpu: Couldn't get PPT limit
[500967.623543] amdgpu 0000:bd:00.0: amdgpu: Couldn't get PPT limit
[500967.626774] amdgpu 0000:cd:00.0: amdgpu: Couldn't get PPT limit
[500967.627733] amdgpu 0000:dd:00.0: amdgpu: GPU reset begin!
[500967.628695] Pid 4141112(accelerate) over core_pipe_limit
[500967.628699] Skipping core dump
[500967.651415] amdgpu 0000:dd:00.0: amdgpu: Dumping IP State
[500967.651419] amdgpu 0000:dd:00.0: amdgpu: Dumping IP State Completed
[500967.651971] amdgpu 0000:bd:00.0: amdgpu: MODE1 reset
[500967.651972] amdgpu 0000:3d:00.0: amdgpu: MODE1 reset
[500967.651971] amdgpu 0000:dd:00.0: amdgpu: MODE1 reset
[500967.651973] amdgpu 0000:5f:00.0: amdgpu: MODE1 reset
[500967.651976] amdgpu 0000:9d:00.0: amdgpu: MODE1 reset
[500967.651977] amdgpu 0000:3d:00.0: amdgpu: GPU mode1 reset
[500967.651977] amdgpu 0000:4e:00.0: amdgpu: MODE1 reset
[500967.651979] amdgpu 0000:5f:00.0: amdgpu: GPU mode1 reset
[500967.651979] amdgpu 0000:bd:00.0: amdgpu: GPU mode1 reset
[500967.651977] amdgpu 0000:1b:00.0: amdgpu: MODE1 reset
[500967.651979] amdgpu 0000:cd:00.0: amdgpu: MODE1 reset
[500967.651983] amdgpu 0000:9d:00.0: amdgpu: GPU mode1 reset
[500967.651986] amdgpu 0000:dd:00.0: amdgpu: GPU mode1 reset
[500967.651990] amdgpu 0000:cd:00.0: amdgpu: GPU mode1 reset
[500967.651990] amdgpu 0000:1b:00.0: amdgpu: GPU mode1 reset
[500967.651993] amdgpu 0000:4e:00.0: amdgpu: GPU mode1 reset
[500967.654206] amdgpu 0000:3d:00.0: amdgpu: GPU smu mode1 reset
[500967.654593] amdgpu 0000:1b:00.0: amdgpu: GPU smu mode1 reset
[500967.654774] amdgpu 0000:bd:00.0: amdgpu: GPU smu mode1 reset
[500967.654797] amdgpu 0000:9d:00.0: amdgpu: GPU smu mode1 reset
[500967.654840] amdgpu 0000:dd:00.0: amdgpu: GPU smu mode1 reset
[500967.654862] amdgpu 0000:cd:00.0: amdgpu: GPU smu mode1 reset
[500967.656096] amdgpu 0000:5f:00.0: amdgpu: GPU smu mode1 reset
[500967.656170] amdgpu 0000:4e:00.0: amdgpu: GPU smu mode1 reset
[500972.423331] amdgpu 0000:dd:00.0: amdgpu: GPU reset succeeded, trying to resume
[500972.424642] [drm] PCIE GART of 512M enabled.
[500972.424647] [drm] PTB located at 0x000002EFED700000
[500972.424765] [drm] VRAM is lost due to GPU reset!
[500972.424767] amdgpu 0000:dd:00.0: amdgpu: PSP is resuming...
[500973.142440] amdgpu 0000:dd:00.0: amdgpu: RAP: optional rap ta ucode is not available
[500973.142448] amdgpu 0000:dd:00.0: amdgpu: Requesting 1 partitions through PSP
[500973.150787] amdgpu 0000:dd:00.0: amdgpu: SMU is resuming...
[500973.150793] amdgpu 0000:dd:00.0: amdgpu: smu driver if version = 0x08042024, smu fw if version = 0x08042027, smu fw program = 0, smu fw version = 0x00556b01 (85.107.1)
[500973.150796] amdgpu 0000:dd:00.0: amdgpu: SMU driver if version not matched
[500973.160234] amdgpu 0000:dd:00.0: amdgpu: SMU is resumed successfully!

Note that in this case, the GPU reports resetting correctly but leaves memory resident on some of the GPUs (usually 1-3, but I’ve seen as many as 6):

Screenshot from 2024-11-17 12-28-55

Running sudo amd-smi reset -g 0 -G doesn’t help, only a complete system reboot will recover the occupied memory.

I actually recently cleared a bunch of logs it looks like, but I noticed that while I get a variety of GPU hang reasons, the [gfxhub0] retry page fault / cookie node_id 1 fault from die / VM_L2_PROTECTION_FAULT_STATUS errors were pretty common.

The Liger kernels which might make it easier to reproduce were the same ones here: https://embeddedllm.com/blog/cuda-to-rocm-portability-case-study-liger-kernel and installed the same way:

python -m pip install "git+https://github.com/linkedin/Liger-Kernel.git#egg=liger-kernel[transformers]"

Note, they did a fair amount of benchmarking/tests presumably without problems on an 8xMI300X node, however only with 7B class models using the Alpaca dataset (so short runs). The earliest I’ve seen a GPU hang is ~3h in. I’m testing exclusively w/ the standard trl/axolotl kernels, not liger since I’m actually trying to finish runs… I’ll keep posting additional crash/logs here if they are different.

For my case, after not seeing any faults for a week or so… I got the fault again today. After several reboots I realized it was now easily repeatable and occurred within a few mins every time. I then looked at the code I had changed earlier, found a bug, fixed it, and now I haven’t seen the fault for hours.

As for @lhl’s case, @jamesxu2 I noticed you were using different versions of PyTorch compared to what @lhl mentioned above – I think that may be a crucial detail in replicating his case, but in general to most quickly replicate you will probably need to ensure every SW component is either the same version if @lhl had downloaded it, or built in the same way if he built it from source.

And AMD, can you please do something about the withheld documentation for these ACA codes so that folks might not have to go through this in future?

Thanks, and good luck everyone.

I’ve encountered this on a MI300X node while running rocPRIM benchmarks in the past weeks. Unfortunately I don’t have any logs, this instance didn’t allow me to access dmesg.

@lhl @anthonix We have an internal decoder for those ACA events but it’s unfortunately not something I can provide information on. It is definitely helpful that you’ve provided them however, so we can make comparisons to other error reports. The failure logs being shared in this thread point to a pretty diverse set of failures (e.g. The initial hang + GPU reset failure are distinct issues, but correlated), so this is quite challenging to debug, but I am trying a few things that you all have suggested in this thread.

Also if you don’t mind, I may make some edits to the formatting of your comments so they don’t appear in a single line.

Yeah, axolotl is a bit of a bear, although the only real conflict is axolotl installing the wrong version of bitsandbytes. I then manually reinstall the multi-backend version:

pip install --force-reinstall 'https://github.com/bitsandbytes-foundation/bitsandbytes/releases/download/continuous-release_multi-backend-refactor/bitsandbytes-0.44.1.dev0-py3-none-manylinux_2_24_x86_64.whl'

I have vLLM method that much more reliably causes GPU hangs that I’ll post when the node frees up but I’m not sure if it’s actually the same bug. I can try to queue up a throwaway overnight torchtune run just to see if I can get that to replicate in a couple days.