ROCm: [Issue]: Crashes / instability on Ubuntu w/ desktop environment
Problem Description
My rocm installation suffers severe instability that results in graphical artifacts, program hangs, and program / X session crashes.
These issues are intermittent; running the same program can work fine but consistently fail after a system reboot. I can reproduce it with both large ML applications (e.g., llama2 with SHARK) and with toy programs like rocm-examples, which makes me think it’s related to the rocm installation instead of the exact application executed.
The iGPU is disabled in the BIOS. I installed rocm using these instructions: https://rocm.docs.amd.com/en/docs-5.7.1/deploy/linux/quick_start.html.
I would like to understand what is the root cause and if this can be fixed or worked around, or confirm if running rocm in a desktop environment is not supported.
Happy to provide any additional details that can help.
Operating System
Ubuntu 23.04
CPU
AMD Ryzen 9 7950X
GPU
AMD Radeon RX 7900 XTX
ROCm Version
ROCm 5.7.1
ROCm Component
No response
Steps to Reproduce
No response
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
ROCk module is loaded
=====================
HSA System Attributes
=====================
Runtime Version: 1.1
System Timestamp Freq.: 1000.000000MHz
Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model: LARGE
System Endianness: LITTLE
Mwaitx: DISABLED
DMAbuf Support: YES
==========
HSA Agents
==========
*******
Agent 1
*******
Name: AMD Ryzen 9 7950X 16-Core Processor
Uuid: CPU-XX
Marketing Name: AMD Ryzen 9 7950X 16-Core Processor
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 0
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 4500
BDFID: 0
Internal Node ID: 0
Compute Unit: 32
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 65550296(0x3e837d8) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 65550296(0x3e837d8) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 65550296(0x3e837d8) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:
*******
Agent 2
*******
Name: gfx1100
Uuid: GPU-4e061d420fdb0000
Marketing Name: Radeon RX 7900 XTX
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 64(0x40)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 1
Device Type: GPU
Cache Info:
L1: 32(0x20) KB
L2: 6144(0x1800) KB
L3: 98304(0x18000) KB
Chip ID: 29772(0x744c)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 2431
BDFID: 768
Internal Node ID: 1
Compute Unit: 96
SIMDs per CU: 2
Shader Engines: 6
Shader Arrs. per Eng.: 2
WatchPts on Addr. Ranges:4
Features: KERNEL_DISPATCH
Fast F16 Operation: TRUE
Wavefront Size: 32(0x20)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 32(0x20)
Max Work-item Per CU: 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
Max fbarriers/Workgrp: 32
Packet Processor uCode:: 528
SDMA engine uCode:: 19
IOMMU Support:: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 25149440(0x17fc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GLOBAL; FLAGS:
Size: 25149440(0x17fc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 3
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx1100
Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
FBarrier Max Size: 32
*** Done ***
Additional Information
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 18 (2 by maintainers)
The issue has been fixed in the GPU driver. The fix should be available in upcoming release of ROCm 6.1.
@ppanchad-amd thanks for the information! I asked because one of the logs indicated bad DMA transactions (blocked by IOMMU), which looked like a potential security vulnerability.
@giorgi1324 as one data point, I also did a fresh install last week (22.04 + rocm 6.1) and so far it seems to be working fine for me for the first time ever. I did not install the dkms driver, only the rocm apt package
I am experiencing the same issue on a fresh install of Ubuntu 22.04.4, ROCm 6.1
Operating System
Ubuntu 22.04.4
CPU
Intel i5-13600k
GPU
AMD Radeon RX 7900 XT
ROCm Version
Steps to Reproduce
Run any tensorflow load. In my case I’m running a local jupyter notebook and training a very simple model, cell 19 in this sample: https://github.com/ageron/handson-ml3/blob/main/11_training_deep_neural_networks.ipynb
Observe jankiness, overall system instability and visual artifacts. Most of the time Gnome crashes and I am logged out. Photo of the visual artifacts:
Dmesg of the machine during the failure: https://drive.google.com/file/d/1hdnEkcrR4Ru0Tbq3YaysqI17ZY9b8plx/view?usp=drive_link Essentially the kernel log is flooded with following two lines"
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support:
I have seen similar things - https://github.com/ROCm/ROCm/issues/2689