ROCm: [Driver] *ERROR* MES failed to response msg=2
Triggered by running https://github.com/RadeonOpenCompute/rocm_bandwidth_test in a loop while running https://github.com/ROCm-Developer-Tools/HIP-Examples/tree/master/gpu-burn in a loop.
1x 7900XTX ASROCK ROMED8-2T EPYC 7662 Ubuntu 22.04, Kernel 6.2.14-060214-generic, ROCm 5.5
sudo cat /sys/kernel/debug/dri/1/amdgpu_gpu_recover will recover the GPU.
[ 111.406216] amdgpu 0000:83:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:10 pasid:32769, for process rocm-bandwidth- pid 3286 thread rocm-bandwidth- pid 3286)
[ 111.406237] amdgpu 0000:83:00.0: amdgpu: in page starting at address 0x00007f0000000000 from client 10
[ 111.406246] amdgpu 0000:83:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00A01A30
[ 111.406253] amdgpu 0000:83:00.0: amdgpu: Faulty UTCL2 client ID: SDMA0 (0xd)
[ 111.406259] amdgpu 0000:83:00.0: amdgpu: MORE_FAULTS: 0x0
[ 111.406265] amdgpu 0000:83:00.0: amdgpu: WALKER_ERROR: 0x0
[ 111.406270] amdgpu 0000:83:00.0: amdgpu: PERMISSION_FAULTS: 0x3
[ 111.406275] amdgpu 0000:83:00.0: amdgpu: MAPPING_ERROR: 0x0
[ 111.406280] amdgpu 0000:83:00.0: amdgpu: RW: 0x0
[ 114.188710] amdgpu 0000:83:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:10 pasid:32769, for process rocm-bandwidth- pid 3286 thread rocm-bandwidth- pid 3286)
[ 114.188729] amdgpu 0000:83:00.0: amdgpu: in page starting at address 0x00007f0000000000 from client 10
[ 114.188738] amdgpu 0000:83:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00A01A30
[ 114.188746] amdgpu 0000:83:00.0: amdgpu: Faulty UTCL2 client ID: SDMA0 (0xd)
[ 114.188754] amdgpu 0000:83:00.0: amdgpu: MORE_FAULTS: 0x0
[ 114.188759] amdgpu 0000:83:00.0: amdgpu: WALKER_ERROR: 0x0
[ 114.188765] amdgpu 0000:83:00.0: amdgpu: PERMISSION_FAULTS: 0x3
[ 114.188770] amdgpu 0000:83:00.0: amdgpu: MAPPING_ERROR: 0x0
[ 114.188776] amdgpu 0000:83:00.0: amdgpu: RW: 0x0
[ 114.302856] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=2
[ 114.303173] amdgpu: failed to add hardware queue to MES, doorbell=0x1202
[ 114.303176] amdgpu: MES might be in unrecoverable state, issue a GPU reset
[ 114.303179] amdgpu: Failed to restore queue 0
[ 114.303182] amdgpu: Failed to restore process queues
[ 114.303184] amdgpu: Failed to restore queues of pasid 0x8001
[ 114.303450] amdgpu 0000:83:00.0: amdgpu: GPU reset begin!
[ 114.303477] amdgpu: Failed to evict queue 1
[ 114.303483] amdgpu: Failed to suspend process 0x8002
[ 114.309700] amdgpu 0000:83:00.0: amdgpu: recover vram bo from shadow start
[ 114.309705] amdgpu 0000:83:00.0: amdgpu: recover vram bo from shadow done
[ 114.412749] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=14
[ 114.413073] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR* failed to reg_write_reg_wait
[ 114.420094] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=14
[ 114.420379] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR* failed to reg_write_reg_wait
[ 114.523222] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=14
[ 114.523499] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR* failed to reg_write_reg_wait
[ 114.530476] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=14
[ 114.530750] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR* failed to reg_write_reg_wait
[ 114.634108] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=14
[ 114.634391] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR* failed to reg_write_reg_wait
...
[ 117.293167] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=2
[ 117.293439] [drm:amdgpu_mes_add_hw_queue [amdgpu]] *ERROR* failed to add hardware queue to MES, doorbell=0x2200
[ 117.293712] [drm:amdgpu_mes_self_test [amdgpu]] *ERROR* failed to add ring
[ 117.294161] amdgpu 0000:83:00.0: amdgpu: GPU reset(1) succeeded!
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 26
- Comments: 41
I can reproduce this on my 7900 XTX as well
I am still seeing this on both ROCm 6.0 and the 6.0.1 packages at https://repo.radeon.com/rocm/apt/6.0.1 / https://repo.radeon.com/amdgpu/apt/6.0.1 on fresh Ubuntu 22.04.
Linux amdsux 6.5.0-15-generic #15~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 12 18:54:30 UTC 2 x86_64 x86_64 x86_64 GNU/LinuxI realize it does no good for everyone else experiencing this issue to add yet another “yeah, same here” and some venting, so the rest of you may stop reading now. But AMD, you really need to know this is not acceptable.
It’s been over a year since the card was released and still the software isn’t even stable, let alone all the trouble it takes to get set up vs nVidia. All whilst AMD releases these frankly patronizing announcements about how great and open and “ready” ROCm 6.0 is, “driving an inflection with developers”. Erm, yeah… it’s making us run away.
Please understand developer sentiment towards AMD is very poor and getting worse, not better. Everyone else like me who is trying to do research but they bought an AMD card because they couldn’t afford a nVidia card, is never ever going to buy an AMD GPU again. And we will tell our employers to never ever buy AMD too, and they will listen, because developer time is far more expensive than nVidia’s hardware. We should have listened to our friends who told us to just buy nVidia cards, but we won’t make this mistake again. This will last years and years. The best you can do now is admit your failures and try to salvage your reputation for CPUs. Because it doesn’t matter how good or cheap your GPUs get, we will never risk wasting our time again.
For all this talk about openness, we’re all here seeing this issue and does anyone outside AMD even understand what this message means? For that matter, does anyone INSIDE AMD understand what it means? Please explain it to us.
So far as I can tell, “MES” is the MicroEngine Scheduler, and https://docs.kernel.org/gpu/amdgpu/driver-core.html#graphics-and-compute-microcontrollers says it is “unused”. Mumblings on the internet suggest it is used now, and well, we’re in this GitHub issue together aren’t we? Beyond that, things get extremely hazy. The kernel code that actually produces that log message is completely opaque. None of these functions are documented. MES is… a separate microcontroller controlled by a firmware blob? Do I have that right? And what exactly does this mean? https://github.com/torvalds/linux/blob/41bccc98fb7931d63d03f326a746ac4d429c1dd3/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c#L563
Is this AMD’s “open software approach”? Hanging binary blobs and sending tarballs to geohot that he’s not allowed to distribute?!
PS: I’m still not over the terrible software support for the RX480 last time. I hoped things had changed. Fool me twice, shame on me. But never again. I’m losing my mind over this because I’ve lost months of productivity… I can’t stress enough, my experiment seems to be working whilst I can get it to run. But the reason I cannot publish is because I chose the wrong GPU. I’m going to have to go get a “real job” instead.
This MES crash tends to segfault other unrelated programs as well.
This one in particular was interesting. Entire system locked up shortly after this screenshot: (I use an Intel dGPU for video out.)
OS: Arch Linux 6.8.4 CPU: AMD Ryzen 7 5800X3D GPU0: AMD RX 7900 XTX Sapphire Pulse (No video output, runs only PyTorch) GPU1: Intel ARC A770 LE (Used for video output and genral use)
ROCm: 6.0.2 PyTorch: 2.3.0+rocm6.0
Journalctl in reverse order:
Nevermind everyone, sorry for the premature excitement. Two days later, there it goes again.
Of course, it had to wait for me to post this to crash…
Sadly it did not help me 😦 High VRAM workloads still cause the MES unrecoverable error, every time, which requires a reboot to solve.
So far not having any crashes with my distros latest, still having the cursor stutters bug perhaps unrelated and if so that’s just mildly annoying and can live with that. Don’t want to jinx it though so going to let to generate idle, if it crashes I guess I’ll look into updating my firmware. Again, I suspect my issue is unrelated but seems others in thread where having mine
Edit: got a crash, seems more stable at least though
You can just clone the repo,
git clone git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git --depth 1, thensudo cp linux-firmware/amdgpu/* /lib/firmware/updates/amdgpu/and reboot. Thanks @saffroy at https://gitlab.freedesktop.org/drm/amd/-/issues/3175 for pointing this out; note that this firmware is relatively untested, hot off the presses, and also your system will overwrite this when it updates anyway, I take no responsibility for damages etc. But also I’ve been running it for two days and my PC hasn’t caught fire yet.If you’ve installed it correctly, you should note a numeric change in
grep MES /sys/kernel/debug/dri/{0,1}/amdgpu_firmware_info. Mine is at MES0x0000004b, MES_KIQ0x00000075.edit: But it crashes anyway.
Using an RX 7900 XTX with no monitor attached, training a diffusion model or just running inference. (Getting monitor output from another GPU.)
Still seeing the same MES hangs with Arch Linux 6.8.1-arch1-1 / ROCm 6.0.2 / PyTorch 2.4.0.dev20240323+rocm6.0 This MES hang causes other unrelated Python programs to core dump too and GPU keeps pulling 120W~ of power when it hangs.
Only way to recover from the hang is running
rocm-smi --gpureset -id 0if you have no monitor attached to the AMD GPU. If you have a monitor attached, only way to recover is hard rebooting.Adding
amdgpu.mes=0 amdgpu.mes_kiq=1to the grub config doesn’t seem to be doing anything either.Log on PyTorch when MES hangs:
Journalctl in reverse order:
<\details>
and just checking - you are on new kernels with the 6.0.2 binaries or your code compiled against the 6.0.2 libs? (yes - i have had to compile pytorch against 6.0.2).
mainly because i finally got there - and closed my ticket… #2689
Im not saying it wasnt easy or simple but I can hit 20-22Gb consistently somehow now… and Im over like 3 days of uptime too… which is a first since november last year…
and here is the worst ever way to show what ubuntu libs got me there…
and then made pytorch find the folder and run the special pytorch convert from cuda to hip and then build, and then build and install, and then build pytorchvision and audio after… so much faf…
But - I run now…
Any reason you want to run the older kernel?
A few people are experiencing better results on the newer kernels, and the 6.0.2 drivers.
I got to a running setup (with some minor issues) with the 6.5.x stock one - ended up closing my ticket too. #2689
I’ve had this exact issue frequently with ROCm 5.7 on Radeon RX 7900 XTX.
Upgrading to ROCm 6.0 has solved it for me. At least Stable Diffusion with torch-2.3.0+rocm5.7 still works on ROCm 6.0, and without crashes now, no need to restart after a failed GPU reset.
The non-critical GCVM_L2_PROTECTION_FAULT_STATUS errors are still there, but they don’t force me to restart after MES failure and a GPU reset attempt.
Currently working:
Linux moon 6.2.0-39-generic #40~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 16 10:53:04 UTC 2 x86_64 x86_64 x86_64 GNU/LinuxI am seeing similar messages from kernel on boot with phoenix apu based system and without the ROCM Stack
1637.604718] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=14 [ 1637.605055] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR* failed to reg_write_reg_wait [ 1637.607290] [drm] PCIE GART of 512M enabled (table at 0x00000080FFD00000). [ 1637.607422] amdgpu 0000:c1:00.0: amdgpu: SMU is resuming... [ 1637.610838] amdgpu 0000:c1:00.0: amdgpu: SMU is resumed successfully! [ 1637.803597] [drm] VCN decode and encode initialized successfully(under DPG Mode). [ 1637.803867] amdgpu 0000:c1:00.0: [drm:jpeg_v4_0_hw_init [amdgpu]] JPEG decode initialized successfully. [ 1637.804227] amdgpu 0000:c1:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0 [ 1637.804231] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0 [ 1637.804234] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0 [ 1637.804235] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 6 on hub 0 [ 1637.804237] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 7 on hub 0 [ 1637.804239] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 8 on hub 0 [ 1637.804241] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 9 on hub 0 [ 1637.804243] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 10 on hub 0 [ 1637.804245] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 11 on hub 0 [ 1637.804246] amdgpu 0000:c1:00.0: amdgpu: ring sdma0 uses VM inv eng 12 on hub 0 [ 1637.804248] amdgpu 0000:c1:00.0: amdgpu: ring vcn_unified_0 uses VM inv eng 0 on hub 8 [ 1637.804250] amdgpu 0000:c1:00.0: amdgpu: ring jpeg_dec uses VM inv eng 1 on hub 8 [ 1637.804252] amdgpu 0000:c1:00.0: amdgpu: ring mes_kiq_3.1.0 uses VM inv eng 13 on hub 0 [ 1637.809886] [drm] ring gfx_32803.1.1 was added [ 1637.810426] [drm] ring compute_32803.2.2 was added [ 1637.810934] [drm] ring sdma_32803.3.3 was added [ 1637.810962] [drm] ring gfx_32803.1.1 ib test pass [ 1637.810991] [drm] ring compute_32803.2.2 ib test pass [ 1637.811092] [drm] ring sdma_32803.3.3 ib test pass [ 1637.820482] PM: resume devices took 0.362 seconds [ 1637.820898] OOM killer enabled. [ 1637.820902] Restarting tasks ... done. [ 1637.822675] random: crng reseeded on system resumption [ 1637.829255] PM: suspend exit [ 1640.456439] wlp1s0: authenticate with 24:4b:fe:62:67:04 (local address=14:ac:60:46:7b:f9) [ 1640.479251] wlp1s0: send auth to 24:4b:fe:62:67:04 (try 1/3) [ 1640.499491] wlp1s0: authenticate with 24:4b:fe:62:67:04 (local address=14:ac:60:46:7b:f9) [ 1640.508904] wlp1s0: send auth to 24:4b:fe:62:67:04 (try 1/3) [ 1640.512391] wlp1s0: authenticated [ 1640.515427] wlp1s0: associate with 24:4b:fe:62:67:04 (try 1/3) [ 1640.526627] wlp1s0: RX AssocResp from 24:4b:fe:62:67:04 (capab=0x1511 status=0 aid=1) [ 1640.558571] wlp1s0: associated [ 1640.613407] wlp1s0: Limiting TX power to 27 (30 - 3) dBm as advertised by 24:4b:fe:62:67:04 [ 1642.696635] ucsi_acpi USBC000:00: ucsi_handle_connector_change: ACK failed (-110) [ 1648.577999] ucsi_acpi USBC000:00: ucsi_handle_connector_change: ACK failed (-110) [ 2325.112034] i2c_designware AMDI0010:00: i2c_dw_handle_tx_abort: lost arbitration [ 2531.128560] wlp1s0: deauthenticating from 24:4b:fe:62:67:04 by local choice (Reason: 3=DEAUTH_LEAVING) [ 2531.678155] PM: suspend entry (s2idle) [ 2531.701397] Filesystems sync: 0.023 seconds [ 2531.710174] Freezing user space processes [ 2531.712040] Freezing user space processes completed (elapsed 0.001 seconds) [ 2531.712044] OOM killer disabled. [ 2531.712045] Freezing remaining freezable tasks [ 2531.713066] Freezing remaining freezable tasks completed (elapsed 0.001 seconds) [ 2531.713070] printk: Suspending console(s) (use no_console_suspend to debug) [ 2532.645147] PM: suspend devices took 0.932 seconds [ 2532.647493] ACPI: EC: interrupt blocked [ 9429.864695] ACPI: EC: interrupt unblocked [ 9430.036343] nvme nvme0: 16/0/0 default/read/poll queues [ 9430.088770] atkbd serio0: Unknown key pressed (translated set 2, code 0x6b on isa0060/serio0). [ 9430.088779] atkbd serio0: Use 'setkeycodes 6b <keycode>' to make it known. [ 9430.089328] atkbd serio0: Unknown key released (translated set 2, code 0x6b on isa0060/serio0). [ 9430.089336] atkbd serio0: Use 'setkeycodes 6b <keycode>' to make it known. [ 9430.175277] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=14 [ 9430.175614] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR* failed to reg_write_reg_wait [ 9430.177835] [drm] PCIE GART of 512M enabled (table at 0x00000080FFD00000). [ 9430.177942] amdgpu 0000:c1:00.0: amdgpu: SMU is resuming... [ 9430.181299] amdgpu 0000:c1:00.0: amdgpu: SMU is resumed successfully! [ 9430.286638] [drm] VCN decode and encode initialized successfully(under DPG Mode). [ 9430.286884] amdgpu 0000:c1:00.0: [drm:jpeg_v4_0_hw_init [amdgpu]] JPEG decode initialized successfully. [ 9430.287225] amdgpu 0000:c1:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0 [ 9430.287229] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0 [ 9430.287231] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0 [ 9430.287233] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 6 on hub 0 [ 9430.287234] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 7 on hub 0 [ 9430.287236] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 8 on hub 0 [ 9430.287237] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 9 on hub 0 [ 9430.287239] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 10 on hub 0 [ 9430.287241] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 11 on hub 0 [ 9430.287242] amdgpu 0000:c1:00.0: amdgpu: ring sdma0 uses VM inv eng 12 on hub 0 [ 9430.287244] amdgpu 0000:c1:00.0: amdgpu: ring vcn_unified_0 uses VM inv eng 0 on hub 8 [ 9430.287246] amdgpu 0000:c1:00.0: amdgpu: ring jpeg_dec uses VM inv eng 1 on hub 8 [ 9430.287247] amdgpu 0000:c1:00.0: amdgpu: ring mes_kiq_3.1.0 uses VM inv eng 13 on hub 0 [ 9430.292387] [drm] ring gfx_32803.1.1 was added [ 9430.292929] [drm] ring compute_32803.2.2 was added [ 9430.293440] [drm] ring sdma_32803.3.3 was added [ 9430.293469] [drm] ring gfx_32803.1.1 ib test pass [ 9430.293499] [drm] ring compute_32803.2.2 ib test pass [ 9430.293598] [drm] ring sdma_32803.3.3 ib test pass💻 Framework Laptop 13 (AMD Ryzen 7040Series) (Laptop) running BIOS 3.3 (03.03) released 10/17/2023 and EC unknown 🐧 Fedora Linux 40 (KDE Plasma Prerelease) 🐧 Kernel 6.7.0fw13ec-rc1+ ✅ AMD Ryzen 7 7840U w/ Radeon 780M Graphics (family 19 model 74) ✅ HSMP driver
amd_hsmpnot detected (blocked: False) ✅ PMC driveramd_pmcloaded (Program 0 Firmware 76.70.0) OpenGL core profile version string: 4.6 (Core Profile) Mesa 24.0.0-devel