rocprofiler: Memory access fault by GPU node-i
HIP application crashes with error:
Memory access fault by GPU node-3 (Agent handle: 0x3c7610) on address 0x155555368000. Reason: Page not present or supervisor privilege.
whenever events from different devices are monitored at the same time. The problem can be reproduced using this simple rocprofiler program rocprofiler-test.tar.gz
This problem was produced with rocm-4.5.0 and rocm-5.0.0 on MI100 GPUs.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 18 (5 by maintainers)
Commits related to this issue
- issue-80: add reproducer Original issue link: https://github.com/ROCm-Developer-Tools/rocprofiler/issues/80 — committed to gcongiu/rocm-issues by gcongiu 3 years ago
My tests did not yet include launching kernels from multiple threads into the same queue. I could extend my own mini test to do this for sure and run it on our ROCm 5.7.1 RC setup.
@vlaindic thank you for the update. If it is of any help I could not see the failure at the end when using ROCm 5.2.0. Maybe the problem was introduced in later versions of the toolkit?
@gcongiu @bertwesarg A brief update. After changing the reproducer to use single thread to launch kernels sequentially, creating a single queue per device, and moving the initialization of the
rocprofiler_properties_tinside this loop, everything seems to work fine on the system with multiple MI210s. However, when using more than one thread (created either directly or via OpenMP) to launch kernels concurrently, the program may still crash at the very end of the process. @bgopesh and I are still trying to figure out what causes the issue in the multi-threaded environment.