rocprofiler: [HSA_STATUS_ERROR] A generic error has occurred

Hi,

I am trying to profile some CEED benchmarks. I’m using a gfx906 card with rocm 2.10.

I’m using hipcc and the compilation/run seem fine, but I can’t get any output from the profiler.

I tried different options, but I keep getting this generic error:

rcprof -C  ./bp3 -o 2 -l 8 -d hip
Options used:
   --mesh-dimension 3
   --refinement-level 8
   --order 2
   --device hip
Radeon Compute Profiler V5.6.7262 is enabled
No counter file specified. Only counters that will fit into a single pass will be enabled.
Device configuration: hip,cpu
Processor partitioning: 1 1 1
Mesh dimensions: 8 8 4
Total number of elements: 256
Number of finite element unknowns: 2601
aqlprofile API table load failed: HSA_STATUS_ERROR: A generic error has occurred.
[corona90:mpi_rank_0][error_sighandler] Caught error: Aborted (signal 6)
Failed to generate profile result /g/g91/camier1/Session1.csv.

Have you seen this kind of error?

Thank you for your help,

Jean-Sylvain

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 27

Commits related to this issue

Most upvoted comments

In case anyone else is bumping against this error aqlprofile API table load failed, installing hsa-amd-aqlprofile package seemed to solve it. This issue pops up when you search for aqlprofile API table load failed. I only realised you can install that package because calling https://github.com/ROCm-Developer-Tools/rocprofiler/blob/207458f251f223803dbbce64821dde15107a1781/test/util/hsa_rsrc_factory.cpp#L131 in debug spits out the library name isn’t found.

Mine was failing because ctrl was falling back onto trying to load HSA_EXTENSION_AMD_AQLPROFILE and then failing on that. This is where the HSA_STATUS_ERROR was coming from which is being invoked from https://github.com/ROCm-Developer-Tools/rocprofiler/blob/207458f251f223803dbbce64821dde15107a1781/test/util/hsa_rsrc_factory.cpp#L133

@camierjs:

  1. there was a bug in the library paths of roctracer for RHEL + ROCm 2.10, this can be fixed via setting export LD_LIBRARY_PATH=$(LD_LIBRARY_PATH):/opt/rocm/roctracer/lib in your environment. This may resolve your tool lib error.
  2. Regardless, I believe that the hsa-amd-aqlprofile.x86_64 package must be installed via sudo yum install hsa-amd-aqlprofile.x86_64, as this library isn’t open-sourced yet. On my system this lives in /opt/rocm/hsa-amd-aqlprofile/lib/libhsa-amd-aqlprofile64.so.1.0.0.

Otherwise, if you don’t have sudo access and need to build from source, perhaps try building the rocm-2.10.x branch of roctracer since you’re using ROCm 2.10. The master branch has already pulled in many changes for ROCm 3.0. The software stack is moving fast.