mfem: HYPRE+HIP Runtime Error

I’m trying to compile with HIP & HYPRE. Compiling with just HIP works fine, but I’d like to use HypreBoomerAMG. I get the following runtime error when running ex1p. Any help would be much appreciated.

Options used:
   --mesh ../data/star.mesh
   --order 1
   --no-static-condensation
   --no-partial-assembly
   --device cpu
   --visualization
Device configuration: cpu
Memory configuration: host-std
Number of finite element unknowns: 82561
Memory access fault by GPU node-1 (Agent handle: 0x24fa560) on address 0x7f3f67a0e000. Reason: Page not present or supervisor privilege.
[jezebel:1158855] *** Process received signal ***
[jezebel:1158855] Signal: Aborted (6)
[jezebel:1158855] Signal code:  (-6)
[jezebel:1158855] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x143c0)[0x7f4103a4b3c0]
[jezebel:1158855] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f40e7a7703b]
[jezebel:1158855] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f40e7a56859]
[jezebel:1158855] [ 3] /opt/rocm-5.0.0/hip/lib/../../lib/libhsa-runtime64.so.1(+0x72847)[0x7f40e75d1847]
[jezebel:1158855] [ 4] /opt/rocm-5.0.0/hip/lib/../../lib/libhsa-runtime64.so.1(+0x7545b)[0x7f40e75d445b]
[jezebel:1158855] [ 5] /opt/rocm-5.0.0/hip/lib/../../lib/libhsa-runtime64.so.1(+0x18757)[0x7f40e7577757]
[jezebel:1158855] [ 6] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609)[0x7f4103a3f609]
[jezebel:1158855] [ 7] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f40e7b53163]
[jezebel:1158855] *** End of error message ***
Aborted (core dumped)

I’m using the master branch of MFEM & HYPRE v2.24.0.

HYRPE config

./configure \
  --with-hip \
  --disable-fortran \
  CC=/home/wcdawn/bin/openmpi-4.1.2/bin/mpicc \
  CXX=/home/wcdawn/bin/openmpi-4.1.2/bin/mpicxx \
  --with-MPI-include='/home/wcdawn/bin/openmpi-4.1.2/include' \
  --with-MPI-libs='mpi mpi_cxx' \
  --with-MPI-lib-dirs='/home/wcdawn/bin/openmpi-4.1.2/lib'

MFEM config

make config \
  MFEM_USE_HIP=YES HIP_ARCH=gfx1030 \
  MFEM_USE_MPI=YES \
  MPICC=/home/wcdawn/bin/openmpi-4.1.2/bin/mpicc \
  MPICXX=/home/wcdawn/bin/openmpi-4.1.2/bin/mpicxx \
  HYPRE_DIR=@MFEM_DIR@/../hypre-2.24.0/src/hypre \
  METIS_DIR=@MFEM_DIR@/../metis-5.1.0 MFEM_USE_METIS_5=YES

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 21 (17 by maintainers)

Most upvoted comments

Hi folks. There is certainly an issue with HYPRE at present that I will try to address when I can. The issue is that the Navi gaming cards (gfx1030 indicates an RDNA2 card, so something like a 6900XT) run with warp/wavefront sizes of 32. Currently, HYPRE on AMD GPUs is setup for warp/wavefront size of 64. I’ll post a note here when we update HYPRE to support wavefront size 32 on AMD GPUs.