mfem: Cannot run ex1p in SuperLU example [MFEM-4.2]

Hello Everybody! I have upgraded MFEM-4.1 to MFEM-4.2. When I run the example 1p SuperLU like “make ex1p”, it returns:

mpicxx -O3 -std=c++11 -I../.. -I../../../hypre/src/hypre/include -I../../../SuperLU_DIST_6.3.1/include -I../../../parmetis-4.0.3/include ex1p.cpp -o ex1p -L../.. -lmfem -L../../../hypre/src/hypre/lib -lHYPRE -Wl,-rpath,../../../SuperLU_DIST_6.3.1/lib64 -L../../../SuperLU_DIST_6.3.1/lib64 -lsuperlu_dist -lblas -L../../../parmetis-4.0.3/lib -lparmetis -lmetis -lrt /usr/bin/ld: cannot find -lsuperlu_dist collect2: error: ld returned 1 exit status makefile:41: recipe for target 'ex1p' failed make: *** [ex1p] Error 1 I try seeing SuperLU_DIST_6.3.1 directory but I cannot see the library “lib64”. I just see the library “lib”. Then I comeback the directory of example SuperLU and try typing like: “mpicxx -O3 -std=c++11 -I…/… -I…/…/…/hypre/src/hypre/include -I…/…/…/SuperLU_DIST_6.3.1/include -I…/…/…/parmetis-4.0.3/include ex1p.cpp -o ex1p -L…/… -lmfem -L…/…/…/hypre/src/hypre/lib -lHYPRE -Wl,-rpath,…/…/…/SuperLU_DIST_6.3.1/lib -L…/…/…/SuperLU_DIST_6.3.1/lib -lsuperlu_dist -lblas -L…/…/…/parmetis-4.0.3/lib -lparmetis -lmetis -lrt” .but It returns: ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrf.c.o): In function pdgstrf’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrf.c:559: undefined reference to GOMP_parallel' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrf.c.o): In function pdgstrf’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dlook_ahead_update.c:124: undefined reference to GOMP_parallel' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrf.c.o): In function pdgstrf’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dSchCompUdt-2Ddynamic.c:243: undefined reference to GOMP_parallel' /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dSchCompUdt-2Ddynamic.c:303: undefined reference to GOMP_parallel’ /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dSchCompUdt-2Ddynamic.c:343: undefined reference to GOMP_parallel' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrf.c.o):/home/ubuntu/SuperLU_DIST_6.3.1/SRC/dSchCompUdt-2Ddynamic.c:405: more undefined references to GOMP_parallel’ follow …/…/…/SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrf.c.o): In function pdgstrf._omp_fn.0': /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrf.c:560: undefined reference to omp_get_thread_num’ /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrf.c:563: undefined reference to omp_get_num_threads' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrf.c.o): In function pdgstrf._omp_fn.1’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dlook_ahead_update.c:124: undefined reference to GOMP_loop_dynamic_start' /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dlook_ahead_update.c:144: undefined reference to omp_get_thread_num’ /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dlook_ahead_update.c:195: undefined reference to GOMP_loop_dynamic_next' /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dlook_ahead_update.c:136: undefined reference to GOMP_loop_end_nowait’ …/…/…/SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrf.c.o): In function pdgstrf._omp_fn.2': /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dSchCompUdt-2Ddynamic.c:243: undefined reference to GOMP_loop_guided_start’ /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dSchCompUdt-2Ddynamic.c:265: undefined reference to GOMP_loop_guided_next' /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dSchCompUdt-2Ddynamic.c:249: undefined reference to GOMP_loop_end_nowait’ …/…/…/SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrf.c.o): In function pdgstrf._omp_fn.3': /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dSchCompUdt-2Ddynamic.c:303: undefined reference to omp_get_num_threads’ /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dSchCompUdt-2Ddynamic.c:303: undefined reference to omp_get_thread_num' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrf.c.o): In function pdgstrf._omp_fn.4’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dSchCompUdt-2Ddynamic.c:343: undefined reference to GOMP_loop_guided_start' /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dSchCompUdt-2Ddynamic.c:361: undefined reference to GOMP_loop_guided_next’ /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dSchCompUdt-2Ddynamic.c:350: undefined reference to GOMP_loop_end_nowait' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrf.c.o): In function pdgstrf._omp_fn.5’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dSchCompUdt-2Ddynamic.c:407: undefined reference to omp_get_thread_num' /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dSchCompUdt-2Ddynamic.c:434: undefined reference to GOMP_loop_dynamic_start’ /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dSchCompUdt-2Ddynamic.c:497: undefined reference to GOMP_loop_dynamic_next' /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dSchCompUdt-2Ddynamic.c:450: undefined reference to GOMP_loop_end’ …/…/…/SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrf.c.o): In function pdgstrf._omp_fn.6': /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dSchCompUdt-2Ddynamic.c:600: undefined reference to omp_get_thread_num’ /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dSchCompUdt-2Ddynamic.c:627: undefined reference to GOMP_loop_dynamic_start' /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dSchCompUdt-2Ddynamic.c:664: undefined reference to GOMP_loop_dynamic_next’ /home/ubuntu/SuperLU_DIST_6.3.1/SRC/dSchCompUdt-2Ddynamic.c:643: undefined reference to GOMP_loop_end' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrf2.c.o): In function pdgstrs2_omp’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrf2.c:847: undefined reference to GOMP_parallel' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrf2.c.o): In function pdgstrs2_omp.omp_fn.0’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrf2.c:847: undefined reference to omp_get_num_threads' /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrf2.c:847: undefined reference to omp_get_thread_num’ /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrf2.c:871: undefined reference to GOMP_task' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrs.c.o): In function pdReDistribute_B_to_X’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:205: undefined reference to GOMP_parallel' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrs.c.o): In function pdReDistribute_X_to_B’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:460: undefined reference to GOMP_parallel' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrs.c.o): In function pdCompute_Diag_Inv’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:747: undefined reference to dtrtri_' /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:749: undefined reference to dtrtri’ …/…/…/SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrs.c.o): In function pdgstrs': /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:972: undefined reference to GOMP_parallel’ /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:1066: undefined reference to GOMP_parallel' /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:1085: undefined reference to GOMP_parallel’ /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:1242: undefined reference to GOMP_parallel' /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:1405: undefined reference to GOMP_parallel’ …/…/…/SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrs.c.o):/home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:1466: more undefined references to GOMP_parallel' follow ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrs.c.o): In function pdReDistribute_B_to_X._omp_fn.0’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:209: undefined reference to omp_get_thread_num' /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:214: undefined reference to GOMP_taskloop’ …/…/…/SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrs.c.o): In function pdReDistribute_X_to_B._omp_fn.2': /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:464: undefined reference to omp_get_thread_num’ /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:469: undefined reference to GOMP_taskloop' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrs.c.o): In function pdgstrs._omp_fn.4’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:974: undefined reference to omp_get_thread_num' /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:977: undefined reference to omp_get_thread_num’ /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:975: undefined reference to omp_get_num_threads' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrs.c.o): In function pdgstrs._omp_fn.7’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:1337: undefined reference to omp_get_num_threads' /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:1337: undefined reference to omp_get_thread_num’ /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:1250: undefined reference to omp_get_num_threads' /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:1250: undefined reference to omp_get_thread_num’ …/…/…/SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrs.c.o): In function pdgstrs._omp_fn.8': /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:1411: undefined reference to omp_get_thread_num’ /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:1416: undefined reference to GOMP_taskloop' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrs.c.o): In function pdgstrs._omp_fn.10’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:1470: undefined reference to omp_get_thread_num' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrs.c.o): In function pdgstrs._omp_fn.12’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:1947: undefined reference to omp_get_thread_num' /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:1951: undefined reference to GOMP_taskloop’ …/…/…/SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrs.c.o): In function pdgstrs._omp_fn.14': /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:2042: undefined reference to omp_get_thread_num’ /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:2046: undefined reference to GOMP_taskloop' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrs.c.o): In function pdgstrs._omp_fn.16’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs.c:2091: undefined reference to omp_get_thread_num' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrs_lsum.c.o): In function dlsum_fmod_inv’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs_lsum.c:526: undefined reference to GOMP_taskloop' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrs_lsum.c.o): In function dlsum_fmod_inv_master’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs_lsum.c:1072: undefined reference to GOMP_taskloop' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrs_lsum.c.o): In function dlsum_bmod_inv’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs_lsum.c:1437: undefined reference to GOMP_taskloop' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrs_lsum.c.o): In function dlsum_bmod_inv_master’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs_lsum.c:1909: undefined reference to omp_get_thread_num' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrs_lsum.c.o): In function dlsum_fmod_inv._omp_fn.0’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs_lsum.c:531: undefined reference to omp_get_thread_num' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrs_lsum.c.o): In function dlsum_fmod_inv_master._omp_fn.1’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs_lsum.c:1077: undefined reference to omp_get_thread_num' ../../../SuperLU_DIST_6.3.1/lib/libsuperlu_dist.a(pdgstrs_lsum.c.o): In function dlsum_bmod_inv._omp_fn.2’: /home/ubuntu/SuperLU_DIST_6.3.1/SRC/pdgstrs_lsum.c:1442: undefined reference to omp_get_thread_num' collect2: error: ld returned 1 exit status

I have no idea about it. May you help me to fix it? Thank you!

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 18 (10 by maintainers)

Most upvoted comments

I tried this myself with superlu_dist 6.3.1 and mfem 4.2 on Ubuntu 20.04, with gcc 9.3.0, openblas 0.3.8 (from ubuntu repo), openmpi 4.0.3.

It works for me.

I build parmetis as follows:

wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz
tar -xvzf parmetis-4.0.3.tar.gz
cd parmetis-4.0.3/
mkdir install
make config prefix=`pwd`/install
make
make install

I build superlu_dist as follows:

tar -xvzf v6.3.1.tar.gz
cd superlu_dist-6.3.1/
mkdir build
mkdir install
export PARMETIS_ROOT=`pwd`/../parmetis-4.0.3/
export PARMETIS_BUILD_DIR=`pwd`/../parmetis-4.0.3/build/Linux-x86_64
cd build
cmake .. \
   -DCMAKE_INSTALL_PREFIX=../install \
   -DTPL_PARMETIS_INCLUDE_DIRS="${PARMETIS_ROOT}/include;${PARMETIS_ROOT}/metis/include" \
   -DTPL_PARMETIS_LIBRARIES="${PARMETIS_BUILD_DIR}/libparmetis/libparmetis.a;${PARMETIS_BUILD_DIR}/libmetis/libmetis.a"
make
make install

To get superlu_dist to compile, in SRC/CMakeLists.txt I had to change

set(superlu_dist_libs ${MPI_C_LIBRARIES} ${BLAS_LIB} ${LAPACK_LIB}
    ${PARMETIS_LIB} ${COMBBLAS_LIB})

set(superlu_dist_libs ${MPI_C_LIBRARIES} ${MPI_CXX_LIBRARIES} ${BLAS_LIB} ${LAPACK_LIB}
    ${PARMETIS_LIB} ${COMBBLAS_LIB})

I also installed hypre 2.20.0.

For mfem 4.2:

wget https://bit.ly/mfem-4-2
tar -xvzf mfem-4-2
mkdir build
cd build
cmake ../ -DMFEM_USE_MPI=YES \
    -DMETIS_INCLUDE_DIRS=${PARMETIS_ROOT}/metis/include \
    -DMFEM_USE_SUPERLU=YES \
    -DSuperLUDist_DIR=/home/pieterg/LBL/mfem/superlu_dist-6.3.1/install \
    -DMFEM_USE_SUPERLU5=YES \
    -DParMETIS_INCLUDE_DIRS="${PARMETIS_ROOT}/include;${PARMETIS_ROOT}/metis/include" \
    -DMETIS_LIBRARIES="${PARMETIS_BUILD_DIR}/libparmetis/libparmetis.a;${PARMETIS_BUILD_DIR}/libmetis/libmetis.a" \
    -DMFEM_USE_METIS_5=YES

Then in examples/superlu:

make
OMP_NUM_THREADS=1 mpirun -n 4 superlu_ex1p

Gives as output:

No protocol specified
Options used:
   --mesh ../../data/star.mesh
   --order 1
   --device cpu
   --visualization
   --colperm 4
   --rowperm 1
   --rowperm 2
Device configuration: cpu
Memory configuration: host-std
Number of finite element unknowns: 5281
	Nonzeros in L         175170
	Nonzeros in U         175170
	nonzeros in L+U       345059
	nonzeros in LSUB       79907

** Memory Usage **********************************
** NUMfact space (MB): (sum-of-all-processes)
    L\U :            3.15 |  Total :    15.67
** Total highmark (MB):
    Sum-of-all :    18.32 | Avg :     4.58  | Max :     4.92
**************************************************

(the example is called superlu_ex1p for me)

pghysels on Nov 4, 2020

For AmgX, something like this should work (set CUDA_ARCH based on your GPU):

cd amgx
mkdir build
cd build
cmake .. -DCUDA_ARCH="70" -DCMAKE_INSTALL_PREFIX=../install
make all -j 8
make install

And then build MFEM with

cd mfem
make pcuda MFEM_USE_AMGX=YES AMGX_DIR="/path/to/amgx/install"

replacing /path/to with the appropriate direcctory.

v-dobrev on Nov 5, 2020

The default for superlu_dist is to use 32 bit integers for sparse matrices, but you can change that to 64 bit by running CMake with : -DXSDK_INDEX_SIZE=64 Is that what you did?

Make sure ParMETIS is also built with 32 bit integers. Check this line in parmetis-4.0.3/metis/include/metis.h: #define IDXTYPEWIDTH 32

pghysels on Nov 5, 2020

Looking at MFEM’s CMake build system, it looks like -DMFEM_USE_SUPERLU5=YES is actually not used, e.g. because it is not mentioned in https://github.com/mfem/mfem/blob/master/config/cmake/config.hpp.in – we should fix that.

We can probably auto-detect the version too, similar to the version detection for METIS 5 in https://github.com/mfem/mfem/blob/master/config/cmake/modules/FindMETIS.cmake.

v-dobrev on Nov 4, 2020

The name of the library directory should not matter, lib or lib64. On some platforms cmake uses lib64, on others lib. If you get lib instead of lib64, just replace lib64 with lib in defaults.mk here: https://github.com/mfem/mfem/blob/7cd6686e42db2d57c525e7b4c1691015b3bfafc3/config/defaults.mk#L246 Also, you should replace -lblas in the above line with the BLAS library that you used for building SuperLU_DIST – this library may have been autodetected by cmake – look at the output of the cmake command.

With that, if both SuperLU_DIST and MFEM build without errors, the example should work.

v-dobrev on Nov 4, 2020

What version of metis do you have installed? And where did you get it from? I see you add -I./../parmetis-4.0.3/include. But you are also using /usr/local/include/metis.h. It might be better to use the metis.h that comes with parmetis. If I download and build parmetis 4.0.3 or metis 5.1.0, I don’t have METIS_EXPORT anywhere

pghysels on Nov 2, 2020

I cannot reproduce this issue.

Are you by any chance using Ubuntu 18.04 with its default OpenMPI? If that is the case then the issue you see may be related to the issues described in #1773 – for a suggested fix see: https://github.com/mfem/mfem/issues/1773#issuecomment-701407853. Another option is to try to switch to MPICH.

v-dobrev on Nov 2, 2020

Try adding -fopenmp

pghysels on Nov 2, 2020