mfem: An error when using AMGX solver.

Hi Everybody! I am trying to apply AMGX solver in my problem. When I perform FEM using AMGX solver (combine with cuda), the vector displacement I receive is not right. Then I comeback the example 1 (https://github.com/mfem/mfem/blob/master/examples/amgx/ex1.cpp) and run the example 1 with cuda. I try printing the vector displacement like:

for(int i=0;i<X.Size();i++){
  cout<<X(i)<<"   ";
}

but I receive the result not right, too. It returns all garbage values. I think it is an error or I do not understand it. After FEM process, I would like to use the vector displacement to implement my problem. May you help me? Thank you!

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 18 (11 by maintainers)

Most upvoted comments

@artv3 thank you for your code of 2p and your advice. This is useful for me!!! Last question, I would like to ask you. I used to read the paper comparing the conditioning Hypre and AMGX. When I run the example 2p, the BoomerAMG preconditioner is very good for the problem. May you recommend me some preconditioners of AMGX good or better than BoomerAMG? It is very good if you have the file json. Sorry to bother you. I really appreciate your help. Thank you very much!

Great glad I can help. The integration of AmgX into MFEM is still fairly new and I’m not sure if we have tried it for elasticity problems yet (aside from here). I’m not 100% sure on what boomer is doing and if it can be done with AmgX yet, but as we learn more we can definitely add an example 2p with AmgX. Still early days.

artv3 on Nov 30, 2020

@artv3 Thanks for the quick reply.

Yes. I was using default settings, but I will give your settings a shot.

Edit: Yes. That seems to have done the trick. Thanks.

jonwong12 on Dec 8, 2020

@jonwong12 thanks for reaching out. Was it the default settings that you tried? The integration is fairly new and we are still assessing settings/performance with AmgX. One thing to try is a different smoother in AmgX, using branch (https://github.com/mfem/mfem/pull/1904), example 1p can be modified to take in an external JSON configuration:

AmgXSolver amgx;                                                                                                                                               
amgx.ReadParameters(amgx_json_file, AmgXSolver::EXTERNAL);  
amgx.InitExclusiveGPU(MPI_COMM_WORLD);  
amgx.SetOperator(*A.As<HypreParMatrix>());                                                                                                                     
amgx.ConfigureAs(AmgXSolver::PRECONDITIONER);                                                                                                                  
                                                                                                                                                                     
CGSolver cg(MPI_COMM_WORLD);                                                                                                                                   
cg.SetRelTol(1e-12);                                                                                                                                           
cg.SetMaxIter(2000);                                                                                                                                           
cg.SetPrintLevel(1);                                                                                                                                           
cg.SetPreconditioner(amgx);                                                                                                                                    
cg.SetOperator(*A.As<HypreParMatrix>());                                                                                                                       
cg.Mult(B, X);

Using the following JACOBI_L1 smoother I was able to go up to order 4 using your mesh:

{                                                                                                                                                                    
    "config_version": 2,                                                                                                                                             
    "solver": {                                                                                                                                                      
        "solver": "AMG",                                                                                                                                             
        "smoother": "JACOBI_L1",                                                                                                                                     
        "presweeps": 1,                                                                                                                                              
        "interpolator": "D2",                                                                                                                                        
        "max_iters": 2,                                                                                                                                             
        "scope": "main",                                                                                                                                             
        "max_row_sum": 0.9,                                                                                                                                          
        "strength_threshold": 0.25,                                                                                                                                  
        "max_levels": 1000,                                                                                                                                          
        "postsweeps": 1,                                                                                                                                             
        "tolerance": 0.0,                                                                                                                                            
        "norm": "L1",                                                                                                                                                
        "cycle": "V"                                                                                                                                                 
    }                                                                                                                                                                
}

 lrun -n 8 ./ex1p -o 4 --amgx-file JACOBI_L1.json -d cuda 
Options used:
   --mesh ballmesh.mesh
   --order 4
   --no-static-condensation
   --no-partial-assembly
   --amgx-lib
   --amgx-file JACOBI_L1.json
   --amgx-mpi-gpu-exclusive
   --device cuda
   --visualization
   --gpus-per-node-in-teams-mode 1
Device configuration: cuda,cpu
Memory configuration: host-std,cuda
Number of finite element unknowns: 2829313
AMGX version 2.1.0.131-opensource
Built on Oct 27 2020, 15:02:10
Compiled with CUDA Runtime 10.1, using CUDA driver 10.2
Cannot read file as JSON object, trying as AMGX config
Converting config string to current config version
Parsing configuration string: exception_handling=1 ; 
Using Normal MPI (Hostbuffer) communicator...
Using Normal MPI (Hostbuffer) communicator...
   Iteration :   0  (B r, r) = 0.12083
....
   Iteration :  42  (B r, r) = 2.79164e-25
   Iteration :  43  (B r, r) = 8.28688e-26
Average reduction factor = 0.523631

artv3 on Dec 7, 2020