mfem: PETSc examples 5p/11p broken starting with PETSc v3.14.0

When running make test in the PETSc examples directory (examples/petsc), the examples ex5p and ex11p (ex11p is only run when SLEPc support is enabled) fail randomly. In order to see the issue consistently, one has to run with valgrind or build MUMPS with ASan (e.g. adding the flags -fsanitize=address -fno-omit-frame-pointer -g). The ASan error looks like this:

...

=================================================================
==98745==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020002957cc at pc 0x0001115c76b7 bp 0x7ffee6c55ef0 sp 0x7ffee6c55ee8
READ of size 4 at 0x6020002957cc thread T0
    #0 0x1115c76b6 in dmumps_dr_assemble_local.4 dsol_distrhs.F:275
    #1 0x1115c3a90 in dmumps_scatter_dist_rhs_ dsol_distrhs.F:160
    #2 0x11159d0aa in dmumps_solve_driver_ dsol_driver.F:3608
    #3 0x111774e9a in dmumps_ dmumps_driver.F:2009
    #4 0x111786cc7 in dmumps_f77_ dmumps_f77.F:289
    #5 0x111756cf9 in dmumps_c mumps_c.c:485
    #6 0x10d895c1f in MatSolve_MUMPS mumps.c:1188
    #7 0x10ca25d44 in MatSolve matrix.c:3398
    #8 0x10ee8e36f in PCApply_Cholesky cholesky.c:175
    #9 0x10f6de5a7 in PCApply precon.c:444
    #10 0x10f96e48d in KSP_PCApply kspimpl.h:283
    #11 0x10f96fc7e in KSPSolve_PREONLY preonly.c:22
    #12 0x10f742264 in KSPSolve_Private itfunc.c:719
    #13 0x10f7469f8 in KSPSolve itfunc.c:889
    #14 0x10f131081 in PCBDDCApplyInterfacePreconditioner bddcprivate.c:5919
    #15 0x10f033ad3 in PCApply_BDDC bddc.c:1986
    #16 0x10f6de5a7 in PCApply precon.c:444
    #17 0x10fa5614c in KSP_PCApply kspimpl.h:283
    #18 0x10fa59bc5 in KSPSolve_CG cg.c:142
    #19 0x10f742264 in KSPSolve_Private itfunc.c:719
    #20 0x10f7469f8 in KSPSolve itfunc.c:889
    #21 0x1091c235b in mfem::PetscLinearSolver::MultKernel(mfem::Vector const&, mfem::Vector&, bool) const petsc.cpp:3095
    #22 0x1091c28d0 in mfem::PetscLinearSolver::Mult(mfem::Vector const&, mfem::Vector&) const petsc.cpp:3103
    #23 0x108f9160a in main ex5p.cpp:416
    #24 0x7fff20952f3c in start+0x0 (libdyld.dylib:x86_64+0x15f3c)

0x6020002957cc is located 4 bytes to the left of 1-byte region [0x6020002957d0,0x6020002957d1)
allocated by thread T0 here:
    #0 0x114252597 in wrap_malloc+0xb7 (libasan.6.dylib:x86_64+0x4c597)
    #1 0x1115c282b in dmumps_scatter_dist_rhs_ dsol_distrhs.F:130
    #2 0x11159d0aa in dmumps_solve_driver_ dsol_driver.F:3608
    #3 0x111774e9a in dmumps_ dmumps_driver.F:2009
    #4 0x111786cc7 in dmumps_f77_ dmumps_f77.F:289
    #5 0x111756cf9 in dmumps_c mumps_c.c:485
    #6 0x10d895c1f in MatSolve_MUMPS mumps.c:1188
    #7 0x10ca25d44 in MatSolve matrix.c:3398
    #8 0x10ee8e36f in PCApply_Cholesky cholesky.c:175
    #9 0x10f6de5a7 in PCApply precon.c:444
    #10 0x10f96e48d in KSP_PCApply kspimpl.h:283
    #11 0x10f96fc7e in KSPSolve_PREONLY preonly.c:22
    #12 0x10f742264 in KSPSolve_Private itfunc.c:719
    #13 0x10f7469f8 in KSPSolve itfunc.c:889
    #14 0x10f131081 in PCBDDCApplyInterfacePreconditioner bddcprivate.c:5919
    #15 0x10f033ad3 in PCApply_BDDC bddc.c:1986
    #16 0x10f6de5a7 in PCApply precon.c:444
    #17 0x10fa5614c in KSP_PCApply kspimpl.h:283
    #18 0x10fa59bc5 in KSPSolve_CG cg.c:142
    #19 0x10f742264 in KSPSolve_Private itfunc.c:719
    #20 0x10f7469f8 in KSPSolve itfunc.c:889
    #21 0x1091c235b in mfem::PetscLinearSolver::MultKernel(mfem::Vector const&, mfem::Vector&, bool) const petsc.cpp:3095
    #22 0x1091c28d0 in mfem::PetscLinearSolver::Mult(mfem::Vector const&, mfem::Vector&) const petsc.cpp:3103
    #23 0x108f9160a in main ex5p.cpp:416
    #24 0x7fff20952f3c in start+0x0 (libdyld.dylib:x86_64+0x15f3c)

SUMMARY: AddressSanitizer: heap-buffer-overflow dsol_distrhs.F:275 in dmumps_dr_assemble_local.4
Shadow bytes around the buggy address:
  0x1c0400052aa0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c0400052ab0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c0400052ac0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c0400052ad0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x1c0400052ae0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x1c0400052af0: fa fa fa fa fa fa fa fa fa[fa]01 fa fa fa fd fa
  0x1c0400052b00: fa fa 00 00 fa fa 00 00 fa fa fd fa fa fa 00 00
  0x1c0400052b10: fa fa 00 fa fa fa fd fa fa fa 00 00 fa fa 00 00
  0x1c0400052b20: fa fa 00 fa fa fa fd fa fa fa fd fa fa fa fd fa
  0x1c0400052b30: fa fa fd fa fa fa fd fa fa fa 04 fa fa fa fd fa
  0x1c0400052b40: fa fa fd fa fa fa fd fa fa fa 00 fa fa fa fd fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==98745==ABORTING

...

This is with the latest MUMPS, v5.4.1, however, older versions have the same issue.

Newer versions of PETSc and SLEPc (up to the latest development versions, I think) show the same behavior.

Even though the reported error is in MUMPS, the actual problem seems to be in PETSc v3.14.0 (and newer) because the previous release, v3.13.6, and older versions, do not show this issue.