spack: spack crashing during openmpi installation due to fortran issues

@jwillenbring @tgamblin

My name is Alicia, and I’m a postdoc at SNL. I’ve been trying to install Trilinos via spack on my RHEL 6 machine with little success. It crashes during the openmpi installation, and reports that it’s because my fortran compiler is unable to produce working executables.

[amklinv@s995692 bin]$ ./spack install trilinos                                                     
==> Installing trilinos                                                                             
==> Installing superlu-dist                                                                         
==> Installing openmpi                                                                              
==> hwloc is already installed in /home/amklinv/spack/opt/spack/linux-redhat6-x86_64/gcc-5.3.0/hwloc-1.11.4-npuz6xtizklh6v5ckrk6wdv4wo7kfulq                                                            
==> Already downloaded /home/amklinv/spack/var/spack/stage/openmpi-2.0.1-ddxjy6r7tisyulta73rimvwzpmpcprol/openmpi-2.0.1.tar.bz2                                                                         
==> Already staged openmpi-2.0.1-ddxjy6r7tisyulta73rimvwzpmpcprol in /home/amklinv/spack/var/spack/stage/openmpi-2.0.1-ddxjy6r7tisyulta73rimvwzpmpcprol                                                 
==> No patches needed for openmpi                                                                   
==> Building openmpi                                                                                
==> WRITE LOCK: /home/amklinv/spack/opt/spack/.spack-db/prefix_lock[898272675117208261:1] [Released]
==> WRITE LOCK: /home/amklinv/spack/var/spack/stage/.lock[6797382048177427243:1] [Released]         
==> Error: Command exited with status 1:                                                            
'./configure' '--prefix=/home/amklinv/spack/opt/spack/linux-redhat6-x86_64/gcc-5.3.0/openmpi-2.0.1-ddxjy6r7tisyulta73rimvwzpmpcprol' '--with-hwloc=/home/amklinv/spack/opt/spack/linux-redhat6-x86_64/gcc-5.3.0/hwloc-1.11.4-npuz6xtizklh6v5ckrk6wdv4wo7kfulq' '--enable-shared' '--enable-static' '--enable-mpi-cxx' '--without-tm' '--without-slurm' '--without-psm' '--without-psm2' '--without-mxm' '--disable-mpi-thread-multiple' '--without-pmi' '--without-sqlite3' '--enable-vt' '--without-verbs'         

See build log for details:
  /tmp/amklinv/spack-stage/spack-stage-G8npQ1/openmpi-2.0.1/spack-build.out

/home/amklinv/spack/var/spack/repos/builtin/packages/openmpi/package.py:207, in install:
     163      def install(self, spec, prefix):                                          
     164          config_args = ["--prefix=%s" % prefix,                                
     165                         "--with-hwloc=%s" % spec['hwloc'].prefix,              
     166                         "--enable-shared",                                     
     167                         "--enable-static"]                                     
     168                                                                                
     169          # for Open-MPI 2.0:, C++ bindings are disabled by default.            
     170          if self.spec.satisfies('@2.0:'):                                      
     171              config_args.extend(['--enable-mpi-cxx'])                          
     172                                                                                
     173          if getattr(self, 'config_extra', None) is not None:                   
     174              config_args.extend(self.config_extra)                             
     175                                                                                
     176          # Variant based arguments                                             
     177          config_args.extend([                                                  
     178              # Schedulers                                                      
     179              '--with-tm' if '+tm' in spec else '--without-tm',                 
     180              '--with-slurm' if '+slurm' in spec else '--without-slurm',        
     181              # Fabrics                                                         
     182              '--with-psm' if '+psm' in spec else '--without-psm',              
     183              '--with-psm2' if '+psm2' in spec else '--without-psm2',           
     184              '--with-mxm' if '+mxm' in spec else '--without-mxm',              
     185              # Other options                                                   
     186              ('--enable-mpi-thread-multiple' if '+thread_multiple' in spec     
     187                  else '--disable-mpi-thread-multiple'),                        
     188              '--with-pmi' if '+pmi' in spec else '--without-pmi',              
     189              '--with-sqlite3' if '+sqlite3' in spec else '--without-sqlite3',  
     190              '--enable-vt' if '+vt' in spec else '--disable-vt'                
     191          ])                                                                    
     192          if '+verbs' in spec:
     193              path = _verbs_dir()
     194              if path is not None and path not in ('/usr', '/usr/local'):
     195                  config_args.append('--with-%s=%s' % (self.verbs, path))
     196              else:
     197                  config_args.append('--with-%s' % self.verbs)
     198          else:
     199              config_args.append('--without-%s' % self.verbs)
     200
     201          # TODO: use variants for this, e.g. +lanl, +llnl, etc.
     202          # use this for LANL builds, but for LLNL builds, we need:
     203          #     "--with-platform=contrib/platform/llnl/optimized"
     204          if self.version == ver("1.6.5") and '+lanl' in spec:
     205              config_args.append("--with-platform=contrib/platform/lanl/tlcc2/optimized-nopanasas")  # NOQA: ignore=E501
     206
  >> 207          configure(*config_args)
     208          make()
     209          make("install")
     210
     211          self.filter_compilers()
==> Error: Installation process had nonzero exit code : 256

My log file reads

[amklinv@s995692 bin]$ cat /tmp/amklinv/spack-stage/spack-stage-G8npQ1/openmpi-2.0.1/spack-build.out
==> './configure' '--prefix=/home/amklinv/spack/opt/spack/linux-redhat6-x86_64/gcc-5.3.0/openmpi-2.0.1-ddxjy6r7tisyulta73rimvwzpmpcprol' '--with-hwloc=/home/amklinv/spack/opt/spack/linux-redhat6-x86_64/gcc-5.3.0/hwloc-1.11.4-npuz6xtizklh6v5ckrk6wdv4wo7kfulq' '--enable-shared' '--enable-static' '--enable-mpi-cxx' '--without-tm' '--without-slurm' '--without-psm' '--without-psm2' '--without-mxm' '--disable-mpi-thread-multiple' '--without-pmi' '--without-sqlite3' '--enable-vt' '--without-verbs'     
configure: WARNING: unrecognized options: --without-sqlite3, --enable-vt                            

============================================================================
== Configuring Open MPI                                                     
============================================================================

*** Startup tests
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu 
checking target system type... x86_64-unknown-linux-gnu
checking for gcc... /home/amklinv/spack/lib/spack/env/gcc/gcc
checking whether the C compiler works... yes                 
checking for C compiler default output file name... a.out    
checking for suffix of executables...                        
checking whether we are cross compiling... no                
checking for suffix of object files... o                     
checking whether we are using the GNU C compiler... yes      
checking whether /home/amklinv/spack/lib/spack/env/gcc/gcc accepts -g... yes
checking for /home/amklinv/spack/lib/spack/env/gcc/gcc option to accept ISO C89... none needed
checking whether /home/amklinv/spack/lib/spack/env/gcc/gcc understands -c and -o together... yes
checking how to run the C preprocessor... /home/amklinv/spack/lib/spack/env/gcc/gcc -E          
checking for grep that handles long lines and -e... /bin/grep                                   
checking for egrep... /bin/grep -E                                                              
checking for ANSI C header files... yes                                                         
checking for sys/types.h... yes                                                                 
checking for sys/stat.h... yes                                                                  
checking for stdlib.h... yes                                                                    
checking for string.h... yes                                                                    
checking for memory.h... yes                                                                    
checking for strings.h... yes                                                                   
checking for inttypes.h... yes                                                                  
checking for stdint.h... yes                                                                    
checking for unistd.h... yes                                                                    
checking minix/config.h usability... no                                                         
checking minix/config.h presence... no                                                          
checking for minix/config.h... no                                                               
checking whether it is safe to define __EXTENSIONS__... yes                                     
checking for a BSD-compatible install... /usr/bin/install -c                                    
checking whether build environment is sane... yes                                               
checking for a thread-safe mkdir -p... /bin/mkdir -p                                            
checking for gawk... gawk                                                                       
checking whether make sets $(MAKE)... yes                                                       
checking for style of include used by make... GNU                                               
checking whether make supports nested variables... yes                                          
checking whether UID '85185' is supported by ustar format... yes                                
checking whether GID '29684' is supported by ustar format... yes                                
checking how to create a ustar tar archive... gnutar                                            
checking dependency style of /home/amklinv/spack/lib/spack/env/gcc/gcc... gcc3                  
checking whether make supports nested variables... (cached) yes                                 

*** Checking versions
checking for repo version... v2.0.0-257-gee86e07
checking Open MPI version... 2.0.1              
checking Open MPI release date... Sep 02, 2016  
checking Open MPI repository version... v2.0.0-257-gee86e07
checking for repo version... v2.0.0-257-gee86e07           
checking Open MPI Run-Time Environment version... 2.0.1    
checking Open MPI Run-Time Environment release date... Sep 02, 2016
checking Open MPI Run-Time Environment repository version... v2.0.0-257-gee86e07
checking for repo version... v2.0.0-257-gee86e07                                
checking Open SHMEM version... 2.0.1                                            
checking Open SHMEM release date... Sep 02, 2016                                
checking Open SHMEM repository version... v2.0.0-257-gee86e07                   
checking for repo version... v2.0.0-257-gee86e07                                
checking Open Portable Access Layer version... 2.0.1                            
checking Open Portable Access Layer release date... Sep 02, 2016                
checking Open Portable Access Layer repository version... v2.0.0-257-gee86e07   
checking for bootstrap Autoconf version... 2.69                                 
checking for bootstrap Automake version... 1.15                                 
checking for boostrap Libtool version... 2.4.6                                  

*** Initialization, setup
configure: builddir: /tmp/amklinv/spack-stage/spack-stage-G8npQ1/openmpi-2.0.1
configure: srcdir: /tmp/amklinv/spack-stage/spack-stage-G8npQ1/openmpi-2.0.1  
installing to directory "/home/amklinv/spack/opt/spack/linux-redhat6-x86_64/gcc-5.3.0/openmpi-2.0.1-ddxjy6r7tisyulta73rimvwzpmpcprol"                                                                   

*** OPAL Configuration options
checking if want to run code coverage... no
checking if want to compile with branch probabilities... no
checking if want to debug memory usage... no               
checking if want to profile memory usage... no             
checking if want developer-level compiler pickyness... no  
checking if want developer-level debugging code... no      
checking if want to developer-level timing framework... no 
checking if want to install project-internal header files... no
checking if want pretty-print stacktrace... yes                
checking if want pty support... yes                            
checking if want weak symbol support... yes                    
checking if want dlopen support... yes                         
checking if want heterogeneous support... no                   
checking if word-sized integers must be word-size aligned... no
checking if want IPv6 support... no                            
checking if want package/brand string... Open MPI amklinv@s995692.srn.sandia.gov Distribution
checking if want ident string... 2.0.1                                                       
checking if want to use an alternative checksum algo for messages... no                      
checking maximum length of processor name... 256                                             
checking maximum length of error string... 256                                               
checking maximum length of object name... 64                                                 
checking maximum length of info key... 36                                                    
checking maximum length of info val... 256                                                   
checking maximum length of port name... 1024                                                 
checking maximum length of datarep string... 128                                             
checking if want getpwuid support... yes                                                     
checking __NetBSD__... no                                                                    
checking __FreeBSD__... no                                                                   
checking __OpenBSD__... no                                                                   
checking __DragonFly__... no                                                                 
checking __386BSD__... no                                                                    
checking __bsdi__... no                                                                      
checking __APPLE__... no                                                                     
checking __linux__... yes                                                                    
checking __sun__... no                                                                       
checking __sun... no                                                                         
checking netdb.h usability... yes                                                            
checking netdb.h presence... yes                                                             
checking for netdb.h... yes                                                                  
checking netinet/in.h usability... yes                                                       
checking netinet/in.h presence... yes                                                        
checking for netinet/in.h... yes                                                             
checking netinet/tcp.h usability... yes                                                      
checking netinet/tcp.h presence... yes                                                       
checking for netinet/tcp.h... yes                                                            
checking for struct sockaddr_in... yes                                                       
checking if --with-cuda is set... not set (--with-cuda=)                                     
checking whether CU_POINTER_ATTRIBUTE_SYNC_MEMOPS is declared... no                          
checking whether cuPointerGetAttributes is declared... no                                    
checking if have cuda support... no                                                          

*** ORTE Configuration options
checking if want orterun "--prefix" behavior to be enabled by default... no
checking if want orte static ports... yes                                  

*** OMPI Configuration options
checking if want compile-time warnings inside of mpi.h... yes
checking if want sparse process groups... no                 
checking if want peruse support... no                        
checking if want Fortran MPI bindings...  (try)              
checking if want C++ bindings... yes                         
checking if want MPI::SEEK_SET support... yes                
checking if want run-time MPI parameter checking... runtime  
checking which 'use mpi_f08' implementation to use... regular (no subarray support)

*** OSHMEM Configuration options
checking if want oshmem... yes  
checking if want SGI/Quadrics compatibility mode... yes
checking if want OSHMEM API parameter checking... always
checking for on_exit... yes                             
checking if want pshmem... yes                          
checking if want to build OSHMEM fortran bindings... yes
checking if want wrapper compiler rpath support... yes  

============================================================================
== Compiler and preprocessor tests                                          
============================================================================

*** C compiler and preprocessor
checking for gcc... (cached) /home/amklinv/spack/lib/spack/env/gcc/gcc
checking whether we are using the GNU C compiler... (cached) yes      
checking whether /home/amklinv/spack/lib/spack/env/gcc/gcc accepts -g... (cached) yes
checking for /home/amklinv/spack/lib/spack/env/gcc/gcc option to accept ISO C89... (cached) none needed                                                                                                 
checking whether /home/amklinv/spack/lib/spack/env/gcc/gcc understands -c and -o together... (cached) yes                                                                                               
checking for /home/amklinv/spack/lib/spack/env/gcc/gcc option to accept ISO C99... none needed      
checking for the C compiler vendor... gnu                                                           
checking for ANSI C header files... (cached) yes                                                    
checking if /home/amklinv/spack/lib/spack/env/gcc/gcc supports -finline-functions... yes            
checking if /home/amklinv/spack/lib/spack/env/gcc/gcc supports -fno-strict-aliasing... yes          
configure: WARNING:  -fno-strict-aliasing has been added to CFLAGS                                  
checking if /home/amklinv/spack/lib/spack/env/gcc/gcc supports __builtin_expect... yes              
checking if /home/amklinv/spack/lib/spack/env/gcc/gcc supports __builtin_prefetch... yes            
checking if /home/amklinv/spack/lib/spack/env/gcc/gcc supports __builtin_clz... yes                 
checking for C optimization flags... -O3 -DNDEBUG -finline-functions -fno-strict-aliasing           
checking for Interix environment... no                                                              
checking for C ident string support... #ident                                                       
checking for int8_t... yes                                                                          
checking for uint8_t... yes                                                                         
checking for int16_t... yes                                                                         
checking for uint16_t... yes                                                                        
checking for int32_t... yes                                                                         
checking for uint32_t... yes                                                                        
checking for int64_t... yes                                                                         
checking for uint64_t... yes                                                                        
checking for int128_t... no                                                                         
checking for __int128... yes                                                                        
checking for uint128_t... no                                                                        
checking for long long... yes                                                                       
checking for __float128... yes                                                                      
checking for long double... yes                                                                     
checking complex.h usability... yes                                                                 
checking complex.h presence... yes                                                                  
checking for complex.h... yes                                                                       
checking for float _Complex... yes                                                                  
checking for double _Complex... yes                                                                 
checking for long double _Complex... yes                                                            
checking for intptr_t... yes                                                                        
checking for uintptr_t... yes                                                                       
checking for mode_t... yes                                                                          
checking for ssize_t... yes                                                                         
checking for ptrdiff_t... yes                                                                       
checking size of char... 1                                                                          
checking size of short... 2                                                                         
checking size of int... 4                                                                           
checking size of long... 8                                                                          
checking size of long long... 8                                                                     
checking size of float... 4                                                                         
checking size of double... 8                                                                        
checking size of long double... 16                                                                  
checking size of __float128... 16                                                                   
checking size of float _Complex... 8                                                                
checking size of double _Complex... 16                                                              
checking size of long double _Complex... 32                                                         
checking size of void *... 8                                                                        
checking size of size_t... 8                                                                        
checking size of ssize_t... 8                                                                       
checking size of ptrdiff_t... 8                                                                     
checking size of wchar_t... 4                                                                       
checking size of pid_t... 4                                                                         
checking alignment of _Bool... 1                                                                    
checking alignment of int8_t... 1                                                                   
checking alignment of int16_t... 2                                                                  
checking alignment of int32_t... 4                                                                  
checking alignment of int64_t... 8                                                                  
checking alignment of char... 1                                                                     
checking alignment of short... 2                                                                    
checking alignment of wchar_t... 4                                                                  
checking alignment of int... 4                                                                      
checking alignment of long... 8                                                                     
checking alignment of long long... 8                                                                
checking alignment of float... 4                                                                    
checking alignment of double... 8                                                                   
checking alignment of long double... 16                                                             
checking alignment of __float128... 16                                                              
checking alignment of float _Complex... 4                                                           
checking alignment of double _Complex... 8                                                          
checking alignment of long double _Complex... 16                                                    
checking alignment of void *... 8                                                                   
checking alignment of size_t... 8                                                                   
checking for C bool type... no                                                                      
checking size of _Bool... 1                                                                         
checking for weak symbol support... yes                                                             
checking for macro weak symbol support... yes                                                       
checking for functional offsetof macro... yes                                                       

*** C++ compiler and preprocessor
checking whether we are using the GNU C++ compiler... yes
checking whether /home/amklinv/spack/lib/spack/env/gcc/g++ accepts -g... yes
checking dependency style of /home/amklinv/spack/lib/spack/env/gcc/g++... gcc3
checking how to run the C++ preprocessor... /home/amklinv/spack/lib/spack/env/gcc/g++ -E
checking for the C++ compiler vendor... gnu                                             
checking if /home/amklinv/spack/lib/spack/env/gcc/g++ supports -finline-functions... yes
configure: WARNING:  -finline-functions has been added to CXXFLAGS                      
checking if C and C++ are link compatible... yes                                        
checking for C++ optimization flags... -O3 -DNDEBUG -finline-functions                  
checking size of bool... 1                                                              
checking alignment of bool... 1                                                         

*** C++ compiler and preprocessor
checking whether we are using the GNU C++ compiler... (cached) yes
checking whether /home/amklinv/spack/lib/spack/env/gcc/g++ accepts -g... (cached) yes
checking dependency style of /home/amklinv/spack/lib/spack/env/gcc/g++... (cached) gcc3
checking how to run the C++ preprocessor... /home/amklinv/spack/lib/spack/env/gcc/g++ -E
checking if C++ compiler works... yes                                                   
checking if /home/amklinv/spack/lib/spack/env/gcc/g++ supports -finline-functions... yes
configure: WARNING:  -finline-functions has been added to CXXFLAGS                      
checking if C and C++ are link compatible... (cached) yes                               
checking for C++ optimization flags... -O3 -DNDEBUG -finline-functions                  
checking size of bool... (cached) 1                                                     
checking alignment of bool... (cached) 1                                                
checking if able to build the MPI C++ bindings... yes                                   
checking for the C++ compiler vendor... (cached) gnu                                    
checking if want C++ exception handling... no                                           
checking for C++ template_repository_directory... not used                              
checking for C++ compiler template parameters... none needed                            
checking for C++ ident string support... #ident                                         
checking if /home/amklinv/spack/lib/spack/env/gcc/g++ supports __builtin_expect... yes  
checking if /home/amklinv/spack/lib/spack/env/gcc/g++ supports __builtin_prefetch... yes
checking if /home/amklinv/spack/lib/spack/env/gcc/g++ supports const_cast<> properly... yes

*** Compiler characteristics
checking for __attribute__... yes
checking for __attribute__(aligned)... yes
checking for __attribute__(always_inline)... yes
checking for __attribute__(cold)... yes         
checking for __attribute__(const)... yes        
checking for __attribute__(deprecated)... yes   
checking for __attribute__(deprecated_argument)... yes
checking for __attribute__(format)... yes             
checking for __attribute__(format_funcptr)... yes     
checking for __attribute__(hot)... yes                
checking for __attribute__(malloc)... yes             
checking for __attribute__(may_alias)... yes          
checking for __attribute__(no_instrument_function)... yes
checking for __attribute__(noinline)... yes              
checking for __attribute__(nonnull)... yes               
checking for __attribute__(noreturn)... yes              
checking for __attribute__(noreturn_funcptr)... yes      
checking for __attribute__(packed)... yes                
checking for __attribute__(pure)... yes                  
checking for __attribute__(sentinel)... yes              
checking for __attribute__(unused)... yes                
checking for __attribute__(visibility)... yes            
checking for __attribute__(warn_unused_result)... yes    
checking for __attribute__(weak_alias)... yes            
checking for __attribute__(destructor)... yes            
checking for compiler familyid... 1                      
checking for compiler familyname... GNU                  
checking for compiler version... 328448                  
checking for compiler version_str... 5.3.0               

*** Java compiler
checking OSX locations... not found
checking Linux locations... not found
checking Solaris locations... not found
checking Java support available... no  

*** Java MPI bindings
checking if want Java bindings... no

*** OpenSHMEM profiling
checking if pshmem will be enabled... yes (weak symbols supported)

*** Assembler
checking dependency style of /home/amklinv/spack/lib/spack/env/gcc/gcc... gcc3
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B         
checking the name lister (/usr/bin/nm -B) interface... BSD nm                 
checking for fgrep... /bin/grep -F                                            
checking if .proc/endp is needed... no                                        
checking directive for setting text section... .text                          
checking directive for exporting symbols... .globl                            
checking for objdump... objdump                                               
checking if .note.GNU-stack is needed... yes                                  
checking suffix for labels... :                                               
checking prefix for global symbol labels...                                   
checking prefix for lsym labels... .L                                         
checking prefix for function in .type... @                                    
checking if .size is needed... yes                                            
checking if .align directive takes logarithmic value... no                    
checking if processor supports x86_64 16-byte compare-and-exchange... yes     
checking if compiler correctly handles volatile 128bits... yes                
checking if /home/amklinv/spack/lib/spack/env/gcc/gcc supports GCC inline assembly... yes
checking if /home/amklinv/spack/lib/spack/env/gcc/gcc supports DEC inline assembly... no 
checking if /home/amklinv/spack/lib/spack/env/gcc/gcc supports XLC inline assembly... no 
checking for assembly format... default-.text-.globl-:--.L-@-1-0-1-1-1                   
checking for assembly architecture... AMD64                                              
checking for builtin atomics... BUILTIN_NO
checking for perl... perl
checking for pre-built assembly file... yes (atomic-amd64-linux.s)
checking for atomic assembly filename... atomic-amd64-linux.s

*** Fortran compiler
checking whether we are using the GNU Fortran compiler... yes
checking whether /home/amklinv/spack/lib/spack/env/gcc/gfortran accepts -g... yes
configure: WARNING: Open MPI now ignores the F77 and FFLAGS environment variables; only the FC and FCFLAGS environment variables are used.
checking whether ln -s works... yes
checking if Fortran compiler works... no
**********************************************************************
* It appears that your Fortran compiler is unable to produce working
* executables.  A simple test application failed to properly
* execute.  Note that this is likely not a problem with Open MPI,
* but a problem with the local compiler installation.  More
* information (including exactly what command was given to the
* compiler and what error resulted when the command was executed) is
* available in the config.log file in the Open MPI build directory.
**********************************************************************
configure: error: Could not run a simple Fortran program.  Aborting.

I would appreciate any help you could provide in resolving this issue and would be happy to provide any additional information that might be helpful.

About this issue

  • Original URL
  • State: closed
  • Created 10 years ago
  • Comments: 39 (29 by maintainers)

Most upvoted comments

I had spack install an updated version of binutils, and I thought I loaded the module correctly, but I guess I didn’t. Loading the binutils module fixed the issue, and I have a spack install of Trilinos now. Thanks all!

@davydden: it should, unless there’s something else missing from the environment. If spack install --dirty fixes this we should look at why the module setting doesn’t.

spack install --dirty trilinos

ought to do the trick. The --dirty flag prevents Spack from sanitizing the build environment. Obviously this isn’t ideal since you’ll need to do it with every installation. I can’t wait for #943 to be merged 😃

Edit: @tgamblin, you beat me to it.

@amklinv the error is ./conftest: error while loading shared libraries: libquadmath.so.0: cannot open shared object file: No such file or directory This may be related to https://www.centos.org/forums/viewtopic.php?t=3237. You need to check that this library exists, since it is not on your system by default.