arbor: error: 'make_value_iterator' is not a member of 'pybind11'

Describe the bug

Hi! While building Arbor on the JSC JUSUF HPC resource, I encounter an issue with an undefined function in pybind11:

...
[100%] Building CXX object python/CMakeFiles/pyarb_obj.dir/cells.cpp.o
[100%] Linking CXX executable ../bin/lmorpho
[100%] Built target lmorpho
/p/home/jusers/hagen2/jusuf/sources/arbor/python/cells.cpp: In lambda function:
/p/home/jusers/hagen2/jusuf/sources/arbor/python/cells.cpp:290:35: error: 'make_value_iterator' is not a member of 'pybind11'; did you mean 'make_key_iterator'?
  290 |                  return pybind11::make_value_iterator(ld.cache.begin(), ld.cache.end());
      |                                   ^~~~~~~~~~~~~~~~~~~
      |                                   make_key_iterator
make[2]: *** [python/CMakeFiles/pyarb_obj.dir/build.make:104: python/CMakeFiles/pyarb_obj.dir/cells.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:2017: python/CMakeFiles/pyarb_obj.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

To me, the issue appears to be that pybind11-v2.8.1 in the git submodule should be used (CMake finds it), but the system provided pybind11-v2.7.1 is still used during compilation. Seems like the required version was bumped via https://github.com/arbor-sim/arbor/pull/1742.

I could get it to compile by replacing make_value_iterator by make_key_iterator in cells.cpp, but I don’t know what sort of problems that may have introduced.

Installation script:

module load Python/3.6.9 SciPy-Stack GCC/11.2.0 ParaStationMPI/5.5.0-1 mpi4py/3.1.3 h5py/3.5.0 GSL/2.7 git/2.33.1-nodocs CMake/3.21.1 Doxygen/1.9.1 Autotools/20210726 Boost/1.78.0 PyYAML/5.4.1

cd arbor
git checkout master
git pull
git submodule update --init --recursive

mkdir build
cd build
rm -rf *

export CC=`which gcc`
export CXX=`which g++`
cmake -DARB_WITH_PYTHON=ON   -DARB_USE_BUNDLED_LIBS=ON   -DARB_WITH_MPI=OFF   -DARB_PYTHON_LIB_PATH=~/.local/lib/python3.9/site-packages/  -DCMAKE_INSTALL_PREFIX=~/.local ..
make -j

CMake output:

-- The C compiler identification is GNU 11.2.0
-- The CXX compiler identification is GNU 11.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /p/software/jusuf/stages/2022/software/GCCcore/11.2.0/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /p/software/jusuf/stages/2022/software/GCCcore/11.2.0/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Using Random123 submodule: /p/home/jusers/hagen2/jusuf/sources/arbor/ext>/random123
-- Found Python3: /p/software/jusuf/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/bin/python3.9 (found suitable version "3.9.6", minimum required is "3.7.0") found components: Interpreter Development Development.Module Development.Embed 
-- PYTHON_EXECUTABLE: /p/software/jusuf/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/bin/python3.9
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
Catalogue name:       dummy
Catalogue mechanisms: dummy
Extra cxx files:      
Catalogue sources:    /p/home/jusers/hagen2/jusuf/sources/arbor/test/unit/dummy
Catalogue output:     /p/home/jusers/hagen2/jusuf/sources/arbor/build/test/unit/generated/dummy
Build as standalone:  ON
Arbor cxx flags:      -march=native;$<$<BUILD_INTERFACE:$<COMPILE_LANGUAGE:CXX>>:-fvisibility=hidden>;$<$<BUILD_INTERFACE:$<COMPILE_LANGUAGE:CUDA>>:-Xcompiler=-fvisibility=hidden>
Arbor cxx compiler:   
Script prefix:        /p/home/jusers/hagen2/jusuf/sources/arbor/mechanisms
Current cxx compiler: /p/software/jusuf/stages/2022/software/GCCcore/11.2.0/bin/g++
-- Found Sphinx: /p/software/jusuf/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/bin/sphinx-build  
-- Found PY_svgwrite: /p/home/jusers/hagen2/jusuf/.local/lib/python3.9/site-packages/svgwrite  
-- pybind11 v2.8.1 
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Python module installation path: /p/home/jusers/hagen2/jusuf/.local/lib/python3.9/site-packages
-- Configuring done
-- Generating done
-- Build files have been written to: /p/home/jusers/hagen2/jusuf/sources/arbor/build

Context

Cluster info: https://apps.fz-juelich.de/jsc/hps/jusuf/index.html Arbor version: db887a9e4d5a986fa8f03c80dcf71eaf140588ea

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 23 (20 by maintainers)

Most upvoted comments

Here’s the culprit: SciPy-bundle, loaded via SciPy-Stack and is setting this

whatis("Description: Bundle of Python packages for scientific software")
whatis("Homepage: https://python.org/")
whatis("URL: https://python.org/")
whatis("Extensions: beniget-0.4.1, Bottleneck-1.3.2, deap-1.3.1, gast-0.5.2, mpmath-1.2.1, numexpr-2.7.3, numpy-1.21.3, pandas-1.3.4, ply-3.11, pythran-0.10.0, scipy-1.7.1")
conflict("SciPy-bundle")
load("imkl/.2021.4.0")
load("Python/3.9.6")
load("pybind11/.2.7.1")
prepend_path("CMAKE_PREFIX_PATH","/p/software/jusuf/stages/2022/software/SciPy-bundle/2021.10-gcccoremkl-11.2.0-2021.4.0")
prepend_path("LIBRARY_PATH","/p/software/jusuf/stages/2022/software/SciPy-bundle/2021.10-gcccoremkl-11.2.0-2021.4.0/lib")
prepend_path("PATH","/p/software/jusuf/stages/2022/software/SciPy-bundle/2021.10-gcccoremkl-11.2.0-2021.4.0/bin")
setenv("EBROOTSCIPYMINBUNDLE","/p/software/jusuf/stages/2022/software/SciPy-bundle/2021.10-gcccoremkl-11.2.0-2021.4.0")
setenv("EBVERSIONSCIPYMINBUNDLE","2021.10")
setenv("EBDEVELSCIPYMINBUNDLE","/p/software/jusuf/stages/2022/software/SciPy-bundle/2021.10-gcccoremkl-11.2.0-2021.4.0/easybuild/Compiler-GCCcore-11.2.0-SciPy-bundle-2021.10-easybuild-devel")
prepend_path("PYTHONPATH","/p/software/jusuf/stages/2022/software/SciPy-bundle/2021.10-gcccoremkl-11.2.0-2021.4.0/lib/python3.9/site-packages")
setenv("EBEXTSLISTSCIPYMINBUNDLE","numpy-1.21.3,ply-3.11,gast-0.5.2,beniget-0.4.1,pythran-0.10.0,scipy-1.7.1,numexpr-2.7.3,Bottleneck-1.3.2,pandas-1.3.4,mpmath-1.2.1,deap-1.3.1")

critically CMAKE_PREFIX_PATH is being set, which might be the issue and also why I cannot reproduce this locally, since I do not use modules.

For now, yes, just unload SciPy-Stack until the build is through and will figure out what’s wrong.

Hi @espenhgn,

thanks for reporting, that seems like a regression in our CMake indeed!