xtensor: Some errors when submit pull requests to conda-forge using xtensor and xtensor-blas

Greetings, Recently I developed a package using xtensor as C++ backend and numpy as a python interface, which is usually like @tdegeus done for some of his packages. I also used the xtensor-blas in package, and everything is good in my local computer and github actions for pypi publishment, I also tried conda build . in my computer. But when I tried to submit the PR to conda-forge, I got the error like followings:

{} is not a Mach-O file
Can't parse the binary at the index #0
{} is not a Mach-O file
Can't parse the binary at the index #1
   INFO: sysroot: '/Applications/Xcode_13.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/' files: '['usr/share/man/mann/zero.n', 'usr/share/man/mann/yencode.n', 'usr/share/man/mann/yaml.n', 'usr/share/man/mann/xsxp.n']'
  ERROR (noiseflow,lib/python3.10/site-packages/noiseflow/lib/signal_share.cpython-310-darwin.so): Needed DSO lib/libopenblas.0.dylib found in ['libopenblas']
  ERROR (noiseflow,lib/python3.10/site-packages/noiseflow/lib/signal_share.cpython-310-darwin.so): .. but ['libopenblas'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
   INFO (noiseflow,lib/python3.10/site-packages/noiseflow/lib/signal_share.cpython-310-darwin.so): Needed DSO /Applications/Xcode_13.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/lib/libSystem.B.dylib found in $SYSROOT
   INFO (noiseflow,lib/python3.10/site-packages/noiseflow/lib/signal_share.cpython-310-darwin.so): Needed DSO lib/libomp.dylib found in conda-forge::llvm-openmp-15.0.7-h61d9ccf_0
   INFO (noiseflow,lib/python3.10/site-packages/noiseflow/lib/signal_share.cpython-310-darwin.so): Needed DSO lib/libc++.1.dylib found in conda-forge::libcxx-16.0.6-hd57cbcb_0
  ERROR (noiseflow,lib/python3.10/site-packages/noiseflow/lib/cc_share.cpython-310-darwin.so): Needed DSO lib/libopenblas.0.dylib found in ['libopenblas']
  ERROR (noiseflow,lib/python3.10/site-packages/noiseflow/lib/cc_share.cpython-310-darwin.so): .. but ['libopenblas'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
   INFO (noiseflow,lib/python3.10/site-packages/noiseflow/lib/cc_share.cpython-310-darwin.so): Needed DSO /Applications/Xcode_13.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/lib/libSystem.B.dylib found in $SYSROOT
   INFO (noiseflow,lib/python3.10/site-packages/noiseflow/lib/cc_share.cpython-310-darwin.so): Needed DSO lib/libfftw3.3.dylib found in conda-forge::fftw-3.3.10-nompi_h4fa670e_108
   INFO (noiseflow,lib/python3.10/site-packages/noiseflow/lib/cc_share.cpython-310-darwin.so): Needed DSO lib/libfftw3f.3.dylib found in conda-forge::fftw-3.3.10-nompi_h4fa670e_108
   INFO (noiseflow,lib/python3.10/site-packages/noiseflow/lib/cc_share.cpython-310-darwin.so): Needed DSO lib/libomp.dylib found in conda-forge::llvm-openmp-15.0.7-h61d9ccf_0
   INFO (noiseflow,lib/python3.10/site-packages/noiseflow/lib/cc_share.cpython-310-darwin.so): Needed DSO lib/libc++.1.dylib found in conda-forge::libcxx-16.0.6-hd57cbcb_0
WARNING (noiseflow): run-exports library package conda-forge::numpy-1.21.6-py310hb608527_0 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
Traceback (most recent call last):
  File "/Users/runner/work/1/s/.ci_support/build_all.py", line 218, in <module>
    build_all(os.path.join(root_dir, "recipes"), args.arch)
  File "/Users/runner/work/1/s/.ci_support/build_all.py", line 118, in build_all
    build_folders(recipes_dir, folders, arch, channel_urls)
  File "/Users/runner/work/1/s/.ci_support/build_all.py", line 174, in build_folders
    conda_build.api.build([recipe], config=get_config(arch, channel_urls))
  File "/Users/runner/mambaforge/lib/python3.10/site-packages/conda_build/api.py", line 253, in build
    return build_tree(
  File "/Users/runner/mambaforge/lib/python3.10/site-packages/conda_build/build.py", line 3799, in build_tree
    packages_from_this = build(
  File "/Users/runner/mambaforge/lib/python3.10/site-packages/conda_build/build.py", line 2875, in build
    newly_built_packages = bundlers[pkg_type](output_d, m, env, stats)
  File "/Users/runner/mambaforge/lib/python3.10/site-packages/conda_build/build.py", line 2002, in bundle_conda
    files = post_process_files(metadata, initial_files)
  File "/Users/runner/mambaforge/lib/python3.10/site-packages/conda_build/build.py", line 1813, in post_process_files
    post_build(m, new_files, build_python=python)
  File "/Users/runner/mambaforge/lib/python3.10/site-packages/conda_build/post.py", line 1721, in post_build
    check_overlinking(m, files, host_prefix)
  File "/Users/runner/mambaforge/lib/python3.10/site-packages/conda_build/post.py", line 1605, in check_overlinking
    return check_overlinking_impl(
  File "/Users/runner/mambaforge/lib/python3.10/site-packages/conda_build/post.py", line 1576, in check_overlinking_impl
    raise OverLinkingError(overlinking_errors)
conda_build.exceptions.OverLinkingError: overlinking check failed 
["  ERROR (noiseflow,lib/python3.10/site-packages/noiseflow/lib/signal_share.cpython-310-darwin.so): .. but ['libopenblas'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)", "  ERROR (noiseflow,lib/python3.10/site-packages/noiseflow/lib/cc_share.cpython-310-darwin.so): .. but ['libopenblas'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)"]
##[error]Bash exited with code '1'.
Finishing: Run OSX build

And the following is my cmake file:

###################################################################  
#   Optimization Parameters
################################################################### 
cmake_minimum_required(VERSION 3.18)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_FLAGS "-Wall -g -fPIC -O3")
set(XTENSOR_USE_XSIMD 1) # set xtensor to use xsimd for SIMD optimization
set(CMAKE_PREFIX_PATH $ENV{CONDA_PREFIX}) # set conda env path, so that cmake can find those libraries in conda env




###################################################################  
#   Project Info & Compiler Detection
###################################################################  
project(noiseflow_share VERSION 0.1.0 LANGUAGES CXX)
message("Check C++ compiler: ${CMAKE_CXX_COMPILER_ID}")
if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
    if (MINGW)
        message("Using MinGW-w64")
        set(COMPILER "MinGW-w64")
    else()
        message("Using GCC")
        set(COMPILER "GCC")
    endif()
elseif ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "Clang")
    message("Using Clang")
    set(COMPILER "Clang")
elseif ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "AppleClang")
    message("Using AppleClang")
    set(COMPILER "AppleClang")
elseif ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "MSVC")
    message("Using MSVC")
    set(COMPILER "MSVC")
elseif ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "Intel")
    message("Using Intel Compiler")
    set(COMPILER "Intel")
elseif ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "IntelLLVM")
    message("Using Intel oneAPI Compiler")
    set(COMPILER "Intel oneAPI")
else()
    message(FATAL_ERROR "Unrecognized compiler: ${CMAKE_CXX_COMPILER_ID}")
endif()




###################################################################  
#   Find Libraries
###################################################################  
### [a]. use find_package for numpy
find_package(Python REQUIRED COMPONENTS Interpreter Development NumPy)
### [b]. use execute_process for numpy
# execute_process(
#   COMMAND "${PYTHON_EXECUTABLE}"
#   -c "import numpy; print(numpy.get_include())"
#   OUTPUT_VARIABLE NumPy_INCLUDE_DIRS
#   OUTPUT_STRIP_TRAILING_WHITESPACE
# )


### OpenMP, check if the BREW_OPENMP environment variable is set (clang doesn't bring openmp, you need to install it by brew, and find_package can't find it, you need to set the PATH as below. refer to https://www.kancloud.cn/csyangbinbin/cmake-cookbook1/2157938  https://iscinumpy.gitlab.io/post/omp-on-high-sierra/)
if(DEFINED ENV{BREW_OPENMP})
  if($ENV{BREW_OPENMP} EQUAL 1)
    message(STATUS "Searching for OpenMP in brew path")
    execute_process(COMMAND brew --prefix OUTPUT_VARIABLE BREW_PREFIX OUTPUT_STRIP_TRAILING_WHITESPACE)
    set(OpenMP_CXX_FLAGS "-Xpreprocessor -fopenmp -I${BREW_PREFIX}/opt/libomp/include")
    set(OpenMP_CXX_LIB_NAMES "omp")
    set(OpenMP_omp_LIBRARY "${BREW_PREFIX}/opt/libomp/lib")
    find_package(OpenMP REQUIRED)
  else()
    message(STATUS "Searching for OpenMP in default path")
    find_package(OpenMP REQUIRED)
  endif()
else()
  message(STATUS "BREW_OPENMP not set, searching for OpenMP in default path")
  find_package(OpenMP REQUIRED)
endif()


### pybind11 xtensor xtl xsimd xtensor-blas xtensor-python
find_package(pybind11 REQUIRED CONFIG)
find_package(xtensor REQUIRED)
find_package(xtl REQUIRED)
find_package(xsimd REQUIRED)
find_package(xtensor-blas REQUIRED)
find_package(xtensor-python REQUIRED)


### xtensor-fftw
set(FFTW_USE_FLOAT ON CACHE BOOL "Use float precision") # set fftw3 to use float precision
set(FFTW_USE_DOUBLE ON CACHE BOOL "Use double precision") # set fftw3 to use double precision
set(FFTW_USE_LONG_DOUBLE OFF CACHE BOOL "Use long double precision") # set fftw3 to not use long double precision
include_directories("./extern/xtensor-fftw/include")


### FFTW3 using FindFFTW.cmake (note this command will seach some default path to find fftw3, for example the conda env path)
set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_CURRENT_SOURCE_DIR}/extern/xtensor-fftw/cmake/Modules/findFFTW") # use FindFFTW.cmake to find fftw3
find_package(FFTW REQUIRED COMPONENTS ${REQUIRE_FLOAT_LIB} ${REQUIRE_DOUBLE_LIB} ${REQUIRE_LONG_DOUBLE_LIB})
if(FFTW_FOUND)
    message(STATUS "FFTW Found!")
endif()


### KFR 
include_directories("./extern/kfr/include")




###################################################################  
#   Target -- Python Package Module
################################################################### 
pybind11_add_module(cc_share ${CMAKE_SOURCE_DIR}/noiseflow/cc/src/pybind11.cpp)
target_include_directories(cc_share PRIVATE ${CMAKE_SOURCE_DIR}/noiseflow/cc/src ${CMAKE_SOURCE_DIR}/noiseflow/cc/include)
target_link_libraries(cc_share PUBLIC ${BLAS_LIBRARIES} ${LAPACK_LIBRARIES} ${FFTW_LIBRARIES} pybind11::module xtensor xtensor::optimize xtensor::use_xsimd xtensor-blas xtensor-python Python::NumPy OpenMP::OpenMP_CXX) 


pybind11_add_module(signal_share ${CMAKE_SOURCE_DIR}/noiseflow/signal/src/pybind11.cpp)
target_include_directories(signal_share PRIVATE ${CMAKE_SOURCE_DIR}/noiseflow/signal/src ${CMAKE_SOURCE_DIR}/noiseflow/signal/include)
target_link_libraries(signal_share PUBLIC ${BLAS_LIBRARIES} ${LAPACK_LIBRARIES} ${FFTW_LIBRARIES} pybind11::module xtensor xtensor::optimize xtensor::use_xsimd xtensor-blas xtensor-python Python::NumPy OpenMP::OpenMP_CXX)


### setup.py will install the module to the python site-packages
# install(TARGETS cc_share DESTINATION lib)
# install(TARGETS signal_share DESTINATION lib)

The following is my meta.yml file

{% set org = "Environmental-Seismology" %}
{% set name = "noiseflow" %}
{% set version = "0.0.7a" %}

package:
  name: {{ name|lower }}-split
  version: {{ version }}

source:
  url: https://github.com/{{ org }}/{{ name }}/archive/v{{ version }}.tar.gz
  sha256: d997fddfaf8072469f26f807ac7d3bd4b66398d70abe865d95b2232076d63759

build:
  number: 0
  skip: True  # [python_impl == 'pypy']
  


outputs:

  - name: {{ name|lower }}
    script: build_osx.sh  # [osx]
    script: build_linux.sh  # [linux]
    script: build_win.bat  # [win]

    requirements:
      build:
        - "{{ compiler('cxx') }}"
        - python >=3.9                              
        - poetry-core
        - setuptools                          
        - cmake
        - numpy 
        - pip  
        - make  # [unix]
        - ninja    
        - llvm-openmp  
           

      host:
        - pip
        - python >=3.9
        - numpy 
        - fftw
        - pybind11
        - xtensor
        - xtensor-python
        - xsimd =8.0.5
        - xtl 
        - xtensor-blas
        - llvm-openmp  

      run:
        - numpy >=1.25.0
        - scipy =1.10.1
        - matplotlib-base
        - h5py
        - joblib 
        - tqdm 
        - obspy
        - faker
        - openblas

      test:
        imports:
          - noiseflow
          - noiseflow.cc
          - noiseflow.signal

about:
  home: https://github.com/{{ org }}/{{ name }}
  summary: An ambient noise package.
  license: Apache-2.0
  license_file: LICENSE

extra:
  recipe-maintainers:
    - OUCyf

Problem

I am really confused about the error ERROR Needed DSO lib/libopenblas.0.dylib found in ['libopenblas'].. but ['libopenblas'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely), maybe it’s xtensor-blas problem?I have tried many solutions, it still not work.

The project is in https://github.com/Environmental-Seismology/noiseflow, the staged-recipes is in https://github.com/Environmental-Seismology/staged-recipes. I am sorry to put an issue here, somehow it’s conda-forge problem, but I’m really helpless.

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Comments: 16 (8 by maintainers)

Most upvoted comments

Thanks for your help!!! Maybe I can publish my first package in conda-forge in a few days. Thanks!

It works! Add the libopenblas is right, thanks @tdegeus. BTW I have learned a lot from some of your packages and code style, so much thanks!