xtensor: Some errors when submit pull requests to conda-forge using xtensor and xtensor-blas
Greetings,
Recently I developed a package using xtensor as C++ backend and numpy as a python interface, which is usually like @tdegeus done for some of his packages. I also used the xtensor-blas in package, and everything is good in my local computer and github actions for pypi publishment, I also tried conda build .
in my computer. But when I tried to submit the PR to conda-forge, I got the error like followings:
{} is not a Mach-O file
Can't parse the binary at the index #0
{} is not a Mach-O file
Can't parse the binary at the index #1
INFO: sysroot: '/Applications/Xcode_13.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/' files: '['usr/share/man/mann/zero.n', 'usr/share/man/mann/yencode.n', 'usr/share/man/mann/yaml.n', 'usr/share/man/mann/xsxp.n']'
ERROR (noiseflow,lib/python3.10/site-packages/noiseflow/lib/signal_share.cpython-310-darwin.so): Needed DSO lib/libopenblas.0.dylib found in ['libopenblas']
ERROR (noiseflow,lib/python3.10/site-packages/noiseflow/lib/signal_share.cpython-310-darwin.so): .. but ['libopenblas'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
INFO (noiseflow,lib/python3.10/site-packages/noiseflow/lib/signal_share.cpython-310-darwin.so): Needed DSO /Applications/Xcode_13.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/lib/libSystem.B.dylib found in $SYSROOT
INFO (noiseflow,lib/python3.10/site-packages/noiseflow/lib/signal_share.cpython-310-darwin.so): Needed DSO lib/libomp.dylib found in conda-forge::llvm-openmp-15.0.7-h61d9ccf_0
INFO (noiseflow,lib/python3.10/site-packages/noiseflow/lib/signal_share.cpython-310-darwin.so): Needed DSO lib/libc++.1.dylib found in conda-forge::libcxx-16.0.6-hd57cbcb_0
ERROR (noiseflow,lib/python3.10/site-packages/noiseflow/lib/cc_share.cpython-310-darwin.so): Needed DSO lib/libopenblas.0.dylib found in ['libopenblas']
ERROR (noiseflow,lib/python3.10/site-packages/noiseflow/lib/cc_share.cpython-310-darwin.so): .. but ['libopenblas'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
INFO (noiseflow,lib/python3.10/site-packages/noiseflow/lib/cc_share.cpython-310-darwin.so): Needed DSO /Applications/Xcode_13.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/lib/libSystem.B.dylib found in $SYSROOT
INFO (noiseflow,lib/python3.10/site-packages/noiseflow/lib/cc_share.cpython-310-darwin.so): Needed DSO lib/libfftw3.3.dylib found in conda-forge::fftw-3.3.10-nompi_h4fa670e_108
INFO (noiseflow,lib/python3.10/site-packages/noiseflow/lib/cc_share.cpython-310-darwin.so): Needed DSO lib/libfftw3f.3.dylib found in conda-forge::fftw-3.3.10-nompi_h4fa670e_108
INFO (noiseflow,lib/python3.10/site-packages/noiseflow/lib/cc_share.cpython-310-darwin.so): Needed DSO lib/libomp.dylib found in conda-forge::llvm-openmp-15.0.7-h61d9ccf_0
INFO (noiseflow,lib/python3.10/site-packages/noiseflow/lib/cc_share.cpython-310-darwin.so): Needed DSO lib/libc++.1.dylib found in conda-forge::libcxx-16.0.6-hd57cbcb_0
WARNING (noiseflow): run-exports library package conda-forge::numpy-1.21.6-py310hb608527_0 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
Traceback (most recent call last):
File "/Users/runner/work/1/s/.ci_support/build_all.py", line 218, in <module>
build_all(os.path.join(root_dir, "recipes"), args.arch)
File "/Users/runner/work/1/s/.ci_support/build_all.py", line 118, in build_all
build_folders(recipes_dir, folders, arch, channel_urls)
File "/Users/runner/work/1/s/.ci_support/build_all.py", line 174, in build_folders
conda_build.api.build([recipe], config=get_config(arch, channel_urls))
File "/Users/runner/mambaforge/lib/python3.10/site-packages/conda_build/api.py", line 253, in build
return build_tree(
File "/Users/runner/mambaforge/lib/python3.10/site-packages/conda_build/build.py", line 3799, in build_tree
packages_from_this = build(
File "/Users/runner/mambaforge/lib/python3.10/site-packages/conda_build/build.py", line 2875, in build
newly_built_packages = bundlers[pkg_type](output_d, m, env, stats)
File "/Users/runner/mambaforge/lib/python3.10/site-packages/conda_build/build.py", line 2002, in bundle_conda
files = post_process_files(metadata, initial_files)
File "/Users/runner/mambaforge/lib/python3.10/site-packages/conda_build/build.py", line 1813, in post_process_files
post_build(m, new_files, build_python=python)
File "/Users/runner/mambaforge/lib/python3.10/site-packages/conda_build/post.py", line 1721, in post_build
check_overlinking(m, files, host_prefix)
File "/Users/runner/mambaforge/lib/python3.10/site-packages/conda_build/post.py", line 1605, in check_overlinking
return check_overlinking_impl(
File "/Users/runner/mambaforge/lib/python3.10/site-packages/conda_build/post.py", line 1576, in check_overlinking_impl
raise OverLinkingError(overlinking_errors)
conda_build.exceptions.OverLinkingError: overlinking check failed
[" ERROR (noiseflow,lib/python3.10/site-packages/noiseflow/lib/signal_share.cpython-310-darwin.so): .. but ['libopenblas'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)", " ERROR (noiseflow,lib/python3.10/site-packages/noiseflow/lib/cc_share.cpython-310-darwin.so): .. but ['libopenblas'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)"]
##[error]Bash exited with code '1'.
Finishing: Run OSX build
And the following is my cmake file:
###################################################################
# Optimization Parameters
###################################################################
cmake_minimum_required(VERSION 3.18)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_FLAGS "-Wall -g -fPIC -O3")
set(XTENSOR_USE_XSIMD 1) # set xtensor to use xsimd for SIMD optimization
set(CMAKE_PREFIX_PATH $ENV{CONDA_PREFIX}) # set conda env path, so that cmake can find those libraries in conda env
###################################################################
# Project Info & Compiler Detection
###################################################################
project(noiseflow_share VERSION 0.1.0 LANGUAGES CXX)
message("Check C++ compiler: ${CMAKE_CXX_COMPILER_ID}")
if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
if (MINGW)
message("Using MinGW-w64")
set(COMPILER "MinGW-w64")
else()
message("Using GCC")
set(COMPILER "GCC")
endif()
elseif ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "Clang")
message("Using Clang")
set(COMPILER "Clang")
elseif ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "AppleClang")
message("Using AppleClang")
set(COMPILER "AppleClang")
elseif ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "MSVC")
message("Using MSVC")
set(COMPILER "MSVC")
elseif ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "Intel")
message("Using Intel Compiler")
set(COMPILER "Intel")
elseif ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "IntelLLVM")
message("Using Intel oneAPI Compiler")
set(COMPILER "Intel oneAPI")
else()
message(FATAL_ERROR "Unrecognized compiler: ${CMAKE_CXX_COMPILER_ID}")
endif()
###################################################################
# Find Libraries
###################################################################
### [a]. use find_package for numpy
find_package(Python REQUIRED COMPONENTS Interpreter Development NumPy)
### [b]. use execute_process for numpy
# execute_process(
# COMMAND "${PYTHON_EXECUTABLE}"
# -c "import numpy; print(numpy.get_include())"
# OUTPUT_VARIABLE NumPy_INCLUDE_DIRS
# OUTPUT_STRIP_TRAILING_WHITESPACE
# )
### OpenMP, check if the BREW_OPENMP environment variable is set (clang doesn't bring openmp, you need to install it by brew, and find_package can't find it, you need to set the PATH as below. refer to https://www.kancloud.cn/csyangbinbin/cmake-cookbook1/2157938 https://iscinumpy.gitlab.io/post/omp-on-high-sierra/)
if(DEFINED ENV{BREW_OPENMP})
if($ENV{BREW_OPENMP} EQUAL 1)
message(STATUS "Searching for OpenMP in brew path")
execute_process(COMMAND brew --prefix OUTPUT_VARIABLE BREW_PREFIX OUTPUT_STRIP_TRAILING_WHITESPACE)
set(OpenMP_CXX_FLAGS "-Xpreprocessor -fopenmp -I${BREW_PREFIX}/opt/libomp/include")
set(OpenMP_CXX_LIB_NAMES "omp")
set(OpenMP_omp_LIBRARY "${BREW_PREFIX}/opt/libomp/lib")
find_package(OpenMP REQUIRED)
else()
message(STATUS "Searching for OpenMP in default path")
find_package(OpenMP REQUIRED)
endif()
else()
message(STATUS "BREW_OPENMP not set, searching for OpenMP in default path")
find_package(OpenMP REQUIRED)
endif()
### pybind11 xtensor xtl xsimd xtensor-blas xtensor-python
find_package(pybind11 REQUIRED CONFIG)
find_package(xtensor REQUIRED)
find_package(xtl REQUIRED)
find_package(xsimd REQUIRED)
find_package(xtensor-blas REQUIRED)
find_package(xtensor-python REQUIRED)
### xtensor-fftw
set(FFTW_USE_FLOAT ON CACHE BOOL "Use float precision") # set fftw3 to use float precision
set(FFTW_USE_DOUBLE ON CACHE BOOL "Use double precision") # set fftw3 to use double precision
set(FFTW_USE_LONG_DOUBLE OFF CACHE BOOL "Use long double precision") # set fftw3 to not use long double precision
include_directories("./extern/xtensor-fftw/include")
### FFTW3 using FindFFTW.cmake (note this command will seach some default path to find fftw3, for example the conda env path)
set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_CURRENT_SOURCE_DIR}/extern/xtensor-fftw/cmake/Modules/findFFTW") # use FindFFTW.cmake to find fftw3
find_package(FFTW REQUIRED COMPONENTS ${REQUIRE_FLOAT_LIB} ${REQUIRE_DOUBLE_LIB} ${REQUIRE_LONG_DOUBLE_LIB})
if(FFTW_FOUND)
message(STATUS "FFTW Found!")
endif()
### KFR
include_directories("./extern/kfr/include")
###################################################################
# Target -- Python Package Module
###################################################################
pybind11_add_module(cc_share ${CMAKE_SOURCE_DIR}/noiseflow/cc/src/pybind11.cpp)
target_include_directories(cc_share PRIVATE ${CMAKE_SOURCE_DIR}/noiseflow/cc/src ${CMAKE_SOURCE_DIR}/noiseflow/cc/include)
target_link_libraries(cc_share PUBLIC ${BLAS_LIBRARIES} ${LAPACK_LIBRARIES} ${FFTW_LIBRARIES} pybind11::module xtensor xtensor::optimize xtensor::use_xsimd xtensor-blas xtensor-python Python::NumPy OpenMP::OpenMP_CXX)
pybind11_add_module(signal_share ${CMAKE_SOURCE_DIR}/noiseflow/signal/src/pybind11.cpp)
target_include_directories(signal_share PRIVATE ${CMAKE_SOURCE_DIR}/noiseflow/signal/src ${CMAKE_SOURCE_DIR}/noiseflow/signal/include)
target_link_libraries(signal_share PUBLIC ${BLAS_LIBRARIES} ${LAPACK_LIBRARIES} ${FFTW_LIBRARIES} pybind11::module xtensor xtensor::optimize xtensor::use_xsimd xtensor-blas xtensor-python Python::NumPy OpenMP::OpenMP_CXX)
### setup.py will install the module to the python site-packages
# install(TARGETS cc_share DESTINATION lib)
# install(TARGETS signal_share DESTINATION lib)
The following is my meta.yml file
{% set org = "Environmental-Seismology" %}
{% set name = "noiseflow" %}
{% set version = "0.0.7a" %}
package:
name: {{ name|lower }}-split
version: {{ version }}
source:
url: https://github.com/{{ org }}/{{ name }}/archive/v{{ version }}.tar.gz
sha256: d997fddfaf8072469f26f807ac7d3bd4b66398d70abe865d95b2232076d63759
build:
number: 0
skip: True # [python_impl == 'pypy']
outputs:
- name: {{ name|lower }}
script: build_osx.sh # [osx]
script: build_linux.sh # [linux]
script: build_win.bat # [win]
requirements:
build:
- "{{ compiler('cxx') }}"
- python >=3.9
- poetry-core
- setuptools
- cmake
- numpy
- pip
- make # [unix]
- ninja
- llvm-openmp
host:
- pip
- python >=3.9
- numpy
- fftw
- pybind11
- xtensor
- xtensor-python
- xsimd =8.0.5
- xtl
- xtensor-blas
- llvm-openmp
run:
- numpy >=1.25.0
- scipy =1.10.1
- matplotlib-base
- h5py
- joblib
- tqdm
- obspy
- faker
- openblas
test:
imports:
- noiseflow
- noiseflow.cc
- noiseflow.signal
about:
home: https://github.com/{{ org }}/{{ name }}
summary: An ambient noise package.
license: Apache-2.0
license_file: LICENSE
extra:
recipe-maintainers:
- OUCyf
Problem
I am really confused about the error ERROR Needed DSO lib/libopenblas.0.dylib found in ['libopenblas'].. but ['libopenblas'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
, maybe it’s xtensor-blas
problem?I have tried many solutions, it still not work.
The project is in https://github.com/Environmental-Seismology/noiseflow, the staged-recipes is in https://github.com/Environmental-Seismology/staged-recipes. I am sorry to put an issue here, somehow it’s conda-forge problem, but I’m really helpless.
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 16 (8 by maintainers)
Thanks for your help!!! Maybe I can publish my first package in
conda-forge
in a few days. Thanks!It works! Add the
libopenblas
is right, thanks @tdegeus. BTW I have learned a lot from some of your packages and code style, so much thanks!