LightGBM: LightGBM is incompatible with libomp 12 and 13 on macOS
Description
LightGBM cannot be used to fit multiple models in parallel using threads with the latest libomp. On 2014 MacBook Pro:
OMP: Error #13: Assertion failure at kmp_runtime.cpp(3689).
OMP: Hint Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see https://bugs.llvm.org/.
[1] 17358 abort python myfile2.py
On 2019 MacBook Pro:
OMP: Error #131: Thread identifier invalid.
Setting nthreads=1 doesn’t solve the problem.
Reproducible example
from lightgbm import LGBMClassifier
import numpy as np
from concurrent.futures import ThreadPoolExecutor
x = np.random.random((200, 4))
y = x.sum(axis=1) >= 2
def myfunc(a=7):
test = LGBMClassifier().fit(x, y)
print(test.predict(x))
with ThreadPoolExecutor(20) as tpe:
print(list(tpe.map(myfunc, range(20))))
Environment info
LightGBM version or commit hash: 3.1.1 (with python 3.7.3) and 3.2.1 (with python 3.9.4)
brew install libomp
libomp: stable 12.0.0 (bottled)
LLVM's OpenMP runtime library
https://openmp.llvm.org/
/usr/local/Cellar/libomp/12.0.0 (9 files, 1.5MB) *
Poured from bottle on 2021-04-26 at 11:06:26
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/libomp.rb
Command(s) you used to install LightGBM
pip install lightgbm
Additional Comments
The code does work with libomp version 11. Downgraded using
wget https://raw.githubusercontent.com/Homebrew/homebrew-core/fb8323f2b170bd4ae97e1bac9bf3e2983af3fdb0/Formula/libomp.rb
brew unlink libomp
brew install libomp.rb
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 23
- Comments: 26
Commits related to this issue
- MacPorts/libomp: pinning to version 11 because of macOS-specific bug Bug https://bugs.llvm.org/show_bug.cgi?id=50579 is not yet fixed in version 13, see also https://github.com/microsoft/LightGBM/issu... — committed to NatronGitHub/Natron by devernay 3 years ago
- macports: keep the pin on libomp 11.1.0, although issue closed on LLVM side llvm hav moved to github issues, here's the corresponding github issue: https://github.com/llvm/llvm-project/issues/49923 wh... — committed to NatronGitHub/Natron by devernay 2 years ago
New major LLVM version 13 was released 4 days ago: https://github.com/llvm/llvm-project/releases/tag/llvmorg-13.0.0. And the latest Homebrew libomp formulae is pointing to that version now: https://github.com/Homebrew/homebrew-core/blob/4343aee9c28d28b9ed3208b5933df54c29b916fb/Formula/libomp.rb#L4.
But unfortunately this bug (https://github.com/microsoft/LightGBM/issues/4229#issuecomment-855839996) wasn’t fixed in stable 13 release. I’m going to reflect this fact in the issue’s title.
I have the same issue and did some testing: basically
libomp12.0 works with Catalina, but results in segfault for Big Sur. Downgrading to 11.1 worked for Big Sur (tested on Intel MBP and M1 MBP via rosetta2)One suggested workaround in the upstream bug report without downgrading
libompversion is to set some environmental variables:https://bugs.llvm.org/show_bug.cgi?id=50579#c1
Moving the import statement
import lightgbm as lgbto line 1 in my file actually got rid of the error. As per suggestion from https://github.com/dmlc/xgboost/issues/7039#issuecomment-860910066libomp version
/usr/local/Cellar/libomp/12.0.0Error dump when loading booster model. Putting it out here in case it is useful:
On XGBoost we are also facing issues with updated libomp. It has internal error: https://github.com/dmlc/xgboost/pull/6912/checks?check_run_id=2459890229
FYI (not sure if this is common knowledge yet): when developing on LightGBM on Apple Silicon, I never turned off OpenMP but used
gccinstead ofclangfor compilation (for me, that was):This fixed any problems I had 😅
Unfortunately, LLVM developers haven’t fixed this bug (https://github.com/microsoft/LightGBM/issues/4229#issuecomment-855839996) in
12.0.1release.Upstream bug report: https://bugs.llvm.org/show_bug.cgi?id=50579.
Thanks @borchero , that’s helpful!
Looking into this a bit today, I also think that some of these failures might not actually be about incompatibility with particular versions of OpenMP, but rather related to #5106.
Fixing the search paths embedded in
lib_lightgbm.soon macOS might eliminate some of these cases where programs segfault because multiple versions of libomp have been loaded.details (click me)
Tried the following today on my intel mac:
lib_lightgbmNotice that even though I was building in an active
condaenvironment, it found Homebrew’s OpenMP,/usr/local/opt/libomp/lib/libomp.dylib.This segfaults, I think because it’s finding the
llvm-openmpfromconda:Running with some debugging stuff set… it looks like that’s exactly what’s happening. 2 versions of OpenMP are being loaded.
Looking a bit more closely, it seems that
scikit-learncomes with ansklearn/utils/_openmp_helpers.cpython-311-darwin.sowhich has an RPATH entry that causesconda’slibomp.dylibto be loaded.Patching out
lib_lightgbm’s corresponding entry so that it will end up not loading a different version, the example runs without segfaulting.Just stopping here for now to post my notes. I’ll continue working on this.
I found tonight that upgrading to the latest
libompshipped by Homebrew (v15.0.6), I was able to compile LightGBM, build the Python package, and run all of its tests without issue on my macbook (Intel chip, macOS 12.2.1).@jameslamb We need to be very careful with embedding libomp library in our wheels:
MacBook Air (M1, 2020) running macOS Monterey version 12.5.1.
Trying to fit
LightGBMModelgives me[1] 36565 segmentation fault python test_lightgbm.py.libomp:
I tried:
But it gives me: