TileDB-SOMA: [Bug] Illegal instruction despite building from source
Describe the bug
Calling cellxgene_census.open_soma() and/or python3 -m pytest TileDB-SOMA/apis/python/tests causes an Illegal instruction error. I’ve read through the issues here & see that building from source usually seems to be the resolution, but that hasn’t worked for me (see below). I’ve also tried building from release 1.12 & release 1.11, but got the same error at the same steps in the workflow.
To Reproduce
Creating environment
> mamba env create -n cellxgene_census -f ./environment.yml
> mamba activate cellxgene_census
> uv pip install -r ./requirements.txt
> mamba list
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
_sysroot_linux-64_curr_repodata_hack 3 h69a702a_16 conda-forge
aiobotocore 2.13.1 pypi_0 pypi
aiohappyeyeballs 2.3.5 pypi_0 pypi
aiohttp 3.10.2 pypi_0 pypi
aioitertools 0.11.0 pypi_0 pypi
aiosignal 1.3.1 pypi_0 pypi
anndata 0.10.8 pypi_0 pypi
array-api-compat 1.8 pypi_0 pypi
async-timeout 4.0.3 pypi_0 pypi
attrs 24.2.0 pypi_0 pypi
binutils 2.40 h4852527_7 conda-forge
binutils_impl_linux-64 2.40 ha1999f0_7 conda-forge
binutils_linux-64 2.40 hb3c18ed_0 conda-forge
botocore 1.34.131 pypi_0 pypi
bzip2 1.0.8 h4bc722e_7 conda-forge
c-ares 1.32.3 h4bc722e_0 conda-forge
c-compiler 1.7.0 hd590300_1 conda-forge
ca-certificates 2024.7.4 hbcca054_0 conda-forge
cellxgene-census 1.15.0 pypi_0 pypi
certifi 2024.7.4 pypi_0 pypi
charset-normalizer 3.3.2 pypi_0 pypi
clang 18.1.8 default_h9e3a008_1 conda-forge
clang-18 18.1.8 default_hf981a13_1 conda-forge
cmake 3.30.2 hf8c4bd3_0 conda-forge
contourpy 1.2.1 pypi_0 pypi
cxx-compiler 1.7.0 h00ab1b0_1 conda-forge
cycler 0.12.1 pypi_0 pypi
exceptiongroup 1.2.2 pypi_0 pypi
fonttools 4.53.1 pypi_0 pypi
frozenlist 1.4.1 pypi_0 pypi
fsspec 2024.6.1 pypi_0 pypi
gcc 12.4.0 h236703b_0 conda-forge
gcc_impl_linux-64 12.4.0 hb2e57f8_0 conda-forge
gcc_linux-64 12.4.0 h6b7512a_0 conda-forge
gxx 12.4.0 h236703b_0 conda-forge
gxx_impl_linux-64 12.4.0 h613a52c_0 conda-forge
gxx_linux-64 12.4.0 h8489865_0 conda-forge
h5py 3.11.0 pypi_0 pypi
icu 75.1 he02047a_0 conda-forge
idna 3.7 pypi_0 pypi
iniconfig 2.0.0 pypi_0 pypi
jmespath 1.0.1 pypi_0 pypi
joblib 1.4.2 pypi_0 pypi
kernel-headers_linux-64 3.10.0 h4a8ded7_16 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.4.5 pypi_0 pypi
krb5 1.21.3 h659f571_0 conda-forge
ld_impl_linux-64 2.40 hf3520f5_7 conda-forge
legacy-api-wrap 1.4 pypi_0 pypi
libclang-cpp18.1 18.1.8 default_hf981a13_1 conda-forge
libcurl 8.9.1 hdb1bdb2_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 hd590300_2 conda-forge
libexpat 2.6.2 h59595ed_0 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-devel_linux-64 12.4.0 ha4f9413_100 conda-forge
libgcc-ng 14.1.0 h77fa898_0 conda-forge
libgomp 14.1.0 h77fa898_0 conda-forge
libiconv 1.17 hd590300_2 conda-forge
libllvm18 18.1.8 h8b73ec9_2 conda-forge
libnghttp2 1.58.0 h47da74e_1 conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libsanitizer 12.4.0 h46f95d5_0 conda-forge
libsqlite 3.46.0 hde9e2c9_0 conda-forge
libssh2 1.11.0 h0841786_0 conda-forge
libstdcxx-devel_linux-64 12.4.0 ha4f9413_100 conda-forge
libstdcxx-ng 14.1.0 hc0a3c3a_0 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libuv 1.48.0 hd590300_0 conda-forge
libxml2 2.12.7 he7c6b58_4 conda-forge
libzlib 1.3.1 h4ab18f5_1 conda-forge
llvmlite 0.43.0 pypi_0 pypi
make 4.3 hd18ef5c_1 conda-forge
matplotlib 3.9.1.post1 pypi_0 pypi
multidict 6.0.5 pypi_0 pypi
natsort 8.4.0 pypi_0 pypi
ncurses 6.5 h59595ed_0 conda-forge
networkx 3.3 pypi_0 pypi
numba 0.60.0 pypi_0 pypi
numpy 1.26.4 pypi_0 pypi
openssl 3.3.1 h4bc722e_2 conda-forge
packaging 24.1 pypi_0 pypi
pandas 2.2.2 pypi_0 pypi
patsy 0.5.6 pypi_0 pypi
pillow 10.4.0 pypi_0 pypi
pip 24.2 pyhd8ed1ab_0 conda-forge
pluggy 1.5.0 pypi_0 pypi
pyarrow 17.0.0 pypi_0 pypi
pyarrow-hotfix 0.6 pypi_0 pypi
pynndescent 0.5.13 pypi_0 pypi
pyparsing 3.1.2 pypi_0 pypi
pytest 8.3.2 pypi_0 pypi
python 3.10.12 hd12c33a_0_cpython conda-forge
python-dateutil 2.9.0.post0 pypi_0 pypi
pytz 2024.1 pypi_0 pypi
readline 8.2 h8228510_1 conda-forge
requests 2.32.3 pypi_0 pypi
rhash 1.4.4 hd590300_0 conda-forge
s3fs 2024.6.1 pypi_0 pypi
scanpy 1.10.2 pypi_0 pypi
scikit-learn 1.5.1 pypi_0 pypi
scipy 1.14.0 pypi_0 pypi
seaborn 0.13.2 pypi_0 pypi
session-info 1.0.0 pypi_0 pypi
setuptools 72.1.0 pyhd8ed1ab_0 conda-forge
six 1.16.0 pypi_0 pypi
somacore 1.0.11 pypi_0 pypi
statsmodels 0.14.2 pypi_0 pypi
stdlib-list 0.10.0 pypi_0 pypi
sysroot_linux-64 2.17 h4a8ded7_16 conda-forge
threadpoolctl 3.5.0 pypi_0 pypi
tiledb 0.29.1 pypi_0 pypi
tiledbsoma 1.11.4 pypi_0 pypi
tk 8.6.13 noxft_h4845f30_101 conda-forge
tomli 2.0.1 pypi_0 pypi
tqdm 4.66.5 pypi_0 pypi
typeguard 4.3.0 pypi_0 pypi
typing-extensions 4.12.2 pypi_0 pypi
tzdata 2024.1 pypi_0 pypi
umap-learn 0.5.6 pypi_0 pypi
urllib3 2.2.2 pypi_0 pypi
uv 0.2.34 pypi_0 pypi
wheel 0.44.0 pyhd8ed1ab_0 conda-forge
wrapt 1.16.0 pypi_0 pypi
xz 5.2.6 h166bdaf_0 conda-forge
yarl 1.9.4 pypi_0 pypi
zstd 1.5.6 ha6fb4c9_0 conda-forge
Building TileDB-SOMA
> git clone https://github.com/single-cell-data/TileDB-SOMA.git # main
> python3 -m uv pip install TileDB-SOMA/apis/python/
Resolved 43 packages in 18.32s
Built tiledbsoma @ file:///scicomp/groups-pure/OID/NCEZID/DSR/BCFB/by-project/scrnagpt/chris/bin/cellxgene_census/TileDB-SOMA/apis/python
Prepared 1 package in 5m 27s
Uninstalled 3 packages in 649ms
Installed 3 packages in 357ms
- somacore==1.0.11
+ somacore==1.0.13
- tiledb==0.29.1
+ tiledb==0.31.1
- tiledbsoma==1.11.4
+ tiledbsoma==1.12.0rc0.post72.dev619176400 (from file:///scicomp/groups-pure/OID/NCEZID/DSR/BCFB/by-project/scrnagpt/chris/bin/cellxgene_census/TileDB-SOMA/apis/python)
Running pytest
> python3 -m pytest TileDB-SOMA/apis/python/tests
========================================================================================================== test session starts ===========================================================================================================
platform linux -- Python 3.10.12, pytest-8.3.2, pluggy-1.5.0
rootdir: /scicomp/groups-pure/OID/NCEZID/DSR/BCFB/by-project/scrnagpt/chris/bin/cellxgene_census/TileDB-SOMA/apis/python
configfile: pyproject.toml
plugins: typeguard-4.3.0
collected 1766 items
TileDB-SOMA/apis/python/tests/test_basic_anndata_io.py Fatal Python error: Illegal instruction
Current thread 0x00007fc1e69fa740 (most recent call first):
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/tiledbsoma/_collection.py", line 130 in create
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/tiledbsoma/io/ingest.py", line 1007 in _create_or_open_collection
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/tiledbsoma/io/ingest.py", line 505 in from_anndata
File "/scicomp/groups-pure/OID/NCEZID/DSR/BCFB/by-project/scrnagpt/chris/bin/cellxgene_census/TileDB-SOMA/apis/python/tests/test_basic_anndata_io.py", line 104 in test_import_anndata
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/_pytest/python.py", line 159 in pytest_pyfunc_call
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/_pytest/python.py", line 1627 in runtest
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/_pytest/runner.py", line 174 in pytest_runtest_call
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/_pytest/runner.py", line 242 in <lambda>
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/_pytest/runner.py", line 241 in call_and_report
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/_pytest/runner.py", line 132 in runtestprotocol
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/_pytest/runner.py", line 113 in pytest_runtest_protocol
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/_pytest/main.py", line 362 in pytest_runtestloop
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/_pytest/main.py", line 337 in _main
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/_pytest/main.py", line 283 in wrap_session
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/_pytest/main.py", line 330 in pytest_cmdline_main
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/_pytest/config/__init__.py", line 175 in main
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/_pytest/config/__init__.py", line 201 in console_main
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/site-packages/pytest/__main__.py", line 9 in <module>
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/runpy.py", line 86 in _run_code
File "/scicomp/home-pure/kxc7/.conda/envs/cellxgene_census/lib/python3.10/runpy.py", line 196 in _run_module_as_main
Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pyarrow.lib, h5py._errors, h5py.defs, h5py._objects, h5py.h5, h5py.utils, h5py.h5t, h5py.h5s, h5py.h5ac, h5py.h5p, h5py.h5r, h5py._proxy, h5py._conv, h5py.h5z, h5py.h5a, h5py.h5d, h5py.h5ds, h5py.h5g, h5py.h5i, h5py.h5f, h5py.h5fd, h5py.h5pl, h5py.h5o, h5py.h5l, h5py._selector, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, tiledb.libtiledb (total: 114)
Illegal instruction
Attempting open_soma()
Python 3.10.12 | packaged by conda-forge | (main, Jun 23 2023, 22:40:32) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cellxgene_census
>>> soma = cellxgene_census.open_soma()
The "stable" release is currently 2024-07-01. Specify 'census_version="2024-07-01"' in future calls to open_soma() to ensure data consistency.
Illegal instruction
Versions (please complete the following information):
- TileDB-SOMA version:
>>> tiledbsoma.show_package_versions()
tiledbsoma.__version__ 1.12.0rc0.post72.dev619176400
TileDB core version (libtiledbsoma) 2.25.0
python version 3.10.12.final.0
OS version Linux 3.10.0-1160.119.1.el7.x86_64
- Language and language version:
Python 3.10.12 - OS:
> cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
- CPU:
> lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 1
Core(s) per socket: 8
Socket(s): 4
NUMA node(s): 4
Vendor ID: GenuineIntel
CPU family: 6
Model: 62
Model name: Intel(R) Xeon(R) CPU E5-4627 v2 @ 3.30GHz
Stepping: 4
CPU MHz: 3256.091
CPU max MHz: 3600.0000
CPU min MHz: 1200.0000
BogoMIPS: 6599.64
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 16384K
NUMA node0 CPU(s): 0,4,8,12,16,20,24,28
NUMA node1 CPU(s): 1,5,9,13,17,21,25,29
NUMA node2 CPU(s): 2,6,10,14,18,22,26,30
NUMA node3 CPU(s): 3,7,11,15,19,23,27,31
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 39 (25 by maintainers)
OK this is somehow not being a build from source then … if it were it would be producing more output, and taking more time
I’ll need to investigate this on a noavx2 system – thank you very much for the experiments @chrish935 …
Here’s the gist from building with a more verbose
cmake: https://gist.github.com/chrish935/7c3cd3e48651b54c9fd7460a9ec5adb9.Turns out that I forgot the
-voption in mypip installcall.Note: This is with grabbing a fresh clone of the repo, so we’re doing this without changing the download URL in
TileDB-SOMA/libtiledbsoma/cmake/Modules/FindTileDB_EP.cmakeor replacing thelibtiledb.so.2.25files. Also worth noting that this substantially increased build time compared to when the download URL was changed.@chrish935 thanks for your time on this.
What you’ve done so far should have worked, so in the words of Sherlock Holmes, when it’s not any of thing things it can be, it has to be one of the things it can’t be.
Something is not what it seems:
noavx2tar file isn’t what it seems, or/usr/local/library is happening in a way I am not seeing, orAVX2instruction, but some other –gdbcan perhaps reveal thatIn any event, we at TileDB need to get a system set up similar to this and debug it. I appreciate your generous time involvement but I think we’re at a next step of rolling up the sleeves internally.
@johnkerl Doesn’t it need an updated sha1 for the changed file too?
Still no dice.
Running pytest
Maybe this bit would need a ‘have avx2 or not’ generalization like the one I show above as this will pull the ‘assume AVX2’ blob when we also have one for ‘-noavx2’ in each release build:
https://github.com/single-cell-data/TileDB-SOMA/blob/6b74aa5aa4994777fffa23df1f2b9fcace95491f/libtiledbsoma/cmake/Modules/FindTileDB_EP.cmake#L92-L95
OK let me ask around internally for a third pair of 👀 – thanks @chrish935 🙏
Doesn’t appear to be, no.
Sure thing.
Running pytest