pudl: Segmentation fault when importing PUDL installed from conda-forge
Describe the bug
I just re-created a new conda environment where I installed PUDL from conda-forge. When I import PUDL in that environment, it results in a segmentation fault.
The last time I installed PUDL from conda-forge, this didn’t happen, though I can’t say for sure when that would have been.
Bug Severity
How badly is this bug affecting you?
- Low: When installed using pip, this doesn’t seem to happen so I can work around it.
To Reproduce
Create the new environment, install PUDL, activate the environment, and launch a python console.
mamba create --name test python=3.10 catalystcoop.pudl
mamba activate test
python
In the console, import PUDL
import pudl
The console then produces this output and exits.
/Users/aengel/mambaforge/envs/test/lib/python3.10/site-packages/pudl/analysis/spatial.py:7: UserWarning: Shapely 2.0 is installed, but because PyGEOS is also installed, GeoPandas will still use PyGEOS by default for now. To force to use and test Shapely 2.0, you have to set the environment variable USE_PYGEOS=0. You can do this before starting the Python process, or in your code before importing geopandas:
import os
os.environ['USE_PYGEOS'] = '0'
import geopandas
In a future release, GeoPandas will switch to using Shapely by default. If you are using PyGEOS directly (calling PyGEOS functions on geometries from GeoPandas), this will then stop working and you are encouraged to migrate from PyGEOS to Shapely 2.0 (https://shapely.readthedocs.io/en/latest/migration_pygeos.html).
import geopandas as gpd
zsh: segmentation fault python
Software Environment?
- Operating System. MacOS 12.6.3
- Python 3.10.9 | packaged by conda-forge | (main, Feb 2 2023, 20:26:08) [Clang 14.0.6 ] on darwin
- How did you install PUDL?
mamba create --name test python=3.10 catalystcoop.pudl
Full set of packages and versions:
# packages in environment at /Users/aengel/mambaforge/envs/test:
#
# Name Version Build Channel
addfips 0.4.0 pyhd8ed1ab_1 conda-forge
aiohttp 3.8.4 py310h8e9501a_0 conda-forge
aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge
arelle-release 2.2.4 pyhd8ed1ab_1 conda-forge
arrow-cpp 10.0.1 hce30654_17_cpu conda-forge
async-timeout 4.0.2 pyhd8ed1ab_0 conda-forge
attrs 22.2.0 pyh71513ae_0 conda-forge
aws-c-auth 0.6.26 h0f48f9c_0 conda-forge
aws-c-cal 0.5.21 h134b2d6_2 conda-forge
aws-c-common 0.8.14 h1a8c8d9_0 conda-forge
aws-c-compression 0.2.16 h59b28b2_5 conda-forge
aws-c-event-stream 0.2.20 ha370d96_4 conda-forge
aws-c-http 0.7.5 hca61f3c_5 conda-forge
aws-c-io 0.13.19 ha2443f3_1 conda-forge
aws-c-mqtt 0.8.6 hcfee220_11 conda-forge
aws-c-s3 0.2.7 h050892e_0 conda-forge
aws-c-sdkutils 0.1.8 h59b28b2_0 conda-forge
aws-checksums 0.1.14 h59b28b2_5 conda-forge
aws-crt-cpp 0.19.8 h0287f93_11 conda-forge
aws-sdk-cpp 1.10.57 h3daea3c_8 conda-forge
blinker 1.5 pyhd8ed1ab_0 conda-forge
blosc 1.21.3 h1d6ff8b_0 conda-forge
bokeh 2.4.3 pyhd8ed1ab_3 conda-forge
boost-cpp 1.78.0 h1cb353e_1 conda-forge
boto3 1.26.94 pyhd8ed1ab_0 conda-forge
botocore 1.29.94 pyhd8ed1ab_0 conda-forge
branca 0.6.0 pyhd8ed1ab_0 conda-forge
brotli 1.0.9 h1a8c8d9_8 conda-forge
brotli-bin 1.0.9 h1a8c8d9_8 conda-forge
brotlipy 0.7.0 py310h8e9501a_1005 conda-forge
bzip2 1.0.8 h3422bc3_4 conda-forge
c-ares 1.18.1 h3422bc3_0 conda-forge
ca-certificates 2022.12.7 h4653dfc_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
cachetools 5.3.0 pyhd8ed1ab_0 conda-forge
cairo 1.16.0 h73a0509_1014 conda-forge
catalystcoop.dbfread 3.0.0 py_0 conda-forge
catalystcoop.ferc-xbrl-extractor 0.6.1 pyhd8ed1ab_1 conda-forge
catalystcoop.pudl 2022.11.30 pyhd8ed1ab_0 conda-forge
cchardet 2.1.7 py310h0f1eb42_4 conda-forge
certifi 2022.12.7 pyhd8ed1ab_0 conda-forge
cffi 1.15.1 py310h2399d43_3 conda-forge
cfitsio 4.2.0 h2f961c4_0 conda-forge
chardet 5.1.0 py310hbe9552e_0 conda-forge
charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge
click 8.1.3 unix_pyhd8ed1ab_2 conda-forge
click-plugins 1.1.1 py_0 conda-forge
cligj 0.7.2 pyhd8ed1ab_1 conda-forge
cloudpickle 2.2.1 pyhd8ed1ab_0 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
coloredlogs 15.0.1 pyhd8ed1ab_3 conda-forge
commonmark 0.9.1 py_0 conda-forge
contourpy 1.0.7 py310h2887b22_0 conda-forge
cryptography 39.0.2 py310hfc83b78_0 conda-forge
curl 7.88.1 h9049daf_1 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
cytoolz 0.12.0 py310h8e9501a_1 conda-forge
dask 2022.11.1 pyhd8ed1ab_0 conda-forge
dask-core 2022.11.1 pyhd8ed1ab_0 conda-forge
dataclasses 0.8 pyhc8e2a94_3 conda-forge
datapackage 1.15.2 pyh44b312d_0 conda-forge
decorator 5.1.1 pyhd8ed1ab_0 conda-forge
distributed 2022.11.1 pyhd8ed1ab_0 conda-forge
dnspython 2.3.0 pyhd8ed1ab_0 conda-forge
email-validator 1.3.1 pyhd8ed1ab_0 conda-forge
et_xmlfile 1.1.0 pyhd8ed1ab_0 conda-forge
expat 2.5.0 hb7217d7_0 conda-forge
fiona 1.9.2 py310h2e51ddd_0 conda-forge
folium 0.14.0 pyhd8ed1ab_0 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.14.2 h82840c6_0 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
fonttools 4.39.2 py310h8e9501a_0 conda-forge
freetype 2.12.1 hd633e50_1 conda-forge
freexl 1.0.6 h1a8c8d9_1 conda-forge
frictionless 4.40.8 pyh6c4a22f_0 conda-forge
frozenlist 1.3.3 py310h8e9501a_0 conda-forge
fsspec 2022.11.0 pyhd8ed1ab_0 conda-forge
future 0.18.3 pyhd8ed1ab_0 conda-forge
gcsfs 2022.11.0 pyhd8ed1ab_0 conda-forge
gdal 3.6.3 py310h0b34360_0 conda-forge
geopandas 0.12.2 pyhd8ed1ab_0 conda-forge
geopandas-base 0.12.2 pyha770c72_0 conda-forge
geos 3.11.1 hb7217d7_0 conda-forge
geotiff 1.7.1 h435296a_7 conda-forge
gettext 0.21.1 h0186832_0 conda-forge
gflags 2.2.2 hc88da5d_1004 conda-forge
giflib 5.2.1 h1a8c8d9_3 conda-forge
glog 0.6.0 h6da1cb0_0 conda-forge
google-api-core 2.11.0 pyhd8ed1ab_0 conda-forge
google-auth 2.16.2 pyh1a96a4e_0 conda-forge
google-auth-oauthlib 1.0.0 pyhd8ed1ab_0 conda-forge
google-cloud-core 2.3.2 pyhd8ed1ab_0 conda-forge
google-cloud-storage 2.7.0 pyh1a96a4e_0 conda-forge
google-crc32c 1.1.2 py310he58995c_4 conda-forge
google-resumable-media 2.4.1 pyhd8ed1ab_0 conda-forge
googleapis-common-protos 1.57.1 pyhd8ed1ab_0 conda-forge
greenlet 2.0.2 py310h0f1eb42_0 conda-forge
grpcio 1.52.1 py310h0f1eb42_1 conda-forge
h3-py 3.7.4 py310h0f1eb42_1 conda-forge
hdf4 4.2.15 h8111dcc_6 conda-forge
hdf5 1.12.2 nompi_ha7af310_101 conda-forge
heapdict 1.0.1 py_0 conda-forge
humanfriendly 10.0 py310hbe9552e_4 conda-forge
icu 70.1 h6b3803e_0 conda-forge
idna 3.4 pyhd8ed1ab_0 conda-forge
ijson 3.2.0.post0 pyhd8ed1ab_0 conda-forge
importlib-metadata 6.1.0 pyha770c72_0 conda-forge
importlib_resources 5.12.0 pyhd8ed1ab_0 conda-forge
iniconfig 2.0.0 pyhd8ed1ab_0 conda-forge
isodate 0.6.1 pyhd8ed1ab_0 conda-forge
jinja2 3.1.2 pyhd8ed1ab_1 conda-forge
jmespath 1.0.1 pyhd8ed1ab_0 conda-forge
joblib 1.2.0 pyhd8ed1ab_0 conda-forge
json-c 0.16 hc449e50_0 conda-forge
jsonlines 3.1.0 pyhd8ed1ab_0 conda-forge
jsonpointer 2.0 py_0 conda-forge
jsonschema 4.17.3 pyhd8ed1ab_0 conda-forge
kealib 1.5.0 hfd766a6_0 conda-forge
kiwisolver 1.4.4 py310h2887b22_1 conda-forge
krb5 1.20.1 h69eda48_0 conda-forge
lcms2 2.15 hd835a16_1 conda-forge
lerc 4.0.0 h9a09cb3_0 conda-forge
libabseil 20230125.0 cxx17_hb7217d7_1 conda-forge
libaec 1.0.6 hb7217d7_1 conda-forge
libarrow 10.0.1 hdcd110a_17_cpu conda-forge
libblas 3.9.0 16_osxarm64_openblas conda-forge
libbrotlicommon 1.0.9 h1a8c8d9_8 conda-forge
libbrotlidec 1.0.9 h1a8c8d9_8 conda-forge
libbrotlienc 1.0.9 h1a8c8d9_8 conda-forge
libcblas 3.9.0 16_osxarm64_openblas conda-forge
libcrc32c 1.1.2 hbdafb3b_0 conda-forge
libcurl 7.88.1 h9049daf_1 conda-forge
libcxx 15.0.7 h75e25f2_0 conda-forge
libdeflate 1.17 h1a8c8d9_0 conda-forge
libedit 3.1.20191231 hc8eb9b7_2 conda-forge
libev 4.33 h642e427_1 conda-forge
libevent 2.1.10 h7673551_4 conda-forge
libffi 3.4.2 h3422bc3_5 conda-forge
libgdal 3.6.3 h2c41457_0 conda-forge
libgfortran 5.0.0 12_2_0_hd922786_31 conda-forge
libgfortran5 12.2.0 h0eea778_31 conda-forge
libglib 2.74.1 h4646484_1 conda-forge
libgoogle-cloud 2.8.0 h7206bc8_1 conda-forge
libgrpc 1.52.1 he98ff75_1 conda-forge
libiconv 1.17 he4db4b2_0 conda-forge
libjpeg-turbo 2.1.5.1 h1a8c8d9_0 conda-forge
libkml 1.3.0 h41464e4_1015 conda-forge
liblapack 3.9.0 16_osxarm64_openblas conda-forge
libllvm11 11.1.0 hfa12f05_5 conda-forge
libnetcdf 4.9.1 nompi_hda66dd9_102 conda-forge
libnghttp2 1.52.0 hae82a92_0 conda-forge
libopenblas 0.3.21 openmp_hc731615_3 conda-forge
libpng 1.6.39 h76d750c_0 conda-forge
libpq 15.2 h1a28acd_0 conda-forge
libprotobuf 3.21.12 hb5ab8b9_0 conda-forge
librttopo 1.1.0 h844f84d_12 conda-forge
libspatialindex 1.9.3 hbdafb3b_4 conda-forge
libspatialite 5.0.1 h14115fc_23 conda-forge
libsqlite 3.40.0 h76d750c_0 conda-forge
libssh2 1.10.0 h7a5bd25_3 conda-forge
libthrift 0.18.1 h6635e49_0 conda-forge
libtiff 4.5.0 hdc14d85_5 conda-forge
libutf8proc 2.8.0 h1a8c8d9_0 conda-forge
libwebp-base 1.3.0 h1a8c8d9_0 conda-forge
libxcb 1.13 h9b22ae9_1004 conda-forge
libxml2 2.10.3 h67585b2_4 conda-forge
libxslt 1.1.37 h1bd8bc4_0 conda-forge
libzip 1.9.2 h76ab92c_1 conda-forge
libzlib 1.2.13 h03a7124_4 conda-forge
linear-tsv 1.1.0 py_1 conda-forge
llvm-openmp 16.0.0 h7cfbb63_0 conda-forge
llvmlite 0.39.1 py310h1e34944_1 conda-forge
locket 1.0.0 pyhd8ed1ab_0 conda-forge
lxml 4.9.2 py310h85b680a_0 conda-forge
lz4 4.3.2 py310ha6df754_0 conda-forge
lz4-c 1.9.4 hb7217d7_0 conda-forge
mapclassify 2.5.0 pyhd8ed1ab_1 conda-forge
marko 1.3.0 pyhd8ed1ab_0 conda-forge
markupsafe 2.1.2 py310h8e9501a_0 conda-forge
matplotlib-base 3.6.3 py310h78c5c2f_0 conda-forge
msgpack-python 1.0.5 py310h2887b22_0 conda-forge
multidict 6.0.4 py310h8e9501a_0 conda-forge
munch 2.5.0 py_0 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
ncurses 6.3 h07bb92c_1 conda-forge
networkx 2.8.8 pyhd8ed1ab_0 conda-forge
nspr 4.35 hb7217d7_0 conda-forge
nss 3.89 h789eff7_0 conda-forge
numba 0.56.4 py310h3124f1e_0 conda-forge
numpy 1.23.5 py310h5d7c261_0 conda-forge
oauthlib 3.2.2 pyhd8ed1ab_0 conda-forge
openjpeg 2.5.0 hbc2ba62_2 conda-forge
openpyxl 3.1.1 py310h8e9501a_0 conda-forge
openssl 3.1.0 h03a7124_0 conda-forge
orc 1.8.3 hef0d403_0 conda-forge
packaging 23.0 pyhd8ed1ab_0 conda-forge
pandas 1.4.4 py310h2b830bf_0 conda-forge
parquet-cpp 1.5.1 2 conda-forge
partd 1.3.0 pyhd8ed1ab_0 conda-forge
pcre2 10.40 hb34f9b4_0 conda-forge
petl 1.7.12 pyhd8ed1ab_0 conda-forge
pillow 9.4.0 py310h07496d3_2 conda-forge
pip 23.0.1 pyhd8ed1ab_0 conda-forge
pixman 0.40.0 h27ca646_0 conda-forge
pkgutil-resolve-name 1.3.10 pyhd8ed1ab_0 conda-forge
pluggy 1.0.0 pyhd8ed1ab_5 conda-forge
poppler 23.03.0 h15f6d88_1 conda-forge
poppler-data 0.4.12 hd8ed1ab_0 conda-forge
postgresql 15.2 h45c140d_0 conda-forge
proj 9.1.1 h13f728c_2 conda-forge
protobuf 4.21.12 py310h0f1eb42_0 conda-forge
psutil 5.9.4 py310h8e9501a_0 conda-forge
pthread-stubs 0.4 h27ca646_1001 conda-forge
py 1.11.0 pyh6c4a22f_0 conda-forge
pyarrow 10.0.1 py310h89f3c6b_17_cpu conda-forge
pyasn1 0.4.8 py_0 conda-forge
pyasn1-modules 0.2.7 py_0 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pydantic 1.10.6 py310h8e9501a_0 conda-forge
pygeos 0.13 py310h9356385_2 conda-forge
pygments 2.14.0 pyhd8ed1ab_0 conda-forge
pyjwt 2.6.0 pyhd8ed1ab_0 conda-forge
pyopenssl 23.0.0 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge
pyproj 3.4.1 py310h9862ddf_1 conda-forge
pyrsistent 0.19.3 py310h8e9501a_0 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
pytest 6.2.5 py310hbe9552e_3 conda-forge
python 3.10.9 h3ba56d0_0_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-slugify 8.0.1 pyhd8ed1ab_1 conda-forge
python-snappy 0.6.1 py310h7f31a72_0 conda-forge
python_abi 3.10 3_cp310 conda-forge
pytz 2022.7.1 pyhd8ed1ab_0 conda-forge
pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge
pyyaml 6.0 py310h8e9501a_5 conda-forge
re2 2023.02.02 hb7217d7_0 conda-forge
readline 8.1.2 h46ed386_0 conda-forge
regex 2022.10.31 py310h8e9501a_0 conda-forge
requests 2.28.2 pyhd8ed1ab_0 conda-forge
requests-oauthlib 1.3.1 pyhd8ed1ab_0 conda-forge
rfc3986 2.0.0 pyhd8ed1ab_0 conda-forge
rich 12.6.0 pyhd8ed1ab_0 conda-forge
rsa 4.9 pyhd8ed1ab_0 conda-forge
rtree 1.0.1 py310ha3239f5_1 conda-forge
s3transfer 0.6.0 pyhd8ed1ab_0 conda-forge
scikit-learn 1.1.3 py310ha00a7cd_1 conda-forge
scipy 1.9.3 py310ha0d8a01_2 conda-forge
setuptools 67.6.0 pyhd8ed1ab_0 conda-forge
shapely 2.0.1 py310h9356385_0 conda-forge
shellingham 1.5.1 pyhd8ed1ab_0 conda-forge
simpleeval 0.9.13 py310h8e9501a_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
snappy 1.1.10 h17c5cce_0 conda-forge
sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge
sqlalchemy 1.4.44 py310h8e9501a_0 conda-forge
sqlite 3.40.0 h2229b38_0 conda-forge
stringcase 1.2.0 py_0 conda-forge
tableschema 1.19.3 pyh9f0ad1d_0 conda-forge
tabulate 0.9.0 pyhd8ed1ab_1 conda-forge
tabulator 1.53.5 pyhd8ed1ab_0 conda-forge
tblib 1.7.0 pyhd8ed1ab_0 conda-forge
text-unidecode 1.3 py_0 conda-forge
threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge
tiledb 2.13.2 h9bd36d0_0 conda-forge
timezonefinder 6.0.2 pyhd8ed1ab_0 conda-forge
tk 8.6.12 he1e0b03_0 conda-forge
toml 0.10.2 pyhd8ed1ab_0 conda-forge
toolz 0.12.0 pyhd8ed1ab_0 conda-forge
tornado 6.1 py310hf8d0d8f_3 conda-forge
typer 0.7.0 pyhd8ed1ab_0 conda-forge
typing-extensions 4.5.0 hd8ed1ab_0 conda-forge
typing_extensions 4.5.0 pyha770c72_0 conda-forge
tzcode 2022g h1a8c8d9_0 conda-forge
tzdata 2022g h191b570_0 conda-forge
unicodecsv 0.14.1 py_1 conda-forge
unicodedata2 15.0.0 py310h8e9501a_0 conda-forge
unidecode 1.3.6 pyhd8ed1ab_0 conda-forge
urllib3 1.26.15 pyhd8ed1ab_0 conda-forge
validators 0.20.0 pyhd8ed1ab_0 conda-forge
wheel 0.40.0 pyhd8ed1ab_0 conda-forge
xerces-c 3.2.4 h627aa08_1 conda-forge
xlrd 2.0.1 pyhd8ed1ab_3 conda-forge
xlsxwriter 3.0.9 pyhd8ed1ab_0 conda-forge
xorg-libxau 1.0.9 h27ca646_0 conda-forge
xorg-libxdmcp 1.1.3 h27ca646_0 conda-forge
xyzservices 2023.2.0 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h57fd34a_0 conda-forge
yaml 0.2.5 h3422bc3_2 conda-forge
yarl 1.8.2 py310h8e9501a_0 conda-forge
zict 2.2.0 pyhd8ed1ab_0 conda-forge
zipp 3.15.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 h03a7124_4 conda-forge
zstd 1.5.2 hf913c23_6 conda-forge
Additional context
Add any other context about the problem here.
======== edit from @jdangerx
### Scope
- [ ] this segfault doesn't happen anymore.
### Next steps
- [ ] see if we can downgrade the h3 dependency in our conda-forge package?
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 22 (21 by maintainers)
Commits related to this issue
- #2426 Pin dependancies to avoid segfault on apple silicon Signed-off-by: Nelson Auner <nelson@auner.org> — committed to nelsonauner/pudl by nelsonauner a year ago
- #2426 Pin dependancies to avoid segfault on apple silicon Signed-off-by: Nelson Auner <nelson@auner.org> — committed to nelsonauner/pudl by nelsonauner a year ago
- Quick fix to #2426 based on last release — committed to nelsonauner/pudl by nelsonauner a year ago
(also I have a PR (#2479) open right now to move all our project metadata / build stuff into
pyproject.tomland get rid ofsetup.pyfinally.This should hopefully be fixed by v2022.11.30.post1 which I will get up on
conda-forgesome time tomorrow (once their bot has picked up the new PyPI release)We have some annoying to build dependencies that need C/C++/Fortran extensions (e.g. geospatial stuff) and some non-python dependencies that
condacan satisfy without needing to resort to using Docker or other heavier weight environment management systems (e.g. the snappy compression library, SQLite binaries) and it’s also nice to use the conda supplied Python rather than hope that the user has the right version of Python available on their system. At least in development we’ve primarily usedcondafor environment virtualization, with the PUDL package getting installed within that environment bypip.Seems like it’s probably a bunch of dependency futzing. We’ve considered using conda-lock to lock exact versions of all of our dependencies so this kind of drift after the fact can’t happen, if that’s something you feel like exploring!
Replicated on a new M2 Macbook air, will play around with this a bit…
We’d like to work on this but not necessarily blow up our whole sprint working on this.
Scope
Out of scope:
Next steps