spack: Broken Python3 on OpenSUSE, lib-dynload not correctly linked
Steps to reproduce the issue
Install py-pip (dependency of amdlibm) with AOCC@4.1.0:
$ spack install py-pip %aocc
==> Warning: config:terminal_title has been replaced by install_status and is ignored
[+] /home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/bzip2-1.0.8-q4pqsh3rxddsvyeuoc4umnhp4fbq3fcc
[+] /home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/libmd-1.0.4-lp7x5tmamdnni6ccsick632krdghn4rl
[+] /home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/ncurses-6.4-twp7yf3grnnzv3whbfr3fmccaml2s5ut
[+] /home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/libiconv-1.17-xo2d4zaqgk4zq54eeic2sd6jxjhbxvuy
[+] /home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/xz-5.4.1-huprvmsxzdvqjt4wiymsravg6ktpretm
[+] /home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/zlib-ng-2.1.3-52s7eq32xbgfb2ecxnytevwzjfextkwo
[+] /home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/zstd-1.5.5-knlbullsxnlliav3ee24osy35foxd26b
[+] /home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/libffi-3.4.4-lqb7gtbxs7ugk2qj5z3eompbhiipra7k
[+] /home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/libxcrypt-4.4.35-vyjg3panmbji2dy6oxo2iwvmvbm6tmr7
[+] /home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/util-linux-uuid-2.38.1-p7rabtpzawmfnfxq5izkas34umhvtppj
[+] /home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/libbsd-0.11.7-7cwjeplhcnehfrravoluexflx6rok44s
[+] /home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/readline-8.2-so6tfuvqypsh7axma25ftc3kkroykzb4
[+] /home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/pigz-2.7-rh55xrdkycsoipc7dlak6bgx2j3pmk3b
[+] /home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/openssl-3.1.2-jufrpmzekubxplnjtioe3n4xri4zpjcr
[+] /home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/libxml2-2.10.3-kcosc37y57aqexfzhvtjghvo2lxiwlve
[+] /home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/expat-2.5.0-n3z6tlhhz3va2d2ympka6pz4jexp2sra
[+] /home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/sqlite-3.42.0-ioreb5arslq7q7iy6m4bmyc6o4juiju4
[+] /home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/gdbm-1.23-4rs5sb5jp4ohcu42kxqaqjcuijwm6wqe
[+] /home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/tar-1.34-dztpp3xiyjxd4dthstdihajji24s7v4x
[+] /home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/gettext-0.21.1-lvmu6twonklvtbhalkpccroswh6xtze2
[+] /home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/python-3.10.12-fvclniug6pdjl475gjvkkcfjmh4cpnyp
==> Installing py-pip-23.1.2-6j6wfgdu4xv4x6z5r56c4bdck3ekjs77 [22/22]
==> No binary for py-pip-23.1.2-6j6wfgdu4xv4x6z5r56c4bdck3ekjs77 found: installing from source
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
==> Using cached archive: /var/tmp/yanghaoyu/spack-cache/_source-cache/archive/3e/3ef6ac33239e4027d9a5598a381b9d30880a1477e50039db2eac6e8a8f6d1b18
==> No patches needed for py-pip
==> py-pip: Executing phase: 'install'
==> Error: ProcessError: Command exited with status 1:
'/home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/python-3.10.12-fvclniug6pdjl475gjvkkcfjmh4cpnyp/bin/python3.10' '/tmp/yanghaoyu/spack-stage/spack-stage-py-pip-23.1.2-6j6wfgdu4xv4x6z5r56c4bdck3ekjs77/spack-src/pip-23.1.2-py3-none-any.whl/pip' '-vvv' '--no-input' '--no-cache-dir' '--disable-pip-version-check' 'install' '--no-deps' '--ignore-installed' '--no-build-isolation' '--no-warn-script-location' '--no-index' '--prefix=/home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/py-pip-23.1.2-6j6wfgdu4xv4x6z5r56c4bdck3ekjs77' '/tmp/yanghaoyu/spack-stage/spack-stage-py-pip-23.1.2-6j6wfgdu4xv4x6z5r56c4bdck3ekjs77/spack-src/pip-23.1.2-py3-none-any.whl'
See build log for details:
/tmp/yanghaoyu/spack-stage/spack-stage-py-pip-23.1.2-6j6wfgdu4xv4x6z5r56c4bdck3ekjs77/spack-build-out.txt
$ spack spec -I py-pip
==> Warning: config:terminal_title has been replaced by install_status and is ignored
Input spec
--------------------------------
- py-pip
Concretized
--------------------------------
- py-pip@23.1.2%aocc@4.1.0 build_system=generic arch=linux-opensusetumbleweed2.38-zen3
[+] ^python@3.10.12%aocc@4.1.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,7d40923,f2fd060 arch=linux-opensusetumbleweed2.38-zen3
[+] ^bzip2@1.0.8%aocc@4.1.0~debug~pic+shared build_system=generic arch=linux-opensusetumbleweed2.38-zen3
[+] ^diffutils@3.9%aocc@4.1.0 build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^expat@2.5.0%aocc@4.1.0+libbsd build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^libbsd@0.11.7%aocc@4.1.0 build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^libmd@1.0.4%aocc@4.1.0 build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^gdbm@1.23%aocc@4.1.0 build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^gettext@0.21.1%aocc@4.1.0+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^libiconv@1.17%aocc@4.1.0 build_system=autotools libs=shared,static arch=linux-opensusetumbleweed2.38-zen3
[+] ^libxml2@2.10.3%aocc@4.1.0~python build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^tar@1.34%aocc@4.1.0 build_system=autotools zip=pigz arch=linux-opensusetumbleweed2.38-zen3
[+] ^pigz@2.7%aocc@4.1.0 build_system=makefile arch=linux-opensusetumbleweed2.38-zen3
[+] ^zstd@1.5.5%aocc@4.1.0+programs build_system=makefile compression=none libs=shared,static arch=linux-opensusetumbleweed2.38-zen3
[+] ^libffi@3.4.4%aocc@4.1.0 build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^libxcrypt@4.4.35%aocc@4.1.0~obsolete_api build_system=autotools patches=4885da3 arch=linux-opensusetumbleweed2.38-zen3
[+] ^perl@5.38.0%aocc@4.1.0+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=linux-opensusetumbleweed2.38-zen3
[+] ^berkeley-db@18.1.40%aocc@4.1.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-opensusetumbleweed2.38-zen3
[+] ^ncurses@6.4%aocc@4.1.0~symlinks+termlib abi=none build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^openssl@3.1.2%aocc@4.1.0~docs+shared build_system=generic certs=mozilla arch=linux-opensusetumbleweed2.38-zen3
[+] ^ca-certificates-mozilla@2023-05-30%aocc@4.1.0 build_system=generic arch=linux-opensusetumbleweed2.38-zen3
[+] ^pkgconf@1.9.5%aocc@4.1.0 build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^readline@8.2%aocc@4.1.0 build_system=autotools patches=bbf97f1 arch=linux-opensusetumbleweed2.38-zen3
[+] ^sqlite@3.42.0%aocc@4.1.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^util-linux-uuid@2.38.1%aocc@4.1.0 build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^xz@5.4.1%aocc@4.1.0~pic build_system=autotools libs=shared,static arch=linux-opensusetumbleweed2.38-zen3
[+] ^zlib-ng@2.1.3%aocc@4.1.0+compat+opt build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
$ spack spec -I python
==> Warning: config:terminal_title has been replaced by install_status and is ignored
Input spec
--------------------------------
- python
Concretized
--------------------------------
[+] python@3.10.12%aocc@4.1.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,7d40923,f2fd060 arch=linux-opensusetumbleweed2.38-zen3
[+] ^bzip2@1.0.8%aocc@4.1.0~debug~pic+shared build_system=generic arch=linux-opensusetumbleweed2.38-zen3
[+] ^diffutils@3.9%aocc@4.1.0 build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^expat@2.5.0%aocc@4.1.0+libbsd build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^libbsd@0.11.7%aocc@4.1.0 build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^libmd@1.0.4%aocc@4.1.0 build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^gdbm@1.23%aocc@4.1.0 build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^gettext@0.21.1%aocc@4.1.0+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^libiconv@1.17%aocc@4.1.0 build_system=autotools libs=shared,static arch=linux-opensusetumbleweed2.38-zen3
[+] ^libxml2@2.10.3%aocc@4.1.0~python build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^tar@1.34%aocc@4.1.0 build_system=autotools zip=pigz arch=linux-opensusetumbleweed2.38-zen3
[+] ^pigz@2.7%aocc@4.1.0 build_system=makefile arch=linux-opensusetumbleweed2.38-zen3
[+] ^zstd@1.5.5%aocc@4.1.0+programs build_system=makefile compression=none libs=shared,static arch=linux-opensusetumbleweed2.38-zen3
[+] ^libffi@3.4.4%aocc@4.1.0 build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^libxcrypt@4.4.35%aocc@4.1.0~obsolete_api build_system=autotools patches=4885da3 arch=linux-opensusetumbleweed2.38-zen3
[+] ^perl@5.38.0%aocc@4.1.0+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=linux-opensusetumbleweed2.38-zen3
[+] ^berkeley-db@18.1.40%aocc@4.1.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-opensusetumbleweed2.38-zen3
[+] ^ncurses@6.4%aocc@4.1.0~symlinks+termlib abi=none build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^openssl@3.1.2%aocc@4.1.0~docs+shared build_system=generic certs=mozilla arch=linux-opensusetumbleweed2.38-zen3
[+] ^ca-certificates-mozilla@2023-05-30%aocc@4.1.0 build_system=generic arch=linux-opensusetumbleweed2.38-zen3
[+] ^pkgconf@1.9.5%aocc@4.1.0 build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^readline@8.2%aocc@4.1.0 build_system=autotools patches=bbf97f1 arch=linux-opensusetumbleweed2.38-zen3
[+] ^sqlite@3.42.0%aocc@4.1.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^util-linux-uuid@2.38.1%aocc@4.1.0 build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
[+] ^xz@5.4.1%aocc@4.1.0~pic build_system=autotools libs=shared,static arch=linux-opensusetumbleweed2.38-zen3
[+] ^zlib-ng@2.1.3%aocc@4.1.0+compat+opt build_system=autotools arch=linux-opensusetumbleweed2.38-zen3
Error message
Error message
==> Installing py-pip-23.1.2-6j6wfgdu4xv4x6z5r56c4bdck3ekjs77 [22/22]
==> No binary for py-pip-23.1.2-6j6wfgdu4xv4x6z5r56c4bdck3ekjs77 found: installing from source
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
==> Using cached archive: /var/tmp/yanghaoyu/spack-cache/_source-cache/archive/3e/3ef6ac33239e4027d9a5598a381b9d30880a1477e50039db2eac6e8a8f6d1b18
==> No patches needed for py-pip
==> py-pip: Executing phase: 'install'
==> Error: ProcessError: Command exited with status 1:
'/home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/python-3.10.12-fvclniug6pdjl475gjvkkcfjmh4cpnyp/bin/python3.10' '/tmp/yanghaoyu/spack-stage/spack-stage-py-pip-23.1.2-6j6wfgdu4xv4x6z5r56c4bdck3ekjs77/spack-src/pip-23.1.2-py3-none-any.whl/pip' '-vvv' '--no-input' '--no-cache-dir' '--disable-pip-version-check' 'install' '--no-deps' '--ignore-installed' '--no-build-isolation' '--no-warn-script-location' '--no-index' '--prefix=/home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/py-pip-23.1.2-6j6wfgdu4xv4x6z5r56c4bdck3ekjs77' '/tmp/yanghaoyu/spack-stage/spack-stage-py-pip-23.1.2-6j6wfgdu4xv4x6z5r56c4bdck3ekjs77/spack-src/pip-23.1.2-py3-none-any.whl'
See build log for details:
/tmp/yanghaoyu/spack-stage/spack-stage-py-pip-23.1.2-6j6wfgdu4xv4x6z5r56c4bdck3ekjs77/spack-build-out.txt
Build Log
=> py-pip: Executing phase: 'install'
==> [2023-08-21-11:19:51.647349] '/home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/python-3.10.12-fvclniug6pdjl475gjvkkcfjmh4cpnyp/bin/python3.10' '/tmp/yanghaoyu/spack-stage/spack-stage-py-pip-23.1.2-6j6wfgdu4xv4x6z5r56c4bdck3ekjs77/spack-src/pip-23.1.2-py3-none-any.whl/pip' '-vvv' '--no-input' '--no-cache-dir' '--disable-pip-version-check' 'install' '--no-deps' '--ignore-installed' '--no-build-isolation' '--no-warn-script-location' '--no-index' '--prefix=/home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/py-pip-23.1.2-6j6wfgdu4xv4x6z5r56c4bdck3ekjs77' '/tmp/yanghaoyu/spack-stage/spack-stage-py-pip-23.1.2-6j6wfgdu4xv4x6z5r56c4bdck3ekjs77/spack-src/pip-23.1.2-py3-none-any.whl'
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Traceback (most recent call last):
File "<frozen zipimport>", line 577, in _get_decompress_func
ModuleNotFoundError: No module named 'zlib'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<frozen zipimport>", line 625, in _get_data
File "<frozen zipimport>", line 580, in _get_decompress_func
zipimport.ZipImportError: can't decompress data; zlib not available
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/python-3.10.12-fvclniug6pdjl475gjvkkcfjmh4cpnyp/lib/python3.10/runpy.py", line 157, in _get_module_details
code = loader.get_code(mod_name)
File "<frozen zipimport>", line 196, in get_code
File "<frozen zipimport>", line 759, in _get_module_code
File "<frozen zipimport>", line 627, in _get_data
zipimport.ZipImportError: can't decompress data; zlib not available
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/python-3.10.12-fvclniug6pdjl475gjvkkcfjmh4cpnyp/lib/python3.10/runpy.py", line 189, in _run_module_as_main
mod_name, mod_spec, code = _get_main_module_details(_Error)
File "/home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/python-3.10.12-fvclniug6pdjl475gjvkkcfjmh4cpnyp/lib/python3.10/runpy.py", line 238, in _get_main_module_details
return _get_module_details(main_name)
File "/home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/python-3.10.12-fvclniug6pdjl475gjvkkcfjmh4cpnyp/lib/python3.10/runpy.py", line 159, in _get_module_details
raise error(format(e)) from e
ImportError: can't decompress data; zlib not available
Running Python3 Directly
$ which python3
/home/yanghaoyu/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/python-3.10.12-fvclniug6pdjl475gjvkkcfjmh4cpnyp/bin/python3.
$ python3
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python 3.10.12 (main, Aug 21 2023, 10:11:54) [Clang 16.0.3 (CLANG: AOCC_4.1.0-Build#270 2023_07_10)] on linux
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last):
File "/etc/pythonstart", line 7, in <module>
import readline
ModuleNotFoundError: No module named 'readline'
Information on your system
$ spack debug report:
* **Spack:** 0.21.0.dev0 (c9ec5fb9ac565033c7db643f0b51da5dc9315ac0)
* **Python:** 3.11.4
* **Platform:** linux-opensusetumbleweed2.38-zen3
==> Warning: config:terminal_title has been replaced by install_status and is ignored
* **Concretizer:** clingo
Additional information
Python maintainers: @adamjstewart @scheibelp @skosukhin
py-pip maintainers: @adamjstewart @pradyunsg
General information
- I have run
spack debug reportand reported the version of Spack/Python/Platform - I have run
spack maintainers <name-of-the-package>and @mentioned any maintainers - I have uploaded the build log and environment files
- I have searched the issues of this repo and believe this is not a duplicate
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 16 (9 by maintainers)
I can confirm that these change fixes the issue in OpenSUSE 15.6
I managed to reproduce this, but only on a local openSUSE Tumbleweed install, not on an opensuse/tumbleweed container. No idea what is different between the containerized and non-containerized version though… 🤷
Great news! Adding
--with-platlibdir=lib64did seem to fix the issue, noCould not find platform dependent libraries <exec_prefix>error is thrown when running Python directly andspack install py-pipfinished smoothly without any issue as well.Meanwhile it appears to me Spack is at least sort of able to read platform(OS)-dependent info as the installation path of python as
/opt/spack/packages/linux-opensusetumbleweed2.38-zen3/aocc-4.1.0/python-3.10.12-qlzfddaqoqws6zxv2pqgifdlaufkqvfj/seem to contain OS-related infoOpenSUSE Tumbleweed. Would this help fix this OpenSUSE-related issue?Hi @adamjstewart, good news!
My friend @quenpsi pinpointed and addressed this issue. And it seems to have been caused by a ten-year old OpenSUSE-specifc Python issue mentioned here . And could be easily addressed by creating a symlink to
$prefix/lib/python3.X/lib-dynloadfrom$prefix/lib64/python3.X/lib-dynload.If you run:
it will tell you the installation directory of Python compiled with AOCC. In that directory, the file you’re looking for is:
Definitely looks like broken Python. Can you upload the
spack-build-out.txtfor Python?