spack: Installation error: Python can't find _iconv symbol

I have been trying to build packages with spack on mac os Mojave for awhile now and I have been struggling. I’ve simplified my software stack as much as possible to reproduce one of the issues I keep seeing.

I created an environment (called nexus, here) with the following commands:

$ spack env create nexus nexus.yaml

Where nexus.yaml is:

spack:
  specs:
    - python

From here, I activate, concretize and install:

$ spack env activate nexus
$ spack concretize
==> Warning: clang@11.0.0-apple cannot build optimized binaries for "skylake". Using best target possible: "x86_64"
==> Warning: clang@11.0.0-apple cannot build optimized binaries for "skylake". Using best target possible: "x86_64"
==> Concretized python
 -   hpjs6nz  python@3.7.4%clang@11.0.0-apple+bz2+ctypes+dbm+lzma~nis~optimizations patches=210df3f28cde02a8135b58cc4168e70ab91dbf9097359d05938f1e2843875e57 +pic+pyexpat+pythoncmd+readline~shared+sqlite3+ssl~tix~tkinter~ucs4~uuid+zlib arch=darwin-mojave-x86_64
 -   ysmjykg      ^bzip2@1.0.8%clang@11.0.0-apple+shared arch=darwin-mojave-x86_64
 -   yn2yzi3          ^diffutils@3.7%clang@11.0.0-apple arch=darwin-mojave-x86_64
 -   jg6ekuh              ^libiconv@1.16%clang@11.0.0-apple arch=darwin-mojave-x86_64
 -   l4gxq46      ^expat@2.2.9%clang@11.0.0-apple~libbsd arch=darwin-mojave-x86_64
 -   3t4dwsv      ^gdbm@1.18.1%clang@11.0.0-apple arch=darwin-mojave-x86_64
 -   h2bjtz2          ^readline@8.0%clang@11.0.0-apple arch=darwin-mojave-x86_64
 -   zbgxvxi              ^ncurses@6.1%clang@11.0.0-apple~symlinks~termlib arch=darwin-mojave-x86_64
 -   f3libie                  ^pkgconf@1.6.3%clang@11.0.0-apple arch=darwin-mojave-x86_64
 -   msyeqht      ^gettext@0.20.1%clang@11.0.0-apple+bzip2+curses+git~libunistring+libxml2+tar+xz arch=darwin-mojave-x86_64
 -   w354vrb          ^libxml2@2.9.9%clang@11.0.0-apple~python arch=darwin-mojave-x86_64
 -   b7n722h              ^xz@5.2.4%clang@11.0.0-apple arch=darwin-mojave-x86_64
 -   xj36ejk              ^zlib@1.2.11%clang@11.0.0-apple+optimize+pic+shared arch=darwin-mojave-x86_64
 -   2e6sifw          ^tar@1.32%clang@11.0.0-apple arch=darwin-mojave-x86_64
 -   7yg7qah      ^libffi@3.2.1%clang@11.0.0-apple arch=darwin-mojave-x86_64
 -   ugvknwc      ^openssl@1.1.1d%clang@11.0.0-apple+systemcerts arch=darwin-mojave-x86_64
 -   l6fc2zw          ^perl@5.30.0%clang@11.0.0-apple+cpanm+shared+threads arch=darwin-mojave-x86_64
 -   hsmq6ou      ^sqlite@3.30.1%clang@11.0.0-apple~column_metadata+fts~functions~rtree arch=darwin-mojave-x86_64

==> Updating view at /Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view

The install process reported no errors whatsoever. Ran for awhile and then afterwards, python executable is active and available, but spack itself is broken:

  File "/Users/corey.adams/spack/spack/bin/spack", line 63, in <module>
    import spack.main  # noqa
  File "/Users/corey.adams/spack/spack/lib/spack/spack/main.py", line 29, in <module>
    import spack.architecture
  File "/Users/corey.adams/spack/spack/lib/spack/spack/architecture.py", line 69, in <module>
    import spack.compiler
  File "/Users/corey.adams/spack/spack/lib/spack/spack/compiler.py", line 19, in <module>
    import spack.spec
  File "/Users/corey.adams/spack/spack/lib/spack/spack/spec.py", line 105, in <module>
    import spack.repo
  File "/Users/corey.adams/spack/spack/lib/spack/spack/repo.py", line 36, in <module>
    import spack.caches
  File "/Users/corey.adams/spack/spack/lib/spack/spack/caches.py", line 15, in <module>
    import spack.fetch_strategy
  File "/Users/corey.adams/spack/spack/lib/spack/spack/fetch_strategy.py", line 42, in <module>
    import spack.util.web as web_util
  File "/Users/corey.adams/spack/spack/lib/spack/spack/util/web.py", line 18, in <module>
    from six.moves.urllib.request import urlopen, Request
  File "/Users/corey.adams/spack/spack/lib/spack/external/six.py", line 92, in __get__
    result = self._resolve()
  File "/Users/corey.adams/spack/spack/lib/spack/external/six.py", line 160, in _resolve
    module = _import_module(self.mod)
  File "/Users/corey.adams/spack/spack/lib/spack/external/six.py", line 82, in _import_module
    __import__(name)
  File "/Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view/lib/python3.7/urllib/request.py", line 2583, in <module>
    from _scproxy import _get_proxy_settings, _get_proxies
ImportError: dlopen(/Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view/lib/python3.7/lib-dynload/_scproxy.cpython-37m-darwin.so, 2): Symbol not found: _iconv
  Referenced from: /usr/lib/libcups.2.dylib
  Expected in: /Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view/lib/libiconv.2.dylib
 in /usr/lib/libcups.2.dylib

I am using clang11:

$ spack config get compilers
compilers:
- compiler:
    spec: clang@11.0.0-apple
    paths:
      cc: /usr/bin/clang
      cxx: /usr/bin/clang++
      f77:
      fc:
    flags: {}
    operating_system: mojave
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []

I note a few things here: The library that causes the issue doesn’t directly link against libcups:

$ otool -L /Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view/lib/python3.7/lib-dynload/_scproxy.cpython-37m-darwin.so
/Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view/lib/python3.7/lib-dynload/_scproxy.cpython-37m-darwin.so:
	/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 1061.40.2)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1673.126.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)

I am not enough of a linking expert to track down where the dependency is coming from. I also note that the library it finds for libcups that spack has build does not link to a library that has _iconv symbols defined:

corey.adams@cadams2 : ~/NEXT
$ otool -L /Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view/lib/libiconv.2.dylib
/Users/corey.adams/spack/spack/var/spack/environments/nexus/.spack-env/view/lib/libiconv.2.dylib:
	/Users/corey.adams/spack/spack/opt/spack/darwin-mojave-x86_64/clang-11.0.0-apple/libiconv-1.16-jg6ekuhbklculym7pyjpx6l5jetym7wp/lib/libiconv.2.dylib (compatibility version 9.0.0, current version 9.1.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)

corey.adams@cadams2 : ~/NEXT
$ nm /Users/corey.adams/spack/spack/opt/spack/darwin-mojave-x86_64/clang-11.0.0-apple/libiconv-1.16-jg6ekuhbklculym7pyjpx6l5jetym7wp/lib/libiconv.2.dylib | grep _iconv
0000000000003360 T _iconv_canonicalize

However, the library available at system level does:

$ otool -L /usr/lib/libcups.2.dylib
/usr/lib/libcups.2.dylib:
	/usr/lib/libcups.2.dylib (compatibility version 2.0.0, current version 2.13.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)
	/System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos (compatibility version 5.0.0, current version 6.0.0)
	/System/Library/Frameworks/GSS.framework/Versions/A/GSS (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1575.17.0)
	/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 963.255.3)
	/usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 58286.255.3)
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)

corey.adams@cadams2 : ~/NEXT
$ nm /usr/lib/libiconv.2.dylib | grep _iconv
0000000000002360 T _iconv
00000000000f0da0 s _iconv_2VersionNumber
00000000000f0d70 s _iconv_2VersionString
000000000000267a T _iconv_canonicalize
0000000000002382 T _iconv_close
0000000000001049 T _iconv_open
000000000000238f T _iconvctl
0000000000002488 T _iconvlist

I don’t have anything special on LD_LIBRARY_PATH - it is unset before spack env activate.

Somehow, spack seems to be building a package that links against the system version of libcups (which in turn links to system iconv) while at run time it is picking up it’s own libcups (and it’s own libiconv) which is missing symbols that it needs.

This is macOS mojave.

Thoughts?

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 32 (32 by maintainers)

Most upvoted comments

Sorry for the bump, but ran into this myself today. Just for info, I have spack installed under my home area (so theoretically everything it installs is outside SIP) on a Catalina system with SIP enabled. Simplest case to reproduce is pretty much as reported:

$ spack env create foo
$ spack env activate -p foo
[foo] $ spack install cmake
...
[foo] $ cmake --version
cmake version 3.15.4

CMake suite maintained and supported by Kitware (kitware.com/cmake).
[foo] $ spack install python
...
[foo] $ cmake --version
dyld: Symbol not found: _iconv
  Referenced from: /usr/lib/libarchive.2.dylib
  Expected in: /Users/<ME>/Software/spack/var/spack/environments/foo/.spack-env/view/lib/libiconv.2.dylib
 in /usr/lib/libarchive.2.dylib
zsh: abort      cmake --version
[foo] $ DYLD_LIBRARY_PATH="" cmake --version
cmake version 3.15.4

CMake suite maintained and supported by Kitware (kitware.com/cmake).
[foo] $

@gartung’s Python test python3 -c "from _scproxy import _get_proxy_settings, _get_proxies also fails in the main environment, but works on unsetting DYLD_LIBRARY_PATH.

So the remedy for me seems, at this stage, to be unsetting DYLD_LIBRARY_PATH in the environment. I’ll see if I can trace the behaviour of SIP vs DYLD_LIBRARY_PATH a bit better though.

I am concretizing and building with the following packages.yaml to avoid building libiconv

packages:
  perl:
    buildable: False
    paths:
      perl: /usr
  tar:
    buildable: False
    paths:
      tar: /usr
  libiconv:
    buildable: False
    paths:
      libiconv: /usr

I tried; It didn’t work.

I couldn’t directly remove scons from the environment, because spack itself will not run with the environment active. I removed the environment, uninstalled scons, and then recreated the environment with only python. It reinstalled python + it’s dependencies, but I get the same error.