spack: Clingo fails on Cray

Steps to reproduce the issue

This is after merging PR#22460 https://github.com/spack/spack/pull/22460

I’ve left some comments in that PR that are PR specific.

Effectively, the patch requires that GCC have static libraries (which isn’t always true).

  1. I swapped to a GCC that does have static libs (but it’s provided by a module)
  2. My new GCC (9.2.0) seemed to fail the Spack C++14 flag test - I commented this out in clingo-bootstrap since I know it will succeed (clingo or clingo-bootstrap will now configure and attempt to compile most things)
  3. I discovered my python3.7’s static lib appears to depend on OpenSSL (I got missing symbols that belong to libssl when it linked the final clingo binaries)
  4. I got clingo-bootstrap@spack to compile
  5. I cleaned, editted the config and added the concretizer as clingo
  6. spack spec zlib throws an error about Clingo
  7. I pushed through a build of plain clingo@spack (adding openssl)
  8. Regular clingo gives the same problem Build ENV is Python3.7 (via a module), GCC 9.2.0 (module), Cmake 3.19.6, build machine is a cray that doesn’t have static libs

The verbose version of that list follows (written as I tried things)

Use a GCC with static libs (hurray) - not detected as having C++14 though (boo)

I swapped to a GCC 9.2.0 build (which does have static libraries). This will pass through CMake… but oddly it’s failing in the spack package for clingo, because spack thinks self.compiler.cxx14_flag is non existent for this compiler. C++14 check that fails for GCC 9.2.0

I’m sure Cmake/spack is actually compiling with GCC 9.2.0, because CMake’s compiler detection reports GCC 9.2.0.

I commented out the c++14 check in clingo’s package, and Cmake configures/compiles (mostly)

Onwards to glory

Cmake gets all the way to to nearly the end and is building bin/gring and it dies in a link error. (missing symbols that look like openssl)

If I edit clingo-bootstrap and add a depends_on(openssl) and tack on spec['openssl'].libs.joined() to the ldflags at line 49 (referenced above). I can get clingo installed.

I’m assuming there’s something in my python (or perhaps an undocumented clingo dependence on SSL).

Enable the clingo concretizer

spack load clingo-boostrap

[coe0201@node003 ~]$ clingo --version
clingo version 5.5.0
Address model: 64-bit

libclingo version 5.5.0
Configuration: with Python 3.7.5, without Lua

libclasp version 3.3.5 (libpotassco version 1.1.0)
Configuration: WITH_THREADS=1
Copyright (C) Benjamin Kaufmann

License: The MIT License <https://opensource.org/licenses/MIT>

I then spack config edit config adding

config:
  concretizer: clingo

I found I really need to spack clean -all after this.

Next,

[coe0201@node003 ~]$ spack clean --all
==> Removing all temporary build stages
==> Removing cached downloads
==> Removing install failure marks
==> Removing cached information on repositories
==> Removing python cache files
==> Removing software in "/home/users/coe0201/.spack/bootstrap/store"

[coe0201@node003 ~]$ spack spec zlib
Input spec
--------------------------------
zlib

Concretized
--------------------------------
==> Error: module 'clingo' has no attribute 'Symbol'

try using plain old clingo

make sure spack unload clingo-bootstrap and spack uninstall clingo-bootstrap

  1. spack edit clingo
  2. add depends_on(openssl) and add '-DCMAKE_EXE_LINKER_FLAGS={}'.format(spec[openssl].libs.joined())
  3. could add static gcc/c++ stuff (like in bootstrap) - but didn’t to keep it simple
  4. clingo will now build/install
spack -d --verbose install -j60 clingo@spack %gcc@9.2.0

[coe0201@node003 ~]$ spack load clingo

[coe0201@node003 ~]$ clingo --version
clingo version 5.5.0
Address model: 64-bit

libclingo version 5.5.0
Configuration: with Python 3.7.5, without Lua

libclasp version 3.3.5 (libpotassco version 1.1.0)
Configuration: WITH_THREADS=1
Copyright (C) Benjamin Kaufmann

License: The MIT License <https://opensource.org/licenses/MIT>

spack clean --all

[coe0201@node003 ~]$ spack spec zlib %gcc@9.2.0
Input spec
--------------------------------
zlib%gcc@9.2.0

Concretized
--------------------------------
==> Error: module 'clingo' has no attribute 'Symbol'
commands
Input spec
--------------------------------
zlib%gcc@9.2.0

Concretized
--------------------------------
==> [2021-03-23-01:40:25.637746, 3585632] Reading config file /home/users/coe0201/src/spack/etc/spack/defaults/config.yaml
==> [2021-03-23-01:40:25.671790, 3585632] Reading config file /home/users/coe0201/.spack/config.yaml
Traceback (most recent call last):
  File "/home/users/coe0201/src/spack/bin/spack", line 76, in <module>
    sys.exit(spack.main.main())
  File "/home/users/coe0201/src/spack/lib/spack/spack/main.py", line 768, in main
    return _invoke_command(command, parser, args, unknown)
  File "/home/users/coe0201/src/spack/lib/spack/spack/main.py", line 496, in _invoke_command
    return_val = command(parser, args)
  File "/home/users/coe0201/src/spack/lib/spack/spack/cmd/spec.py", line 102, in spec
    spec.concretize()
  File "/home/users/coe0201/src/spack/lib/spack/spack/spec.py", line 2552, in concretize
    self._new_concretize(tests)
  File "/home/users/coe0201/src/spack/lib/spack/spack/spec.py", line 2513, in _new_concretize
    import spack.solver.asp
  File "/home/users/coe0201/src/spack/lib/spack/spack/solver/asp.py", line 22, in <module>
    clingo_cffi = hasattr(clingo.Symbol, '_rep')
AttributeError: module 'clingo' has no attribute 'Symbol'

cat  /home/users/coe0201/.spack/config.yaml
config:
  concretizer: clingo

@tgamblin (you are the maintainer listed in the package)

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 18 (18 by maintainers)

Most upvoted comments

@alalazo Yes - I think your summary is correct.

I haven’t tried removing gcc from the packages compiler field. (I don’t mind it being there at lower priority)

I’ll mess around with the package I have and try to pin down the spec problems … that should be in separate dialog though.

The gist of that is forcing downstream dependencies to have constraints on their dependencies. E.g.,

package Foo():
  variant(exec_space, values=('openmp','serial','cuda'), default='serial')
  depends_on('libsci+openmp', when='exec_space=openmp')
  depends_on('superlu-dist+openmp^libsci+openmp', when='exec_space=openmp')

The dependency constraint on superlu-dist’s dependency causes all kinds of problems with the default concretizer.

It’s also worth adding the original motivator to this line of effort.

I developed package (fork of Trilinos) that declares several depends_on. Most of the packages will depend on blas and lapack, and I want my Trilinos build’s blas/lapack to be used by the underlying libraries. E.g., superlu-dist should use the same blas/lapack as Trilinos. What I observed, was that because my package (and others) depend_on(blas|lapack) that caused the default spec for available blas/lapack to get added to the concretizer. (it wouldn’t realize that a spec for blas/lapack was provided and it should use it everywhere else as well).

To that end, I was directed to try the new concretizer by Todd.

Lots to follow up on:

packages.yaml is correct

I clipped the remainder of the file (it goes on to declare externals + more package specific things.

[coe0201@redwood ~]$ cat ~/.spack/packages.yaml
packages:
  all:
    # to make it work:
    compiler:
      - cce@11.0.3
      - gcc@9.2.0
    # what I had originally:
    compiler: [ cce@11.0.3 ]
    providers:
      mpi: [mvapich2]
      blas: [cray-libsci]
      lapack: [cray-libsci]
      fftw-api: [cray-fftw]
    target: [any]

I suppose I could try removing the compiler all together… I am generally building for a specific version of CCE. I need to bump up to 11.0.4, but I’ll cross that bridge after the spack issues.

I suspect that spack’s bootstrap process should ignore any constraints on compilers from packages - those make sense for the user, but they don’t if you need to do something special inside spack.

GCC / CCE etc… are compilers defined in compilers.yaml (gcc) or cray/compilers.yaml (cce) - (both are external via modules)

multiple installs of re2c

That makes sense. I was actually suprised that spack spec zlib %gcc@9.2.0 failed to bootstrap but spack install re2c %gcc@9.2.0 did the right thing (even if it installed re2c and I’ll never use it)

I thought about it afterwards - but perhaps debug flags are causing the PID problem (-ddd)

Either way, To get spack to bootstrap, I need to tell it to install re2c, which then let the bootstrap also work itself out.

I’ll need to try your new PRs and report back. May be 2-3 hours though

@alalazo I pulled develop Then branched and merged #22354 , I applied your patch.

I have spack config edit config with the clingo as the concretizer.

config:
  concretizer: clingo

My packages.yaml does not allow GCC (it only has cce as the compiler) … so, when I tried

spack clean --all
spack spec zlib 

It naturally tried to bootstrap with CCE, then failed with the error that GCC must be used.

I modified my packages.yaml to have compilers: [ gcc@9.2.0, cce... ], and that resolved the issue of picking CCE.

Again, clean + spec: (this time saying %gcc@9.2.0 - which is important, because my gcc 8.3.1 has no static libs - as per my discussion in the other merge request)

spack clean --all
spack -ddd --verbose spec zlib %gcc@9.2.0

==> [2021-03-23-23:09:50.734974, 282165] Error: AttributeError: 'Re2c' object has no attribute 'pid'

/home/users/coe0201/src/spack/lib/spack/spack/build_environment.py:872, in _setup_pkg_and_run:
        869        tb_string = traceback.format_exc()
        870
        871        # build up some context from the offending package so we can
  >>    872        # show that, too.
        873        package_context = get_package_context(tb)
        874
        875        logfile = None


Traceback (most recent call last):
  File "/home/users/coe0201/src/spack/lib/spack/spack/build_environment.py", line 856, in _setup_pkg_and_run
    return_value = function(pkg, kwargs)
  File "/home/users/coe0201/src/spack/lib/spack/spack/installer.py", line 1652, in build_process
    pid = '{0}: '.format(pkg.pid) if tty.show_pid() else ''
AttributeError: 'Re2c' object has no attribute 'pid'

Seen this before…(install re2c first)

spack clean --all
spack install re2c %gcc@9.2.0

The install triggered the clingo bootstrap in a good way… (output below)

[coe0201@redwood ~]$ spack --verbose install re2c %gcc@9.2.0
==> Requeueing a build task for bison-3.0.4-lks34hgwycgwrmtvunse46y2hznirot3 with status 'queued'
==> Requeueing a build task for cmake-3.19.5-32pgptf5yf6mb7h3e7lfe6osqicjfw54 with status 'queued'
==> Requeueing a build task for python-3.6-wcpulhtppxwcub6ok34myhjljhnodigi with status 'queued'
==> Warning: Missing a source id for python@3.6
==> Requeueing a build task for re2c-1.2.1-zd3tqbnpdbb7jb7w6p4yghpemjifxoi4 with status 'queued'
==> Requeueing a build task for clingo-bootstrap-spack-zsqxizzv4homqxa6lxor4gu6iae4bka2 with status 'queued'
==> Processing bison-3.0.4-lks34hgwycgwrmtvunse46y2hznirot3: task=priority=0, status=dequeued, start=0, #dependencies=0
[+] /usr (external bison-3.0.4-lks34hgwycgwrmtvunse46y2hznirot3)
==> Removing build task for clingo-bootstrap-spack-zsqxizzv4homqxa6lxor4gu6iae4bka2 from list
==> Requeueing a build task for clingo-bootstrap-spack-zsqxizzv4homqxa6lxor4gu6iae4bka2 with status 'queued'
==> Processing cmake-3.19.5-32pgptf5yf6mb7h3e7lfe6osqicjfw54: task=priority=0, status=dequeued, start=0, #dependencies=0
==> cmake@3.19.5 : has external module in ['atdm-cmake', 'atdm-ninja']
[+] /home/users/coe0201/src/spack/opt/spack/cray-rhel8-x86_64/gcc-8.3.1/ninja-kitware-n55hguizo6jt3t34vng46bplzhgrrnzq (external cmake-3.19.5-32pgptf5yf6mb7h3e7lfe6osqicjfw54)
==> Removing build task for clingo-bootstrap-spack-zsqxizzv4homqxa6lxor4gu6iae4bka2 from list
==> Requeueing a build task for clingo-bootstrap-spack-zsqxizzv4homqxa6lxor4gu6iae4bka2 with status 'queued'
==> Processing python-3.6-wcpulhtppxwcub6ok34myhjljhnodigi: task=priority=0, status=dequeued, start=0, #dependencies=0
[+] /usr (external python-3.6-wcpulhtppxwcub6ok34myhjljhnodigi)
==> Removing build task for clingo-bootstrap-spack-zsqxizzv4homqxa6lxor4gu6iae4bka2 from list
==> Requeueing a build task for clingo-bootstrap-spack-zsqxizzv4homqxa6lxor4gu6iae4bka2 with status 'queued'
==> Processing re2c-1.2.1-zd3tqbnpdbb7jb7w6p4yghpemjifxoi4: task=priority=0, status=dequeued, start=0, #dependencies=0
==> re2c-1.2.1-zd3tqbnpdbb7jb7w6p4yghpemjifxoi4 is now write locked
==> Installing re2c-1.2.1-zd3tqbnpdbb7jb7w6p4yghpemjifxoi4
==> No binary for re2c-1.2.1-zd3tqbnpdbb7jb7w6p4yghpemjifxoi4 found: installing from source
==> Creating the installation directory /home/users/coe0201/.spack/bootstrap/store/cray-rhel8-x86_64/gcc-9.2.0/re2c-1.2.1-zd3tqbnpdbb7jb7w6p4yghpemjifxoi4
==> Fetching https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/_source-cache/archive/1a/1a4cd706b5b966aeffd78e3cf8b24239470ded30551e813610f9cd1a4e01b817.tar.xz
########################################################################################################################################################################### 100.0%
==> No patches needed for re2c
==> re2c: Executing phase: 'autoreconf'
==> re2c: Executing phase: 'configure'
==> re2c: Executing phase: 'build'
==> re2c: Executing phase: 'install'
==> re2c: Successfully installed re2c-1.2.1-zd3tqbnpdbb7jb7w6p4yghpemjifxoi4
  Fetch: 1.02s.  Build: 11.33s.  Total: 12.35s.
[+] /home/users/coe0201/.spack/bootstrap/store/cray-rhel8-x86_64/gcc-9.2.0/re2c-1.2.1-zd3tqbnpdbb7jb7w6p4yghpemjifxoi4
==> Removing build task for clingo-bootstrap-spack-zsqxizzv4homqxa6lxor4gu6iae4bka2 from list
==> Requeueing a build task for clingo-bootstrap-spack-zsqxizzv4homqxa6lxor4gu6iae4bka2 with status 'queued'
==> re2c-1.2.1-zd3tqbnpdbb7jb7w6p4yghpemjifxoi4 is now read locked
==> Processing clingo-bootstrap-spack-zsqxizzv4homqxa6lxor4gu6iae4bka2: task=priority=0, status=dequeued, start=1616559057.0411685, #dependencies=4
==> clingo-bootstrap-spack-zsqxizzv4homqxa6lxor4gu6iae4bka2 is now write locked
==> Installing clingo-bootstrap-spack-zsqxizzv4homqxa6lxor4gu6iae4bka2
==> No binary for clingo-bootstrap-spack-zsqxizzv4homqxa6lxor4gu6iae4bka2 found: installing from source
==> Creating the installation directory /home/users/coe0201/.spack/bootstrap/store/cray-rhel8-x86_64/gcc-9.2.0/clingo-bootstrap-spack-zsqxizzv4homqxa6lxor4gu6iae4bka2
==> Warning: Missing a source id for python@3.6
==> Ran patch() for clingo-bootstrap
==> clingo-bootstrap: Executing phase: 'cmake'
==> clingo-bootstrap: Executing phase: 'build'
==> clingo-bootstrap: Executing phase: 'install'
==> clingo-bootstrap: Successfully installed clingo-bootstrap-spack-zsqxizzv4homqxa6lxor4gu6iae4bka2
  Fetch: 3.60s.  Build: 55.05s.  Total: 58.65s.
[+] /home/users/coe0201/.spack/bootstrap/store/cray-rhel8-x86_64/gcc-9.2.0/clingo-bootstrap-spack-zsqxizzv4homqxa6lxor4gu6iae4bka2
==> clingo-bootstrap-spack-zsqxizzv4homqxa6lxor4gu6iae4bka2 is now read locked
==> Releasing read lock on re2c-1.2.1-zd3tqbnpdbb7jb7w6p4yghpemjifxoi4
==> Releasing read lock on clingo-bootstrap-spack-zsqxizzv4homqxa6lxor4gu6iae4bka2
==> Requeueing a build task for re2c-1.2.1-4jadz2qm372ungj6ur26agjkgeblfjb7 with status 'queued'
==> Processing re2c-1.2.1-4jadz2qm372ungj6ur26agjkgeblfjb7: task=priority=0, status=dequeued, start=0, #dependencies=0
==> re2c-1.2.1-4jadz2qm372ungj6ur26agjkgeblfjb7 is now write locked
==> Installing re2c-1.2.1-4jadz2qm372ungj6ur26agjkgeblfjb7
==> No binary for re2c-1.2.1-4jadz2qm372ungj6ur26agjkgeblfjb7 found: installing from source
==> Creating the installation directory /home/users/coe0201/src/spack/opt/spack/cray-rhel8-haswell/gcc-9.2.0/re2c-1.2.1-4jadz2qm372ungj6ur26agjkgeblfjb7
==> Using cached archive: /home/users/coe0201/src/spack/var/spack/cache/_source-cache/archive/1a/1a4cd706b5b966aeffd78e3cf8b24239470ded30551e813610f9cd1a4e01b817.tar.xz
==> No patches needed for re2c
==> re2c: Executing phase: 'autoreconf'
==> re2c: Executing phase: 'configure'
==> re2c: Executing phase: 'build'
==> re2c: Executing phase: 'install'
==> re2c: Successfully installed re2c-1.2.1-4jadz2qm372ungj6ur26agjkgeblfjb7
  Fetch: 0.02s.  Build: 11.06s.  Total: 11.08s.
[+] /home/users/coe0201/src/spack/opt/spack/cray-rhel8-haswell/gcc-9.2.0/re2c-1.2.1-4jadz2qm372ungj6ur26agjkgeblfjb7
==> re2c-1.2.1-4jadz2qm372ungj6ur26agjkgeblfjb7 is now read locked
==> Releasing read lock on re2c-1.2.1-4jadz2qm372ungj6ur26agjkgeblfjb7

It looks like it did some installs twice

It did work though.

Problem - it doesn’t work if you spec/install anything with non-gcc

e.g.,

[coe0201@redwood ~]$ spack spec zlib
Input spec
--------------------------------
zlib

Concretized
--------------------------------
==> Error: Conflicts in concretized spec "clingo-bootstrap@spack%cce@11.0.3~docs~ipo+python build_type=Release arch=cray-rhel8-x86_64/72i3grj"

List of matching conflicts for spec:

    clingo-bootstrap@spack%cce@11.0.3 cflags="-g -fPIC" cxxflags="-g -fPIC" fflags="-g -fPIC" ~docs~ipo+python build_type=Release arch=cray-rhel8-x86_64
        ^bison@3.0.4%cce@11.0.3 cflags="-g -fPIC" cxxflags="-g -fPIC" fflags="-g -fPIC"  patches=b72914fe38e54a6fc25f29019e0a0786705c4f61ce20d414cc2010c8d62448c7 arch=cray-rhel8-x86_64
        ^cmake@3.19.5%cce@11.0.3 cflags="-g -fPIC" cxxflags="-g -fPIC" fflags="-g -fPIC" ~doc+ncurses+openssl+ownlibs~qt build_type=Release patches=b48396c0e4f61756248156b6cebe9bc0d7a22228639b47b5aa77c9330588ce88 arch=cray-rhel8-x86_64
        ^python@3.6%cce@11.0.3 cflags="-g -fPIC" cxxflags="-g -fPIC" fflags="-g -fPIC" +bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=ebdca648c9c1d25f586d7e2a495b62e6d91973b55264a13d89eda1beff72ef56 arch=cray-rhel8-x86_64
        ^re2c@1.2.1%cce@11.0.3 cflags="-g -fPIC" cxxflags="-g -fPIC" fflags="-g -fPIC"  arch=cray-rhel8-x86_64

1. "%cce" conflicts with "clingo-bootstrap arch=cray-None-None" [GCC is required to bootstrap clingo on Cray]

Try with GCC still allowed in packages (but with lowest priority)

packages:
  all:
    compiler:
      - cce@11.0.3
      - gcc@9.2.0

still will pick cce, since it’s the default If I put gcc first, then it will just use GCC, which is no fun at all.

action items

  1. I had to install re2c first… Even though that very install triggered the bootstrap correctly. If I tried without re2c installed, I get cryptic errors about missing pid
  2. In the current state, it seems only possible to use clingo if you are also building with GCC as the toolchain. This will likely make all the people at Cray frown at you.