spack: clingo fails with python 3.6 on ppc64le RHEL 7.x

Steps to reproduce

Clingo does not work with the /usr/bin/python3 3.6 that comes with RedHat 7.x on ppc64le. The initial bootstrapping fails with:

$ spack solve zlib  
==> Error: module 'clingo' has no attribute 'Control'

I tried this on a power9 with RHEL 7.7 with /usr/bin/python3 3.6.8, rpm package python3-3.6.8-10.el7.ppc64le.

How to reproduce:

  1. move .spack to another name, eg: dot-spack, so you have a clean, empty setup

  2. make a fresh clone of spack develop

  3. source setup-env.sh, and edit config.yaml to set concretizer to clingo

  4. run: spack compiler find

$ spack compiler find
==> Added 1 new compiler to /home/krentel/.spack/linux/compilers.yaml
    gcc@4.8.5
==> Compilers are defined in the following files:
    /home/krentel/.spack/linux/compilers.yaml
  1. run: spack solve zlib
$ spack solve zlib
==> Error: module 'clingo' has no attribute 'Control'

You can also reproduce the problem on lassen at LLNL. The thing about LLNL machines is that they come with the StdEnv module that puts another python from /usr/tcetmp/bin in front of your path. If you move that out of the way and use /usr/bin/python3, then it has the same problem.

You could unload the StdEnv module, or add /usr/bin to the front of your path. Anything that puts /usr/bin/python3 before /usr/tcetmp will get back to the bare RH 7.x python and reproduce the problem.

You can even mkdir some directory, copy /usr/bin/python3 to the new directory and put that in front of your path. That pretty much confirms the problem is python because it’s the only new thing in your path.

Notes:

  1. As I mentioned, the problem is in python3 shipped with RHEL 7.x on ppc64le. If you build a new python, even the same 3.6 rev, then it works.

  2. The problem is specific to ppc64le. It doesn’t happen on x86_64, even with the same RH rev. I haven’t tried ARM.

  3. I haven’t tried RH 8.x. If someone can suggest a RH 8.x powerpc machine, maybe something at LLNL or Sandia, I could try it there.

I don’t know if the problem is that the RH python lacks some feature, or if spack doesn’t detect or use is correctly. But if the shipped python won’t work with clingo, then that adds an extra step to the bootstrap process to build a new python.

I’ve seen this sort of problem before where the same rev of RH on different architectures don’t provide exactly the same set of features. (But more often with compiler features.)

ping @alalazo

Error message

$ spack -d solve zlib

==> [2021-09-01-12:27:21.791669] Imported solve from built-in commands
==> [2021-09-01-12:27:21.793301] Imported solve from built-in commands
==> [2021-09-01-12:27:21.794287] Reading config file /home/krentel/clingo/spack-repo/etc/spack/defaults/bootstrap.yaml
==> [2021-09-01-12:27:21.818877] Reading config file /home/krentel/clingo/spack-repo/etc/spack/defaults/config.yaml
==> [2021-09-01-12:27:21.870298] Reading config file /home/krentel/clingo/spack-repo/etc/spack/config.yaml
==> [2021-09-01-12:27:21.917261] DATABASE LOCK TIMEOUT: 3s
==> [2021-09-01-12:27:21.917325] PACKAGE LOCK TIMEOUT: No timeout
==> [2021-09-01-12:27:21.917923] Reading config file /home/krentel/clingo/spack-repo/etc/spack/defaults/bootstrap.yaml
==> [2021-09-01-12:27:21.938049] DATABASE LOCK TIMEOUT: 3s
==> [2021-09-01-12:27:21.938110] PACKAGE LOCK TIMEOUT: No timeout
==> [2021-09-01-12:27:21.938208] [BOOTSTRAP CONFIG SCOPE] name=_builtin
==> [2021-09-01-12:27:21.938851] [BOOTSTRAP CONFIG SCOPE] name=defaults, path=/home/krentel/clingo/spack-repo/etc/spack/defaults
==> [2021-09-01-12:27:21.938911] [BOOTSTRAP CONFIG SCOPE] name=defaults/linux, path=/home/krentel/clingo/spack-repo/etc/spack/defaults/linux
==> [2021-09-01-12:27:21.938977] [BOOTSTRAP CONFIG SCOPE] name=system, path=/etc/spack
==> [2021-09-01-12:27:21.939030] [BOOTSTRAP CONFIG SCOPE] name=system/linux, path=/etc/spack/linux
==> [2021-09-01-12:27:21.939094] [BOOTSTRAP CONFIG SCOPE] name=site, path=/home/krentel/clingo/spack-repo/etc/spack
==> [2021-09-01-12:27:21.939146] [BOOTSTRAP CONFIG SCOPE] name=site/linux, path=/home/krentel/clingo/spack-repo/etc/spack/linux
==> [2021-09-01-12:27:21.939208] [BOOTSTRAP CONFIG SCOPE] name=user, path=/home/krentel/.spack
==> [2021-09-01-12:27:21.939259] [BOOTSTRAP CONFIG SCOPE] name=user/linux, path=/home/krentel/.spack/linux
==> [2021-09-01-12:27:21.962406] [BOOTSTRAP ROOT SPEC] clingo: clingo-bootstrap@spack+python %gcc target=ppc64le
==> [2021-09-01-12:27:21.962470] [BOOTSTRAP MODULE clingo] Try importing from Python
==> [2021-09-01-12:27:21.962851] Reading config file /home/krentel/clingo/spack-repo/etc/spack/defaults/bootstrap.yaml
==> [2021-09-01-12:27:22.144795] Reading config file /home/krentel/clingo/spack-repo/etc/spack/defaults/config.yaml
==> [2021-09-01-12:27:22.202184] Reading config file /home/krentel/clingo/spack-repo/etc/spack/config.yaml
==> [2021-09-01-12:27:22.322394] [BOOTSTRAP MODULE clingo] The installed spec "clingo-bootstrap@spack+python %gcc target=ppc64le ^python@3.6 /rfu4la457mdbpoffk2g5hikj2hhoek4s" provides the "clingo" Python module
==> [2021-09-01-12:27:22.322660] Reading config file /home/krentel/clingo/spack-repo/etc/spack/defaults/repos.yaml
==> [2021-09-01-12:27:22.330052] Reading config file /home/krentel/clingo/spack-repo/etc/spack/defaults/config.yaml
==> [2021-09-01-12:27:22.381029] Reading config file /home/krentel/clingo/spack-repo/etc/spack/config.yaml
Traceback (most recent call last):
  File "/home/krentel/clingo/spack-repo/bin/spack", line 100, in <module>
    sys.exit(spack.main.main())
  File "/home/krentel/clingo/spack-repo/lib/spack/spack/main.py", line 797, in main
    return _invoke_command(command, parser, args, unknown)
  File "/home/krentel/clingo/spack-repo/lib/spack/spack/main.py", line 525, in _invoke_command
    return_val = command(parser, args)
  File "/home/krentel/clingo/spack-repo/lib/spack/spack/cmd/solve.py", line 106, in solve
    specs, dump=dump, models=models, timers=args.timers, stats=args.stats
  File "/home/krentel/clingo/spack-repo/lib/spack/spack/solver/asp.py", line 1691, in solve
    return driver.solve(setup, specs, dump, models, timers, stats, tests)
  File "/home/krentel/clingo/spack-repo/lib/spack/spack/solver/asp.py", line 332, in solve
    self.control = clingo.Control()
AttributeError: module 'clingo' has no attribute 'Control'

Information on your system

  • Spack: 0.16.2-4111-7adacf9
  • Python: 3.6.8
  • Platform: linux-rhel7-power9le
  • Concretizer: clingo

General information

  • I have run spack debug report and reported the version of Spack/Python/Platform
  • I have searched the issues of this repo and believe this is not a duplicate
  • I have run the failing commands in debug mode and reported the output

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 22 (21 by maintainers)

Most upvoted comments

sysconfig.get_config_var(“EXT_SUFFIX”)

Ok then maybe the thing to do is to symlink that if it’s different from the manylinux name — that seems more robust than including everything we know of in the buildcache; it might save us on platforms we don’t anticipate.

So I’d say do the quick fix for now, but we need to get more machines validating the build caches w/different containers. We can add runners to them; we should have some power9 runners at LLNL that we can use; also GCP has some (though I dunno if we can use them effectively — they are AIX).

Would it be a simple workaround to add a symlink in the clingo bootstrap to cover the other name?

I tried that manually and it seems to work.

I can reproduce that on Lassen. It seems the extension is not ABI compatible with that Python, even though I am trying to get more information. The weird thing is that the extension has been compiled against a Python 3.6 coming from manylinux2014 so it should be compatible with any Python 3.6. I’ll post findings here as I understand more.