spack: clingo bootstrap failing inside certain environments

Steps to reproduce

I believe this is an issue with trying to bootstrap inside an environment. It could be anonymous environments. I’m not sure. pinging @alalazo

rm -rf ~/.spack $SPACK_ROOT
#clone spack
#source spack-setup.sh
#create anonyomus env with no view and concretization: together
#activate env
spack add nalu-wind
spack concretize

Error message

psakiev@skybridge-login3 disk_dev]$ spack -d concretize
==> [2021-09-16-06:40:17.351737] Imported concretize from built-in commands
==> [2021-09-16-06:40:17.352881] Imported concretize from built-in commands
==> [2021-09-16-06:40:17.367058] Reading config file /projects/wind/spack-manager/environments/disk_dev/general_repos.yaml
==> [2021-09-16-06:40:17.369307] Reading config file /projects/wind/spack-manager/environments/disk_dev/general_config.yaml
==> [2021-09-16-06:40:17.372345] Reading config file /projects/wind/spack-manager/environments/disk_dev/general_packages.yaml
==> [2021-09-16-06:40:17.390171] Reading config file /projects/wind/spack-manager/environments/disk_dev/machine_compilers.yaml
==> [2021-09-16-06:40:17.406069] Reading config file /projects/wind/spack-manager/environments/disk_dev/machine_packages.yaml
==> [2021-09-16-06:40:17.440079] Reading config file /projects/wind/spack-manager/environments/disk_dev/machine_config.yaml
==> [2021-09-16-06:40:17.447444] Reading config file /projects/wind/spack-manager/environments/disk_dev/spack.yaml
==> [2021-09-16-06:40:17.556539] Reading config file /projects/wind/spack-manager/spack/etc/spack/defaults/bootstrap.yaml
==> [2021-09-16-06:40:17.580823] Reading config file /projects/wind/spack-manager/spack/etc/spack/defaults/config.yaml
==> [2021-09-16-06:40:17.614834] DATABASE LOCK TIMEOUT: 3s
==> [2021-09-16-06:40:17.614934] PACKAGE LOCK TIMEOUT: No timeout
==> [2021-09-16-06:40:17.615501] Reading config file /projects/wind/spack-manager/spack/etc/spack/defaults/bootstrap.yaml
==> [2021-09-16-06:40:17.633425] DATABASE LOCK TIMEOUT: 3s
==> [2021-09-16-06:40:17.633528] PACKAGE LOCK TIMEOUT: No timeout
==> [2021-09-16-06:40:17.633668] [BOOTSTRAP CONFIG SCOPE] name=_builtin
==> [2021-09-16-06:40:17.634234] [BOOTSTRAP CONFIG SCOPE] name=defaults, path=/projects/wind/spack-manager/spack/etc/spack/defaults
==> [2021-09-16-06:40:17.634322] [BOOTSTRAP CONFIG SCOPE] name=defaults/linux, path=/projects/wind/spack-manager/spack/etc/spack/defaults/linux
==> [2021-09-16-06:40:17.634403] [BOOTSTRAP CONFIG SCOPE] name=system, path=/etc/spack
==> [2021-09-16-06:40:17.634481] [BOOTSTRAP CONFIG SCOPE] name=system/linux, path=/etc/spack/linux
==> [2021-09-16-06:40:17.634570] [BOOTSTRAP CONFIG SCOPE] name=site, path=/projects/wind/spack-manager/spack/etc/spack
==> [2021-09-16-06:40:17.634648] [BOOTSTRAP CONFIG SCOPE] name=site/linux, path=/projects/wind/spack-manager/spack/etc/spack/linux
==> [2021-09-16-06:40:17.634735] [BOOTSTRAP CONFIG SCOPE] name=user, path=/ascldap/users/psakiev/.spack
==> [2021-09-16-06:40:17.634812] [BOOTSTRAP CONFIG SCOPE] name=user/linux, path=/ascldap/users/psakiev/.spack/linux
==> [2021-09-16-06:40:17.640129] [BOOTSTRAP ROOT SPEC] clingo: clingo-bootstrap@spack+python %gcc target=x86_64
==> [2021-09-16-06:40:17.640215] [BOOTSTRAP MODULE clingo] Try importing from Python
==> [2021-09-16-06:40:17.640689] Reading config file /projects/wind/spack-manager/spack/etc/spack/defaults/bootstrap.yaml
==> [2021-09-16-06:40:17.662455] Bootstrapping clingo from pre-built binaries
==> [2021-09-16-06:40:17.777211] Imported buildcache from built-in commands
==> [2021-09-16-06:40:17.785090] Imported buildcache from built-in commands
==> [2021-09-16-06:40:17.786600] Reading config file /projects/wind/spack-manager/spack/etc/spack/defaults/config.yaml
==> [2021-09-16-06:40:17.839194] DATABASE LOCK TIMEOUT: 3s
==> [2021-09-16-06:40:17.839281] PACKAGE LOCK TIMEOUT: No timeout
==> [2021-09-16-06:40:17.892251] Reading config file /projects/wind/spack-manager/spack/etc/spack/defaults/mirrors.yaml
==> [2021-09-16-06:40:18.207104] Cached index for https://mirror.spack.io/bootstrap/github-actions/v0.1 already up to date
==> [2021-09-16-06:40:18.222695] [BOOTSTRAP MODULE clingo] Try installing "clingo-bootstrap" from binary cache at "https://mirror.spack.io/bootstrap/github-actions/v0.1"
==> [2021-09-16-06:40:25.528251] NoCompilersForArchError: No compilers found for operating system rhel7 and target sandybridge.
If previous installations have succeeded, the operating system may have been updated.
Compilers are defined for the following operating systems and targets:
	rhel5-x86_64
    Run 'spack compiler find' to add compilers.
==> [2021-09-16-06:40:25.535851] Reading config file /projects/wind/spack-manager/spack/etc/spack/defaults/bootstrap.yaml
==> [2021-09-16-06:40:26.009221] diffutils applying constraint iconv
==> [2021-09-16-06:40:26.308821] ncurses applying constraint pkgconfig
==> [2021-09-16-06:40:26.310875] readline applying constraint ncurses
==> [2021-09-16-06:40:26.312950] gdbm applying constraint readline
==> [2021-09-16-06:40:26.368339] bzip2 applying constraint diffutils
==> [2021-09-16-06:40:26.425215] perl applying constraint gdbm@:1.19
==> [2021-09-16-06:40:26.427484] perl applying constraint berkeley-db
==> [2021-09-16-06:40:26.429713] perl applying constraint bzip2
==> [2021-09-16-06:40:26.431957] perl applying constraint zlib
==> [2021-09-16-06:40:26.434233] bison applying constraint diffutils
==> [2021-09-16-06:40:26.436433] bison applying constraint m4
==> [2021-09-16-06:40:26.438630] bison applying constraint perl
==> [2021-09-16-06:40:26.441166] clingo-bootstrap applying constraint python
==> [2021-09-16-06:40:26.442178] python applying constraint pkgconfig@0.9.0:
==> [2021-09-16-06:40:26.445835] python applying constraint pkgconfig@0.9.0:
==> [2021-09-16-06:40:26.451382] clingo-bootstrap applying constraint cmake@3.16.0:
==> [2021-09-16-06:40:26.454091] clingo-bootstrap applying constraint re2c@0.13:
==> [2021-09-16-06:40:26.457168] clingo-bootstrap applying constraint bison@2.5:
==> [2021-09-16-06:40:26.459679] clingo-bootstrap applying constraint python
==> [2021-09-16-06:40:26.464214] Reading config file /projects/wind/spack-manager/spack/etc/spack/defaults/packages.yaml
==> [2021-09-16-06:40:33.312655] Reading config file /projects/wind/spack-manager/spack/etc/spack/defaults/config.yaml
==> [2021-09-16-06:40:33.449589] clingo-bootstrap applying constraint cmake@3.16.0:
==> [2021-09-16-06:40:33.452269] clingo-bootstrap applying constraint re2c@0.13:
==> [2021-09-16-06:40:33.453871] clingo-bootstrap applying constraint bison@2.5:
==> [2021-09-16-06:40:33.455070] bison applying constraint diffutils
==> [2021-09-16-06:40:33.455789] diffutils applying constraint libiconv
==> [2021-09-16-06:40:33.456952] bison applying constraint m4
==> [2021-09-16-06:40:33.458127] bison applying constraint perl
==> [2021-09-16-06:40:33.459316] perl applying constraint gdbm@:1.19
==> [2021-09-16-06:40:33.460453] gdbm applying constraint readline
==> [2021-09-16-06:40:33.461561] readline applying constraint ncurses
==> [2021-09-16-06:40:33.462236] ncurses applying constraint pkgconfig
==> [2021-09-16-06:40:33.463384] perl applying constraint berkeley-db
==> [2021-09-16-06:40:33.464511] perl applying constraint bzip2
==> [2021-09-16-06:40:33.465637] bzip2 applying constraint diffutils
==> [2021-09-16-06:40:33.466754] perl applying constraint zlib
==> [2021-09-16-06:40:33.468113] clingo-bootstrap applying constraint python
==> [2021-09-16-06:40:33.468958] python applying constraint pkgconfig@0.9.0:
==> [2021-09-16-06:40:33.569161] clingo-bootstrap applying constraint cmake@3.16.0:
==> [2021-09-16-06:40:33.571687] clingo-bootstrap applying constraint re2c@0.13:
==> [2021-09-16-06:40:33.573167] clingo-bootstrap applying constraint bison@2.5:
==> [2021-09-16-06:40:33.574240] bison applying constraint diffutils
==> [2021-09-16-06:40:33.574889] diffutils applying constraint libiconv
==> [2021-09-16-06:40:33.575934] bison applying constraint m4
==> [2021-09-16-06:40:33.577110] bison applying constraint perl
==> [2021-09-16-06:40:33.578203] perl applying constraint gdbm@:1.19
==> [2021-09-16-06:40:33.579255] gdbm applying constraint readline
==> [2021-09-16-06:40:33.580285] readline applying constraint ncurses
==> [2021-09-16-06:40:33.580928] ncurses applying constraint pkgconf
==> [2021-09-16-06:40:33.581985] perl applying constraint berkeley-db
==> [2021-09-16-06:40:33.583033] perl applying constraint bzip2
==> [2021-09-16-06:40:33.584051] bzip2 applying constraint diffutils
==> [2021-09-16-06:40:33.585069] perl applying constraint zlib
==> [2021-09-16-06:40:33.586314] clingo-bootstrap applying constraint python
==> [2021-09-16-06:40:33.587051] python applying constraint pkgconf
==> [2021-09-16-06:40:33.603726] clingo-bootstrap applying constraint cmake@3.16.0:
==> [2021-09-16-06:40:33.606278] clingo-bootstrap applying constraint re2c@0.13:
==> [2021-09-16-06:40:33.607738] clingo-bootstrap applying constraint bison@2.5:
==> [2021-09-16-06:40:33.608804] bison applying constraint diffutils
==> [2021-09-16-06:40:33.609509] diffutils applying constraint libiconv
==> [2021-09-16-06:40:33.610546] bison applying constraint m4
==> [2021-09-16-06:40:33.611631] bison applying constraint perl
==> [2021-09-16-06:40:33.612753] perl applying constraint gdbm@:1.19
==> [2021-09-16-06:40:33.613897] gdbm applying constraint readline
==> [2021-09-16-06:40:33.614930] readline applying constraint ncurses
==> [2021-09-16-06:40:33.615579] ncurses applying constraint pkgconf
==> [2021-09-16-06:40:33.616638] perl applying constraint berkeley-db
==> [2021-09-16-06:40:33.617666] perl applying constraint bzip2
==> [2021-09-16-06:40:33.618689] bzip2 applying constraint diffutils
==> [2021-09-16-06:40:33.619706] perl applying constraint zlib
==> [2021-09-16-06:40:33.620960] clingo-bootstrap applying constraint python
==> [2021-09-16-06:40:33.629770] [BOOTSTRAP MODULE clingo] Unexpected error "No compilers with spec gcc found for operating system rhel7 and target x86_64.
    Run 'spack compiler find' to add compilers or 'spack compilers' to see which compilers are already recognized by spack."
==> [2021-09-16-06:40:33.648392] Reading config file /projects/wind/spack-manager/spack/etc/spack/defaults/config.yaml
Traceback (most recent call last):
  File "/projects/wind/spack-manager/spack/bin/spack", line 100, in <module>
    sys.exit(spack.main.main())
  File "/projects/wind/spack-manager/spack/lib/spack/spack/main.py", line 797, in main
    return _invoke_command(command, parser, args, unknown)
  File "/projects/wind/spack-manager/spack/lib/spack/spack/main.py", line 525, in _invoke_command
    return_val = command(parser, args)
  File "/projects/wind/spack-manager/spack/lib/spack/spack/cmd/concretize.py", line 37, in concretize
    concretized_specs = env.concretize(force=args.force, tests=tests)
  File "/projects/wind/spack-manager/spack/lib/spack/spack/environment.py", line 1147, in concretize
    return self._concretize_together(tests=tests)
  File "/projects/wind/spack-manager/spack/lib/spack/spack/environment.py", line 1187, in _concretize_together
    *self.user_specs, tests=tests)
  File "/projects/wind/spack-manager/spack/lib/spack/spack/concretize.py", line 725, in concretize_specs_together
    return _concretize_specs_together_new(*abstract_specs, **kwargs)
  File "/projects/wind/spack-manager/spack/lib/spack/spack/concretize.py", line 730, in _concretize_specs_together_new
    result = spack.solver.asp.solve(abstract_specs)
  File "/projects/wind/spack-manager/spack/lib/spack/spack/solver/asp.py", line 1679, in solve
    driver = PyclingoDriver()
  File "/projects/wind/spack-manager/spack/lib/spack/spack/solver/asp.py", line 292, in __init__
    spack.bootstrap.ensure_clingo_importable_or_raise()
  File "/projects/wind/spack-manager/spack/lib/spack/spack/bootstrap.py", line 559, in ensure_clingo_importable_or_raise
    module='clingo', abstract_spec=clingo_root_spec()
  File "/projects/wind/spack-manager/spack/lib/spack/spack/bootstrap.py", line 399, in ensure_module_importable_or_raise
    raise ImportError(msg)
ImportError: cannot bootstrap the "clingo" Python module from spec "clingo-bootstrap@spack+python %gcc target=x86_64"

Information on your system

* **Spack:** 0.16.2-4267-5b211c90f5
* **Python:** 3.6.8
* **Platform:** linux-rhel7-sandybridge
* **Concretizer:** clingo

General information

  • I have run spack debug report and reported the version of Spack/Python/Platform
  • I have searched the issues of this repo and believe this is not a duplicate
  • I have run the failing commands in debug mode and reported the output

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 22 (22 by maintainers)

Commits related to this issue

Most upvoted comments

@alalazo yes #26029 works for cases with and without a compilers section in spack.yaml when there is no compilers.yaml in ~/.spack.

@alalazo yes that solves it for me as well. Thanks for tracking this down.

Adding a compilers section to spack.yaml I can finally reproduce a failure.

Here is my bootstrap store after successful bootstraping

# spack find -b
==> Showing internal bootstrap store at "/ascldap/users/psakiev/.spack/bootstrap/store"
==> 3 installed packages
-- linux-rhel5-x86_64 / gcc@9.3.0 -------------------------------
clingo-bootstrap@spack  python@3.6

-- linux-rhel7-sandybridge / gcc@4.8.5 --------------------------
patchelf@0.13

I’ll try to reproduce on another machine later today. Unfortunately all the rhel7 machines I use share the same home directory so I need to wipe ~/.spack again. One of my struggles with ~/.spack priority