spack: spack.fetch_strategy.FetchError: All fetchers failed for spack-stage-gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6

Steps to reproduce

On a fresh Apple M1 MacBook Pro, just installed the command line utilities (like git), cloned spack and put spack into $PATH. The bootstrapping fails:

$ spack -d spec zlib

Error message

$ spack -d spec zlib
==> [2021-10-11-16:04:11.989835] Imported spec from built-in commands
==> [2021-10-11-16:04:11.990287] Imported spec from built-in commands
Input spec
--------------------------------
zlib

Concretized
--------------------------------
==> [2021-10-11-16:04:11.991004] Reading config file /Users/certik/repos/spack/etc/spack/defaults/config.yaml
==> [2021-10-11-16:04:12.001797] Reading config file /Users/certik/repos/spack/etc/spack/defaults/bootstrap.yaml
==> [2021-10-11-16:04:12.005618] Reading config file /Users/certik/.spack/bootstrap.yaml
==> [2021-10-11-16:04:12.006980] Reading config file /Users/certik/repos/spack/etc/spack/defaults/config.yaml
==> [2021-10-11-16:04:12.016365] DATABASE LOCK TIMEOUT: 3s
==> [2021-10-11-16:04:12.016388] PACKAGE LOCK TIMEOUT: No timeout
==> [2021-10-11-16:04:12.016570] Reading config file /Users/certik/repos/spack/etc/spack/defaults/bootstrap.yaml
==> [2021-10-11-16:04:12.020202] Reading config file /Users/certik/.spack/bootstrap.yaml
==> [2021-10-11-16:04:12.020790] DATABASE LOCK TIMEOUT: 3s
==> [2021-10-11-16:04:12.020808] PACKAGE LOCK TIMEOUT: No timeout
==> [2021-10-11-16:04:12.020835] [BOOTSTRAP CONFIG SCOPE] name=_builtin
==> [2021-10-11-16:04:12.020991] [BOOTSTRAP CONFIG SCOPE] name=defaults, path=/Users/certik/repos/spack/etc/spack/defaults
==> [2021-10-11-16:04:12.021007] [BOOTSTRAP CONFIG SCOPE] name=defaults/darwin, path=/Users/certik/repos/spack/etc/spack/defaults/darwin
==> [2021-10-11-16:04:12.021022] [BOOTSTRAP CONFIG SCOPE] name=bootstrap, path=/Users/certik/.spack/bootstrap/config
==> [2021-10-11-16:04:12.021034] [BOOTSTRAP CONFIG SCOPE] name=bootstrap/darwin, path=/Users/certik/.spack/bootstrap/config/darwin
==> [2021-10-11-16:04:12.021167] Reading config file /Users/certik/.spack/bootstrap/config/darwin/compilers.yaml
==> [2021-10-11-16:04:12.035914] [BOOTSTRAP ROOT SPEC] clingo: clingo-bootstrap@spack+python %apple-clang target=aarch64
==> [2021-10-11-16:04:12.036039] [BOOTSTRAP MODULE clingo] Try importing from Python
==> [2021-10-11-16:04:12.036269] Reading config file /Users/certik/repos/spack/etc/spack/defaults/bootstrap.yaml
==> [2021-10-11-16:04:12.040212] [BOOTSTRAP MODULE clingo] Skipping source "github-actions" since it is not trusted
==> [2021-10-11-16:04:12.040474] Bootstrapping clingo from sources
==> [2021-10-11-16:04:12.060837] '/usr/bin/bison' '--version'
==> [2021-10-11-16:04:12.074609] Reading config file /Users/certik/repos/spack/etc/spack/defaults/packages.yaml
==> [2021-10-11-16:04:12.088917] Reading config file /Users/certik/repos/spack/etc/spack/defaults/darwin/packages.yaml
==> [2021-10-11-16:04:12.093108] Reading config file /Users/certik/.spack/bootstrap/config/packages.yaml
==> [2021-10-11-16:04:12.098346] Reading config file /Users/certik/repos/spack/etc/spack/defaults/config.yaml
==> [2021-10-11-16:04:12.120945] diffutils applying constraint iconv
==> [2021-10-11-16:04:12.133233] ncurses applying constraint pkgconfig
==> [2021-10-11-16:04:12.133685] readline applying constraint ncurses
==> [2021-10-11-16:04:12.134125] gdbm applying constraint readline
==> [2021-10-11-16:04:12.136248] bzip2 applying constraint diffutils
==> [2021-10-11-16:04:12.137347] perl applying constraint gdbm@:1.19
==> [2021-10-11-16:04:12.137796] perl applying constraint berkeley-db
==> [2021-10-11-16:04:12.138222] perl applying constraint bzip2
==> [2021-10-11-16:04:12.138652] perl applying constraint zlib
==> [2021-10-11-16:04:12.139133] bison applying constraint diffutils
==> [2021-10-11-16:04:12.139562] bison applying constraint m4
==> [2021-10-11-16:04:12.140034] bison applying constraint perl
==> [2021-10-11-16:04:12.140593] clingo-bootstrap applying constraint python
==> [2021-10-11-16:04:12.140989] python applying constraint pkgconfig@0.9.0:
==> [2021-10-11-16:04:12.142296] python applying constraint pkgconfig@0.9.0:
==> [2021-10-11-16:04:12.143889] clingo-bootstrap applying constraint cmake@3.16.0:
==> [2021-10-11-16:04:12.144467] clingo-bootstrap applying constraint re2c@0.13:
==> [2021-10-11-16:04:12.145162] clingo-bootstrap applying constraint bison@2.5:
==> [2021-10-11-16:04:12.145818] clingo-bootstrap applying constraint python
==> [2021-10-11-16:04:12.183392] Reading config file /Users/certik/repos/spack/etc/spack/defaults/config.yaml
==> [2021-10-11-16:04:12.210556] clingo-bootstrap applying constraint cmake@3.16.0:
==> [2021-10-11-16:04:12.211221] clingo-bootstrap applying constraint re2c@0.13:
==> [2021-10-11-16:04:12.211657] clingo-bootstrap applying constraint bison@2.5:
==> [2021-10-11-16:04:12.211970] bison applying constraint diffutils
==> [2021-10-11-16:04:12.212161] diffutils applying constraint libiconv
==> [2021-10-11-16:04:12.212453] bison applying constraint m4
==> [2021-10-11-16:04:12.212751] bison applying constraint perl
==> [2021-10-11-16:04:12.213291] perl applying constraint gdbm@:1.19
==> [2021-10-11-16:04:12.213590] gdbm applying constraint readline
==> [2021-10-11-16:04:12.213877] readline applying constraint ncurses
==> [2021-10-11-16:04:12.214070] ncurses applying constraint pkgconfig
==> [2021-10-11-16:04:12.214335] perl applying constraint berkeley-db
==> [2021-10-11-16:04:12.214619] perl applying constraint bzip2
==> [2021-10-11-16:04:12.214874] bzip2 applying constraint diffutils
==> [2021-10-11-16:04:12.215127] perl applying constraint zlib
==> [2021-10-11-16:04:12.215484] clingo-bootstrap applying constraint python
==> [2021-10-11-16:04:12.215839] python applying constraint pkgconfig@0.9.0:
==> [2021-10-11-16:04:12.221881] clingo-bootstrap applying constraint cmake@3.16.0:
==> [2021-10-11-16:04:12.222562] clingo-bootstrap applying constraint re2c@0.13:
==> [2021-10-11-16:04:12.222957] clingo-bootstrap applying constraint bison@2.5:
==> [2021-10-11-16:04:12.223235] bison applying constraint diffutils
==> [2021-10-11-16:04:12.223424] diffutils applying constraint libiconv
==> [2021-10-11-16:04:12.223706] bison applying constraint m4
==> [2021-10-11-16:04:12.223974] bison applying constraint perl
==> [2021-10-11-16:04:12.224221] perl applying constraint gdbm@:1.19
==> [2021-10-11-16:04:12.224494] gdbm applying constraint readline
==> [2021-10-11-16:04:12.224754] readline applying constraint ncurses
==> [2021-10-11-16:04:12.224980] ncurses applying constraint pkgconf
==> [2021-10-11-16:04:12.225251] perl applying constraint berkeley-db
==> [2021-10-11-16:04:12.225504] perl applying constraint bzip2
==> [2021-10-11-16:04:12.225733] bzip2 applying constraint diffutils
==> [2021-10-11-16:04:12.225961] perl applying constraint zlib
==> [2021-10-11-16:04:12.226293] clingo-bootstrap applying constraint python
==> [2021-10-11-16:04:12.226625] python applying constraint pkgconf
==> [2021-10-11-16:04:12.230544] clingo-bootstrap applying constraint cmake@3.16.0:
==> [2021-10-11-16:04:12.231142] clingo-bootstrap applying constraint re2c@0.13:
==> [2021-10-11-16:04:12.231537] clingo-bootstrap applying constraint bison@2.5:
==> [2021-10-11-16:04:12.231810] bison applying constraint diffutils
==> [2021-10-11-16:04:12.232026] diffutils applying constraint libiconv
==> [2021-10-11-16:04:12.232289] bison applying constraint m4
==> [2021-10-11-16:04:12.232552] bison applying constraint perl
==> [2021-10-11-16:04:12.232793] perl applying constraint gdbm@:1.19
==> [2021-10-11-16:04:12.233056] gdbm applying constraint readline
==> [2021-10-11-16:04:12.233306] readline applying constraint ncurses
==> [2021-10-11-16:04:12.233493] ncurses applying constraint pkgconf
==> [2021-10-11-16:04:12.233776] perl applying constraint berkeley-db
==> [2021-10-11-16:04:12.234027] perl applying constraint bzip2
==> [2021-10-11-16:04:12.234260] bzip2 applying constraint diffutils
==> [2021-10-11-16:04:12.234516] perl applying constraint zlib
==> [2021-10-11-16:04:12.234846] clingo-bootstrap applying constraint python
==> [2021-10-11-16:04:12.353948] clingo-bootstrap applying constraint cmake@3.16.0:
==> [2021-10-11-16:04:12.356914] openssl applying constraint zlib
==> [2021-10-11-16:04:12.357216] openssl applying constraint perl@5.14.0:
==> [2021-10-11-16:04:12.357629] perl applying constraint gdbm@:1.19
==> [2021-10-11-16:04:12.358477] gdbm applying constraint readline
==> [2021-10-11-16:04:12.358891] readline applying constraint gnuconfig
==> [2021-10-11-16:04:12.359163] readline applying constraint ncurses
==> [2021-10-11-16:04:12.359608] ncurses applying constraint gnuconfig
==> [2021-10-11-16:04:12.359911] ncurses applying constraint pkgconf@1.8.0%apple-clang@12.0.5 arch=darwin-bigsur-aarch64
==> [2021-10-11-16:04:12.360324] pkgconf applying constraint gnuconfig
==> [2021-10-11-16:04:12.360689] gdbm applying constraint gnuconfig
==> [2021-10-11-16:04:12.360954] gdbm applying constraint readline
==> [2021-10-11-16:04:12.361288] perl applying constraint berkeley-db
==> [2021-10-11-16:04:12.361738] berkeley-db applying constraint gnuconfig
==> [2021-10-11-16:04:12.362037] perl applying constraint bzip2
==> [2021-10-11-16:04:12.362377] bzip2 applying constraint diffutils
==> [2021-10-11-16:04:12.362774] diffutils applying constraint gnuconfig
==> [2021-10-11-16:04:12.363039] diffutils applying constraint libiconv@1.16%apple-clang@12.0.5 libs=shared,static arch=darwin-bigsur-aarch64
==> [2021-10-11-16:04:12.363508] libiconv applying constraint gnuconfig
==> [2021-10-11-16:04:12.363860] perl applying constraint zlib
==> [2021-10-11-16:04:12.364253] perl applying constraint gdbm@:1.19
==> [2021-10-11-16:04:12.364609] perl applying constraint berkeley-db
==> [2021-10-11-16:04:12.364939] perl applying constraint bzip2
==> [2021-10-11-16:04:12.365316] perl applying constraint zlib
==> [2021-10-11-16:04:12.365591] openssl applying constraint zlib
==> [2021-10-11-16:04:12.365862] openssl applying constraint perl@5.14.0:
==> [2021-10-11-16:04:12.366253] cmake applying constraint ncurses
==> [2021-10-11-16:04:12.367961] cmake applying constraint openssl
==> [2021-10-11-16:04:12.368290] cmake applying constraint ncurses
==> [2021-10-11-16:04:12.368758] clingo-bootstrap applying constraint re2c@0.13:
==> [2021-10-11-16:04:12.369169] re2c applying constraint gnuconfig
==> [2021-10-11-16:04:12.369573] clingo-bootstrap applying constraint bison@2.5:
==> [2021-10-11-16:04:12.369997] bison applying constraint gnuconfig
==> [2021-10-11-16:04:12.370265] bison applying constraint diffutils
==> [2021-10-11-16:04:12.370560] bison applying constraint m4
==> [2021-10-11-16:04:12.370984] m4 applying constraint gnuconfig
==> [2021-10-11-16:04:12.372284] m4 applying constraint gnuconfig
==> [2021-10-11-16:04:12.372580] m4 applying constraint libsigsegv
==> [2021-10-11-16:04:12.372852] bison applying constraint perl
==> [2021-10-11-16:04:12.373268] bison applying constraint gnuconfig
==> [2021-10-11-16:04:12.373534] bison applying constraint diffutils
==> [2021-10-11-16:04:12.373825] bison applying constraint m4
==> [2021-10-11-16:04:12.374125] bison applying constraint perl
==> [2021-10-11-16:04:12.374536] clingo-bootstrap applying constraint python
==> [2021-10-11-16:04:12.375024] clingo-bootstrap applying constraint cmake@3.16.0:
==> [2021-10-11-16:04:12.375510] clingo-bootstrap applying constraint re2c@0.13:
==> [2021-10-11-16:04:12.375947] clingo-bootstrap applying constraint bison@2.5:
==> [2021-10-11-16:04:12.376411] clingo-bootstrap applying constraint python
==> [2021-10-11-16:04:12.408341] clingo-bootstrap applying constraint cmake@3.16.0:
==> [2021-10-11-16:04:12.410179] cmake applying constraint openssl
==> [2021-10-11-16:04:12.410539] openssl applying constraint zlib
==> [2021-10-11-16:04:12.410895] openssl applying constraint perl@5.14.0:
==> [2021-10-11-16:04:12.411322] perl applying constraint gdbm@:1.19
==> [2021-10-11-16:04:12.411748] gdbm applying constraint gnuconfig
==> [2021-10-11-16:04:12.412063] gdbm applying constraint readline
==> [2021-10-11-16:04:12.412504] readline applying constraint gnuconfig
==> [2021-10-11-16:04:12.412808] readline applying constraint ncurses
==> [2021-10-11-16:04:12.413249] ncurses applying constraint gnuconfig
==> [2021-10-11-16:04:12.413630] ncurses applying constraint pkgconf@1.8.0%apple-clang@12.0.5 arch=darwin-bigsur-aarch64
==> [2021-10-11-16:04:12.414055] pkgconf applying constraint gnuconfig
==> [2021-10-11-16:04:12.414409] perl applying constraint berkeley-db
==> [2021-10-11-16:04:12.414878] berkeley-db applying constraint gnuconfig
==> [2021-10-11-16:04:12.415214] perl applying constraint bzip2
==> [2021-10-11-16:04:12.415562] bzip2 applying constraint diffutils
==> [2021-10-11-16:04:12.415972] diffutils applying constraint gnuconfig
==> [2021-10-11-16:04:12.416263] diffutils applying constraint libiconv@1.16%apple-clang@12.0.5 libs=shared,static arch=darwin-bigsur-aarch64
==> [2021-10-11-16:04:12.416700] libiconv applying constraint gnuconfig
==> [2021-10-11-16:04:12.417094] perl applying constraint zlib
==> [2021-10-11-16:04:12.417550] cmake applying constraint ncurses
==> [2021-10-11-16:04:12.418027] clingo-bootstrap applying constraint re2c@0.13:
==> [2021-10-11-16:04:12.418443] re2c applying constraint gnuconfig
==> [2021-10-11-16:04:12.418895] clingo-bootstrap applying constraint bison@2.5:
==> [2021-10-11-16:04:12.419329] bison applying constraint gnuconfig
==> [2021-10-11-16:04:12.419638] bison applying constraint diffutils
==> [2021-10-11-16:04:12.419937] bison applying constraint m4
==> [2021-10-11-16:04:12.420374] m4 applying constraint gnuconfig
==> [2021-10-11-16:04:12.420695] m4 applying constraint libsigsegv
==> [2021-10-11-16:04:12.421154] libsigsegv applying constraint gnuconfig
==> [2021-10-11-16:04:12.421463] bison applying constraint perl
==> [2021-10-11-16:04:12.421874] clingo-bootstrap applying constraint python
==> [2021-10-11-16:04:12.431645] Ordered hashes [m4]: m4/44/bfdffa7c2eb01021d5849b36972c069693654ad826c1a20b53534009a4ec7a89, m4/45/9dc5fbd0d5cb1037ab1e6d0ecc74a30df218d0a94bdd5a02759a97f62daca573
==> [2021-10-11-16:04:12.431749] Ordered hashes [berkeley-db]: berkeley-db/63/b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522
==> [2021-10-11-16:04:12.431951] Ordered hashes [python]: python/22/0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87, python/23/f2fd060afc4b4618fe8104c4c5d771f36dc55b1db5a4623785a4ea707ec72fb4, python/25/4c2457325f2b608b1b6a2c63087df8c26e07db3e3d493caf36a56f0ecf6fb768
==> [2021-10-11-16:04:12.436756] [BOOTSTRAP MODULE clingo] Try installing 'clingo-bootstrap@spack+python %apple-clang target=aarch64' from sources
==> [2021-10-11-16:04:12.436852] Initializing the build queue from the build requests
==> [2021-10-11-16:04:12.436869] Initializing the build queue for clingo-bootstrap
==> [2021-10-11-16:04:12.438035] Processing dependencies for clingo-bootstrap-spack-7ctouvxrqj6vs2mw2txgl5jfsqfrnpuc: ('build', 'link', 'run')
==> [2021-10-11-16:04:12.438134] Pkg id gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6 has the following dependents:
==> [2021-10-11-16:04:12.438149] - libsigsegv-2.13-jx46dmox6wtqotnsvvcrqwx6jfuj72pc
==> [2021-10-11-16:04:12.438161] - bison-3.8.2-comfky2x72qyu32uakmlltap4sc5jfit
==> [2021-10-11-16:04:12.438171] - m4-1.4.19-ghuewuay4dqfxf35fzbqknakd5juhl7a
==> [2021-10-11-16:04:12.438182] - diffutils-3.8-tps25qichnyhdskh2yoimpoeersmou27
==> [2021-10-11-16:04:12.438192] - gdbm-1.19-tcvwmvv2iq7slyhayv5wswqpt7oy3ynw
==> [2021-10-11-16:04:12.438203] - readline-8.1-odmkrknw6jd6bu2rdomd5ivloae6gwig
==> [2021-10-11-16:04:12.438213] - re2c-2.2-ahy7ntwn3wjqkphsppo2cixexk62gmau
==> [2021-10-11-16:04:12.438223] - pkgconf-1.8.0-crd2i745cj5p6am2m5sajcycuvr3jonc
==> [2021-10-11-16:04:12.438234] - ncurses-6.2-exubvurzson2t7xztjvlg7gayp65kg7p
==> [2021-10-11-16:04:12.438244] - libiconv-1.16-fyaei4lfz3n7nzpna2vetg4a5uq7pdbk
==> [2021-10-11-16:04:12.438254] - berkeley-db-18.1.40-7q4rqdl4vtulxj4jx56l2mac2d2embea
==> [2021-10-11-16:04:12.442041] Removing failure marking for gnuconfig
==> [2021-10-11-16:04:12.442179] Pkg id libiconv-1.16-fyaei4lfz3n7nzpna2vetg4a5uq7pdbk has the following dependents:
==> [2021-10-11-16:04:12.442196] - diffutils-3.8-tps25qichnyhdskh2yoimpoeersmou27
==> [2021-10-11-16:04:12.444529] Removing failure marking for libiconv
==> [2021-10-11-16:04:12.444626] Pkg id diffutils-3.8-tps25qichnyhdskh2yoimpoeersmou27 has the following dependents:
==> [2021-10-11-16:04:12.444643] - bison-3.8.2-comfky2x72qyu32uakmlltap4sc5jfit
==> [2021-10-11-16:04:12.444656] - bzip2-1.0.8-xoi7soo72qnpkswxbdby3a4kmip2e3ro
==> [2021-10-11-16:04:12.446460] Removing failure marking for diffutils
==> [2021-10-11-16:04:12.446560] Pkg id libsigsegv-2.13-jx46dmox6wtqotnsvvcrqwx6jfuj72pc has the following dependents:
==> [2021-10-11-16:04:12.446577] - m4-1.4.19-ghuewuay4dqfxf35fzbqknakd5juhl7a
==> [2021-10-11-16:04:12.448850] Removing failure marking for libsigsegv
==> [2021-10-11-16:04:12.448929] Pkg id m4-1.4.19-ghuewuay4dqfxf35fzbqknakd5juhl7a has the following dependents:
==> [2021-10-11-16:04:12.448946] - bison-3.8.2-comfky2x72qyu32uakmlltap4sc5jfit
==> [2021-10-11-16:04:12.452764] Removing failure marking for m4
==> [2021-10-11-16:04:12.452897] Pkg id berkeley-db-18.1.40-7q4rqdl4vtulxj4jx56l2mac2d2embea has the following dependents:
==> [2021-10-11-16:04:12.452915] - perl-5.34.0-xzib42oxhft2bgakyhhovjy5o3lvip4w
==> [2021-10-11-16:04:12.455335] Removing failure marking for berkeley-db
==> [2021-10-11-16:04:12.455492] Pkg id bzip2-1.0.8-xoi7soo72qnpkswxbdby3a4kmip2e3ro has the following dependents:
==> [2021-10-11-16:04:12.455509] - perl-5.34.0-xzib42oxhft2bgakyhhovjy5o3lvip4w
==> [2021-10-11-16:04:12.458842] Removing failure marking for bzip2
==> [2021-10-11-16:04:12.459002] Pkg id pkgconf-1.8.0-crd2i745cj5p6am2m5sajcycuvr3jonc has the following dependents:
==> [2021-10-11-16:04:12.459019] - ncurses-6.2-exubvurzson2t7xztjvlg7gayp65kg7p
==> [2021-10-11-16:04:12.460557] Removing failure marking for pkgconf
==> [2021-10-11-16:04:12.460647] Pkg id ncurses-6.2-exubvurzson2t7xztjvlg7gayp65kg7p has the following dependents:
==> [2021-10-11-16:04:12.460663] - readline-8.1-odmkrknw6jd6bu2rdomd5ivloae6gwig
==> [2021-10-11-16:04:12.460676] - cmake-3.21.3-3ipwxqcnq7h3gvyjuu65z62gunv52xjs
==> [2021-10-11-16:04:12.463761] Removing failure marking for ncurses
==> [2021-10-11-16:04:12.463845] Pkg id readline-8.1-odmkrknw6jd6bu2rdomd5ivloae6gwig has the following dependents:
==> [2021-10-11-16:04:12.463862] - gdbm-1.19-tcvwmvv2iq7slyhayv5wswqpt7oy3ynw
==> [2021-10-11-16:04:12.465553] Removing failure marking for readline
==> [2021-10-11-16:04:12.465633] Pkg id gdbm-1.19-tcvwmvv2iq7slyhayv5wswqpt7oy3ynw has the following dependents:
==> [2021-10-11-16:04:12.465650] - perl-5.34.0-xzib42oxhft2bgakyhhovjy5o3lvip4w
==> [2021-10-11-16:04:12.467355] Removing failure marking for gdbm
==> [2021-10-11-16:04:12.467462] Pkg id zlib-1.2.11-a3p23rnymc7d6ens3mk7wj75r5sucdvr has the following dependents:
==> [2021-10-11-16:04:12.467478] - openssl-1.1.1l-xq4viuhggqsoqii6osjf5k25ugjizejv
==> [2021-10-11-16:04:12.467490] - perl-5.34.0-xzib42oxhft2bgakyhhovjy5o3lvip4w
==> [2021-10-11-16:04:12.468951] Pkg id perl-5.34.0-xzib42oxhft2bgakyhhovjy5o3lvip4w has the following dependents:
==> [2021-10-11-16:04:12.468970] - openssl-1.1.1l-xq4viuhggqsoqii6osjf5k25ugjizejv
==> [2021-10-11-16:04:12.468983] - bison-3.8.2-comfky2x72qyu32uakmlltap4sc5jfit
==> [2021-10-11-16:04:12.474491] Removing failure marking for perl
==> [2021-10-11-16:04:12.474588] Pkg id bison-3.8.2-comfky2x72qyu32uakmlltap4sc5jfit has the following dependents:
==> [2021-10-11-16:04:12.474604] - clingo-bootstrap-spack-7ctouvxrqj6vs2mw2txgl5jfsqfrnpuc
==> [2021-10-11-16:04:12.477075] Removing failure marking for bison
==> [2021-10-11-16:04:12.477173] Pkg id openssl-1.1.1l-xq4viuhggqsoqii6osjf5k25ugjizejv has the following dependents:
==> [2021-10-11-16:04:12.477190] - cmake-3.21.3-3ipwxqcnq7h3gvyjuu65z62gunv52xjs
==> [2021-10-11-16:04:12.481008] Removing failure marking for openssl
==> [2021-10-11-16:04:12.481221] Pkg id cmake-3.21.3-3ipwxqcnq7h3gvyjuu65z62gunv52xjs has the following dependents:
==> [2021-10-11-16:04:12.481241] - clingo-bootstrap-spack-7ctouvxrqj6vs2mw2txgl5jfsqfrnpuc
==> [2021-10-11-16:04:12.486528] Removing failure marking for cmake
==> [2021-10-11-16:04:12.486657] Pkg id python-3.8-yumcagn3zdktmhjepxgxvbokqdl24zaz has the following dependents:
==> [2021-10-11-16:04:12.486674] - clingo-bootstrap-spack-7ctouvxrqj6vs2mw2txgl5jfsqfrnpuc
==> [2021-10-11-16:04:12.498111] Pkg id re2c-2.2-ahy7ntwn3wjqkphsppo2cixexk62gmau has the following dependents:
==> [2021-10-11-16:04:12.498134] - clingo-bootstrap-spack-7ctouvxrqj6vs2mw2txgl5jfsqfrnpuc
==> [2021-10-11-16:04:12.499392] Removing failure marking for re2c
==> [2021-10-11-16:04:12.501576] Removing failure marking for clingo-bootstrap
==> [2021-10-11-16:04:12.501652] Pkg id clingo-bootstrap-spack-7ctouvxrqj6vs2mw2txgl5jfsqfrnpuc has the following dependents:
==> [2021-10-11-16:04:12.501792] Ensure all dependencies know all dependents across specs
==> [2021-10-11-16:04:12.502873] Acquiring a write lock on gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6 with timeout 1e-09
==> [2021-10-11-16:04:12.503354] Installing gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6
==> [2021-10-11-16:04:12.503377] Searching for binary cache of gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6
==> [2021-10-11-16:04:12.503425] Reading config file /Users/certik/repos/spack/etc/spack/defaults/mirrors.yaml
==> [2021-10-11-16:04:12.521719] Did not find darwin-bigsur-aarch64-apple-clang-12.0.5-gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6.spec.json on https://mirror.spack.io/build_cache/darwin-bigsur-aarch64-apple-clang-12.0.5-gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6.spec.json
  Download failed: <urlopen error [Errno 8] nodename nor servname provided, or not known>
==> [2021-10-11-16:04:12.521763] Did not find darwin-bigsur-aarch64-apple-clang-12.0.5-gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6.spec.json on https://mirror.spack.io/build_cache/darwin-bigsur-aarch64-apple-clang-12.0.5-gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6.spec.yaml
  Download failed: <urlopen error [Errno 8] nodename nor servname provided, or not known>
==> [2021-10-11-16:04:12.521808] No binary for gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6 found: installing from source
==> [2021-10-11-16:04:12.528792] '/usr/bin/git' 'describe' '--tags' '--match' 'v*'
==> [2021-10-11-16:04:12.941768] Error: FetchError: All fetchers failed for spack-stage-gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6

/Users/certik/repos/spack/lib/spack/spack/package.py:1379, in do_fetch:
       1376
       1377        self.stage.create()
       1378        err_msg = None if not self.manual_download else self.download_instr
  >>   1379        start_time = time.time()
       1380        self.stage.fetch(mirror_only, err_msg=err_msg)
       1381        self._fetch_time = time.time() - start_time
       1382


Traceback (most recent call last):
  File "/Users/certik/repos/spack/lib/spack/spack/build_environment.py", line 1028, in _setup_pkg_and_run
    return_value = function(pkg, kwargs)
  File "/Users/certik/repos/spack/lib/spack/spack/installer.py", line 1956, in build_process
    return installer.run()
  File "/Users/certik/repos/spack/lib/spack/spack/installer.py", line 1784, in run
    self.pkg.do_patch()
  File "/Users/certik/repos/spack/lib/spack/spack/package.py", line 1416, in do_patch
    self.do_stage()
  File "/Users/certik/repos/spack/lib/spack/spack/package.py", line 1401, in do_stage
    self.do_fetch(mirror_only)
  File "/Users/certik/repos/spack/lib/spack/spack/package.py", line 1380, in do_fetch
    self.stage.fetch(mirror_only, err_msg=err_msg)
  File "/Users/certik/repos/spack/lib/spack/spack/util/pattern.py", line 22, in __call__
    return [getattr(item, self.name)(*args, **kwargs)
  File "/Users/certik/repos/spack/lib/spack/spack/util/pattern.py", line 22, in <listcomp>
    return [getattr(item, self.name)(*args, **kwargs)
  File "/Users/certik/repos/spack/lib/spack/spack/stage.py", line 508, in fetch
    raise fs.FetchError(err_msg or default_msg, None)
spack.fetch_strategy.FetchError: All fetchers failed for spack-stage-gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6
==> [2021-10-11-16:04:12.942105] Flagging gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6 as failed: FetchError: All fetchers failed for spack-stage-gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6
==> [2021-10-11-16:04:12.943007] Warning: Skipping build of libsigsegv-2.13-jx46dmox6wtqotnsvvcrqwx6jfuj72pc since gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6 failed
==> [2021-10-11-16:04:12.943030] Flagging libsigsegv-2.13-jx46dmox6wtqotnsvvcrqwx6jfuj72pc as failed
==> [2021-10-11-16:04:12.943511] Warning: Skipping build of m4-1.4.19-ghuewuay4dqfxf35fzbqknakd5juhl7a since libsigsegv-2.13-jx46dmox6wtqotnsvvcrqwx6jfuj72pc failed
==> [2021-10-11-16:04:12.943529] Flagging m4-1.4.19-ghuewuay4dqfxf35fzbqknakd5juhl7a as failed
==> [2021-10-11-16:04:12.943892] Warning: Skipping build of bison-3.8.2-comfky2x72qyu32uakmlltap4sc5jfit since m4-1.4.19-ghuewuay4dqfxf35fzbqknakd5juhl7a failed
==> [2021-10-11-16:04:12.943911] Flagging bison-3.8.2-comfky2x72qyu32uakmlltap4sc5jfit as failed
==> [2021-10-11-16:04:12.944319] Warning: Skipping build of clingo-bootstrap-spack-7ctouvxrqj6vs2mw2txgl5jfsqfrnpuc since bison-3.8.2-comfky2x72qyu32uakmlltap4sc5jfit failed
==> [2021-10-11-16:04:12.944340] Flagging clingo-bootstrap-spack-7ctouvxrqj6vs2mw2txgl5jfsqfrnpuc as failed
==> [2021-10-11-16:04:12.944790] Warning: Skipping build of diffutils-3.8-tps25qichnyhdskh2yoimpoeersmou27 since gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6 failed
==> [2021-10-11-16:04:12.944815] Flagging diffutils-3.8-tps25qichnyhdskh2yoimpoeersmou27 as failed
==> [2021-10-11-16:04:12.945175] Warning: Skipping build of bzip2-1.0.8-xoi7soo72qnpkswxbdby3a4kmip2e3ro since diffutils-3.8-tps25qichnyhdskh2yoimpoeersmou27 failed
==> [2021-10-11-16:04:12.945195] Flagging bzip2-1.0.8-xoi7soo72qnpkswxbdby3a4kmip2e3ro as failed
==> [2021-10-11-16:04:12.945585] Warning: Skipping build of perl-5.34.0-xzib42oxhft2bgakyhhovjy5o3lvip4w since bzip2-1.0.8-xoi7soo72qnpkswxbdby3a4kmip2e3ro failed
==> [2021-10-11-16:04:12.945616] Flagging perl-5.34.0-xzib42oxhft2bgakyhhovjy5o3lvip4w as failed
==> [2021-10-11-16:04:12.946376] Warning: Skipping build of openssl-1.1.1l-xq4viuhggqsoqii6osjf5k25ugjizejv since perl-5.34.0-xzib42oxhft2bgakyhhovjy5o3lvip4w failed
==> [2021-10-11-16:04:12.946394] Flagging openssl-1.1.1l-xq4viuhggqsoqii6osjf5k25ugjizejv as failed
==> [2021-10-11-16:04:12.946705] Warning: Skipping build of cmake-3.21.3-3ipwxqcnq7h3gvyjuu65z62gunv52xjs since openssl-1.1.1l-xq4viuhggqsoqii6osjf5k25ugjizejv failed
==> [2021-10-11-16:04:12.946723] Flagging cmake-3.21.3-3ipwxqcnq7h3gvyjuu65z62gunv52xjs as failed
==> [2021-10-11-16:04:12.947167] Warning: Skipping build of gdbm-1.19-tcvwmvv2iq7slyhayv5wswqpt7oy3ynw since gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6 failed
==> [2021-10-11-16:04:12.947183] Flagging gdbm-1.19-tcvwmvv2iq7slyhayv5wswqpt7oy3ynw as failed
==> [2021-10-11-16:04:12.947536] Warning: Skipping build of readline-8.1-odmkrknw6jd6bu2rdomd5ivloae6gwig since gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6 failed
==> [2021-10-11-16:04:12.947553] Flagging readline-8.1-odmkrknw6jd6bu2rdomd5ivloae6gwig as failed
==> [2021-10-11-16:04:12.947836] Warning: Skipping build of re2c-2.2-ahy7ntwn3wjqkphsppo2cixexk62gmau since gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6 failed
==> [2021-10-11-16:04:12.947852] Flagging re2c-2.2-ahy7ntwn3wjqkphsppo2cixexk62gmau as failed
==> [2021-10-11-16:04:12.948060] Warning: Skipping build of pkgconf-1.8.0-crd2i745cj5p6am2m5sajcycuvr3jonc since gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6 failed
==> [2021-10-11-16:04:12.948075] Flagging pkgconf-1.8.0-crd2i745cj5p6am2m5sajcycuvr3jonc as failed
==> [2021-10-11-16:04:12.948287] Warning: Skipping build of ncurses-6.2-exubvurzson2t7xztjvlg7gayp65kg7p since pkgconf-1.8.0-crd2i745cj5p6am2m5sajcycuvr3jonc failed
==> [2021-10-11-16:04:12.948302] Flagging ncurses-6.2-exubvurzson2t7xztjvlg7gayp65kg7p as failed
==> [2021-10-11-16:04:12.948518] Warning: Skipping build of libiconv-1.16-fyaei4lfz3n7nzpna2vetg4a5uq7pdbk since gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6 failed
==> [2021-10-11-16:04:12.948534] Flagging libiconv-1.16-fyaei4lfz3n7nzpna2vetg4a5uq7pdbk as failed
==> [2021-10-11-16:04:12.948762] Warning: Skipping build of berkeley-db-18.1.40-7q4rqdl4vtulxj4jx56l2mac2d2embea since gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6 failed
==> [2021-10-11-16:04:12.948777] Flagging berkeley-db-18.1.40-7q4rqdl4vtulxj4jx56l2mac2d2embea as failed
==> [2021-10-11-16:04:12.950695] Creating stage lock spack-stage-gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6
==> [2021-10-11-16:04:12.950750] [BOOTSTRAP MODULE clingo] Unexpected error "Terminating after first install failure: FetchError: All fetchers failed for spack-stage-gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6"
==> [2021-10-11-16:04:12.951678] Reading config file /Users/certik/repos/spack/etc/spack/defaults/config.yaml
Traceback (most recent call last):
  File "/Users/certik/bin/spack", line 100, in <module>
    sys.exit(spack.main.main())
  File "/Users/certik/repos/spack/lib/spack/spack/main.py", line 814, in main
    return _invoke_command(command, parser, args, unknown)
  File "/Users/certik/repos/spack/lib/spack/spack/main.py", line 527, in _invoke_command
    return_val = command(parser, args)
  File "/Users/certik/repos/spack/lib/spack/spack/cmd/spec.py", line 109, in spec
    spec.concretize()
  File "/Users/certik/repos/spack/lib/spack/spack/spec.py", line 2626, in concretize
    self._new_concretize(tests)
  File "/Users/certik/repos/spack/lib/spack/spack/spec.py", line 2596, in _new_concretize
    result = spack.solver.asp.solve([self], tests=tests)
  File "/Users/certik/repos/spack/lib/spack/spack/solver/asp.py", line 1679, in solve
    driver = PyclingoDriver()
  File "/Users/certik/repos/spack/lib/spack/spack/solver/asp.py", line 292, in __init__
    spack.bootstrap.ensure_clingo_importable_or_raise()
  File "/Users/certik/repos/spack/lib/spack/spack/bootstrap.py", line 625, in ensure_clingo_importable_or_raise
    ensure_module_importable_or_raise(
  File "/Users/certik/repos/spack/lib/spack/spack/bootstrap.py", line 418, in ensure_module_importable_or_raise
    raise ImportError(msg)
ImportError: cannot bootstrap the "clingo" Python module from spec "clingo-bootstrap@spack+python %apple-clang target=aarch64" due to the following failures:
    'spack-install' raised InstallError: Terminating after first install failure: FetchError: All fetchers failed for spack-stage-gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6
    Please run `spack -d spec zlib` for more verbose error messages

Information on your system

  • Spack: 0.16.3-4805-95e63e0c29
  • Python: 3.8.2
  • Platform: darwin-bigsur-m1
  • Concretizer: clingo

General information

  • I have run spack debug report and reported the version of Spack/Python/Platform
  • I have searched the issues of this repo and believe this is not a duplicate
  • I have run the failing commands in debug mode and reported the output

About this issue

  • Original URL
  • State: open
  • Created 5 years ago
  • Comments: 16 (13 by maintainers)

Most upvoted comments

You could try if curl works, through spack config edit config:

  # The default url fetch method to use.
  # If set to 'curl', Spack will require curl on the user's system
  # If set to 'urllib', Spack will use python built-in libs to fetch
  url_fetch_method: curl

If you could share your findings on why urllib does not work that’d be interesting, but I don’t think I can help with macos.

Oh no, it is very much related! This is hours of debugging behind this. I am documenting what is going on now.

Anyway, regarding the config, it seems that spack config edit config produces a broken config file. That seems like another bug.

However, I can simply edit the config file ~/.spack/config.yam by hand like this:

config:
  url_fetch_method: curl

and this works!!

Thank you @haampie for this.

Btw, I also tried:

config:
  verify_ssl: false

But that gives the same error as above.

I think that the Python’s urllib is somehow broken.

Ok, now packages download, but I hit another error:

$ spack spec zlib        
Input spec
--------------------------------
zlib

Concretized
--------------------------------
==> Bootstrapping clingo from sources
==> Installing gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6
==> No binary for gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6 found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/69/69b6d2868e70167ba1bdb9030b49beeb20f00b37e30825e83fd04291d96bc5f7.tar.gz
######################################################################### 100.0%
==> No patches needed for gnuconfig
==> gnuconfig: Executing phase: 'install'
==> gnuconfig: Successfully installed gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6
  Fetch: 1.22s.  Build: 0.16s.  Total: 1.38s.
[+] /Users/certik/.spack/bootstrap/store/darwin-bigsur-aarch64/apple-clang-12.0.5/gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6
==> Installing zlib-1.2.11-a3p23rnymc7d6ens3mk7wj75r5sucdvr
==> No binary for zlib-1.2.11-a3p23rnymc7d6ens3mk7wj75r5sucdvr found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/c3/c3e5e9fdd5004dcb542feda5ee4f0ff0744628baf8ed2dd5d66f8ca1197cb1a1.tar.gz
######################################################################### 100.0%
==> No patches needed for zlib
==> zlib: Executing phase: 'install'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j8'

4 errors found in build log:
     19    /Users/certik/repos/spack/lib/spack/env/clang/clang -fPIC -O2 -DHAVE
           _HIDDEN  -c -o deflate.o deflate.c
     20    /Users/certik/repos/spack/lib/spack/env/clang/clang -fPIC -O2 -DHAVE
           _HIDDEN  -c -o infback.o infback.c
     21    /Users/certik/repos/spack/lib/spack/env/clang/clang -fPIC -O2 -DHAVE
           _HIDDEN  -c -o inffast.o inffast.c
     22    /Users/certik/repos/spack/lib/spack/env/clang/clang -fPIC -O2 -DHAVE
           _HIDDEN  -c -o inflate.o inflate.c
     23    /Users/certik/repos/spack/lib/spack/env/clang/clang -fPIC -O2 -DHAVE
           _HIDDEN  -c -o inftrees.o inftrees.c
     24    /Users/certik/repos/spack/lib/spack/env/clang/clang -fPIC -O2 -DHAVE
           _HIDDEN  -c -o trees.o trees.c
  >> 25    make: /Users/certik/repos/spack/lib/spack/env/clang/clang: Permissio
           n denied
     26    /Users/certik/repos/spack/lib/spack/env/clang/clang -fPIC -O2 -DHAVE
           _HIDDEN  -c -o zutil.o zutil.c
  >> 27    make: /Users/certik/repos/spack/lib/spack/env/clang/clang: Permissio
           n denied
  >> 28    make: *** [trees.o] Error 1
     29    make: *** Waiting for unfinished jobs....
  >> 30    make: *** [zutil.o] Error 1

See build log for details:
  /var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc/T/certik/spack-stage/spack-stage-zlib-1.2.11-a3p23rnymc7d6ens3mk7wj75r5sucdvr/spack-build-out.txt

==> Warning: Skipping build of perl-5.34.0-xzib42oxhft2bgakyhhovjy5o3lvip4w since zlib-1.2.11-a3p23rnymc7d6ens3mk7wj75r5sucdvr failed
==> Warning: Skipping build of bison-3.8.2-comfky2x72qyu32uakmlltap4sc5jfit since perl-5.34.0-xzib42oxhft2bgakyhhovjy5o3lvip4w failed
==> Warning: Skipping build of clingo-bootstrap-spack-7ctouvxrqj6vs2mw2txgl5jfsqfrnpuc since bison-3.8.2-comfky2x72qyu32uakmlltap4sc5jfit failed
==> Warning: Skipping build of openssl-1.1.1l-xq4viuhggqsoqii6osjf5k25ugjizejv since perl-5.34.0-xzib42oxhft2bgakyhhovjy5o3lvip4w failed
==> Warning: Skipping build of cmake-3.21.3-3ipwxqcnq7h3gvyjuu65z62gunv52xjs since openssl-1.1.1l-xq4viuhggqsoqii6osjf5k25ugjizejv failed
==> Error: cannot bootstrap the "clingo" Python module from spec "clingo-bootstrap@spack+python %apple-clang target=aarch64" due to the following failures:
    'spack-install' raised InstallError: Terminating after first install failure: ProcessError: Command exited with status 2:
    'make' '-j8'
    Please run `spack -d spec zlib` for more verbose error messages

This one has been fixed by doing:

spack install -j1 zlib

Regarding the error, I tried to replicate what Spack is doing directly, but I think I must be doing something wrong, because I get a different error:

~ $ python3        
Python 3.8.2 (default, Jun  8 2021, 11:59:35) 
[Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib.request
>>> req = urllib.request.Request("https://mirror.spack.io/_source-cache/archive/69/69b6d2868e70167ba1bdb9030b49beeb20f00b37e30825e83fd04291d96bc5f7.tar.gz")
>>> req.get_method = lambda: "GET"
>>> import ssl
>>> context = ssl.create_default_context()
>>> urllib.request.urlopen(req, context=context)
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 1319, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/http/client.py", line 1245, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/http/client.py", line 1291, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/http/client.py", line 1240, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/http/client.py", line 1008, in _send_output
    self.send(msg)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/http/client.py", line 948, in send
    self.connect()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/http/client.py", line 919, in connect
    self.sock = self._create_connection(
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/socket.py", line 787, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 1348, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 1322, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>

Those are some really unhelpful error messages coming from Python. 😦 An exception in exception seems like a bug in Python itself. I assume I called it with incorrect arguments, but still.

So, I went to another computer and just figured out how to get it working:

>>> import urllib.request
>>> urllib.request.urlopen("https://mirror.spack.io/_source-cache/archive/69/69b6d2868e70167ba1bdb9030b49beeb20f00b37e30825e83fd04291d96bc5f7.tar.gz")
<http.client.HTTPResponse object at 0x10af080d0>

This is how it is supposed to look. Now exactly these commands on my new Apple M1 give:

$ python3
Python 3.8.2 (default, Jun  8 2021, 11:59:35) 
[Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib.request
>>> urllib.request.urlopen("https://mirror.spack.io/_source-cache/archive/69/69b6d2868e70167ba1bdb9030b49beeb20f00b37e30825e83fd04291d96bc5f7.tar.gz")
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 1319, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/http/client.py", line 1245, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/http/client.py", line 1291, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/http/client.py", line 1240, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/http/client.py", line 1008, in _send_output
    self.send(msg)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/http/client.py", line 948, in send
    self.connect()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/http/client.py", line 1407, in connect
    super().connect()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/http/client.py", line 919, in connect
    self.sock = self._create_connection(
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/socket.py", line 787, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 1362, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 1322, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>
>>> 

So the default Python installation seems broken.

I think urllib is giving the above set of errors, and then spack thinks that urllib failed to verify the url and prints “Urllib fetch failed to verify url” which mislead me to think it is about ssl certificates. I think it is just some basic connection problem via urllib.

@haampie thank you. Any idea what is actually causing it?

Let’s keep going. The first url actually works in my browser: https://mirror.spack.io/_source-cache/archive/69/69b6d2868e70167ba1bdb9030b49beeb20f00b37e30825e83fd04291d96bc5f7.tar.gz, so let’s figure out which place in Spack is downloading it:

diff --git a/lib/spack/spack/fetch_strategy.py b/lib/spack/spack/fetch_strategy.py
index 6db41e2328..68d6382b86 100644
--- a/lib/spack/spack/fetch_strategy.py
+++ b/lib/spack/spack/fetch_strategy.py
@@ -311,6 +311,7 @@ def candidate_urls(self):
 
     @_needs_stage
     def fetch(self):
+        print("URLLIB...")
         if self.archive_file:
             tty.debug('Already downloaded {0}'.format(self.archive_file))
             return

This prints:

Trying:  /Users/certik/repos/spack/var/spack/cache/_source-cache/archive/69/69b6d2868e70167ba1bdb9030b49beeb20f00b37e30825e83fd04291d96bc5f7.tar.gz
Cache failed
Trying:  /Users/certik/repos/spack/var/spack/cache/gnuconfig/gnuconfig-2021-08-14.tar.gz
Cache failed
Trying:  https://mirror.spack.io/_source-cache/archive/69/69b6d2868e70167ba1bdb9030b49beeb20f00b37e30825e83fd04291d96bc5f7.tar.gz
URLLIB...
ERROR
Failed to fetch file from URL: https://mirror.spack.io/_source-cache/archive/69/69b6d2868e70167ba1bdb9030b49beeb20f00b37e30825e83fd04291d96bc5f7.tar.gz
    Urllib fetch failed to verify url https://mirror.spack.io/_source-cache/archive/69/69b6d2868e70167ba1bdb9030b49beeb20f00b37e30825e83fd04291d96bc5f7.tar.gz

The error messages should be improved. I instrumented the code like this:

diff --git a/lib/spack/spack/stage.py b/lib/spack/spack/stage.py
index 28feb05365..8f0fa993fc 100644
--- a/lib/spack/spack/stage.py
+++ b/lib/spack/spack/stage.py
@@ -487,16 +487,20 @@ def print_errors(errors):
                 tty.debug(msg)
 
         errors = []
+        print("XXX")
         for fetcher in generate_fetchers():
+            print("Trying: ", fetcher)
             try:
                 fetcher.stage = self
                 self.fetcher = fetcher
                 self.fetcher.fetch()
                 break
             except spack.fetch_strategy.NoCacheError:
+                print("Cache failed")
                 # Don't bother reporting when something is not cached.
                 continue
             except spack.error.SpackError as e:
+                print("ERROR")
                 errors.append('Fetching from {0} failed.'.format(fetcher))
                 tty.debug(e)
                 continue
@@ -505,6 +509,7 @@ def print_errors(errors):
 
             self.fetcher = self.default_fetcher
             default_msg = 'All fetchers failed for {0}'.format(self.name)
+            print("2 :(")
             raise fs.FetchError(err_msg or default_msg, None)
 
         print_errors(errors)

Now it gives the following output:

$ spack spec zlib
Input spec
--------------------------------
zlib

Concretized
--------------------------------
==> Bootstrapping clingo from sources
==> Installing gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6
==> No binary for gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6 found: installing from source
XXX
Trying:  /Users/certik/repos/spack/var/spack/cache/_source-cache/archive/69/69b6d2868e70167ba1bdb9030b49beeb20f00b37e30825e83fd04291d96bc5f7.tar.gz
Cache failed
Trying:  /Users/certik/repos/spack/var/spack/cache/gnuconfig/gnuconfig-2021-08-14.tar.gz
Cache failed
Trying:  https://mirror.spack.io/_source-cache/archive/69/69b6d2868e70167ba1bdb9030b49beeb20f00b37e30825e83fd04291d96bc5f7.tar.gz
ERROR
Trying:  https://mirror.spack.io/gnuconfig/gnuconfig-2021-08-14.tar.gz
ERROR
Trying:  https://github.com/spack/gnuconfig/releases/download/2021-08-14/gnuconfig-2021-08-14.tar.gz
ERROR
2 :(
==> Error: FetchError: All fetchers failed for spack-stage-gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6
...

So we can see that it is trying 5 fetchers, the first two try a local cache and we ignore the error. Fine.

The other three fail. But why? The errors are not reported. Let’s see:

diff --git a/lib/spack/spack/stage.py b/lib/spack/spack/stage.py
index 28feb05365..996c5e7e23 100644
--- a/lib/spack/spack/stage.py
+++ b/lib/spack/spack/stage.py
@@ -487,17 +487,22 @@ def print_errors(errors):
                 tty.debug(msg)
 
         errors = []
+        print("XXX")
         for fetcher in generate_fetchers():
+            print("Trying: ", fetcher)
             try:
                 fetcher.stage = self
                 self.fetcher = fetcher
                 self.fetcher.fetch()
                 break
             except spack.fetch_strategy.NoCacheError:
+                print("Cache failed")
                 # Don't bother reporting when something is not cached.
                 continue
             except spack.error.SpackError as e:
+                print("ERROR")
                 errors.append('Fetching from {0} failed.'.format(fetcher))
+                print(e)
                 tty.debug(e)
                 continue
         else:

Now it prints:

Trying:  https://mirror.spack.io/_source-cache/archive/69/69b6d2868e70167ba1bdb9030b49beeb20f00b37e30825e83fd04291d96bc5f7.tar.gz
ERROR
Failed to fetch file from URL: https://mirror.spack.io/_source-cache/archive/69/69b6d2868e70167ba1bdb9030b49beeb20f00b37e30825e83fd04291d96bc5f7.tar.gz
    Urllib fetch failed to verify url https://mirror.spack.io/_source-cache/archive/69/69b6d2868e70167ba1bdb9030b49beeb20f00b37e30825e83fd04291d96bc5f7.tar.gz
Trying:  https://mirror.spack.io/gnuconfig/gnuconfig-2021-08-14.tar.gz
ERROR
Failed to fetch file from URL: https://mirror.spack.io/gnuconfig/gnuconfig-2021-08-14.tar.gz
    Urllib fetch failed to verify url https://mirror.spack.io/gnuconfig/gnuconfig-2021-08-14.tar.gz
Trying:  https://github.com/spack/gnuconfig/releases/download/2021-08-14/gnuconfig-2021-08-14.tar.gz
ERROR
Failed to fetch file from URL: https://github.com/spack/gnuconfig/releases/download/2021-08-14/gnuconfig-2021-08-14.tar.gz
    Urllib fetch failed to verify url https://github.com/spack/gnuconfig/releases/download/2021-08-14/gnuconfig-2021-08-14.tar.gz

I see. That is useful. So the urllib seems to fail over https. That seems like a certificate problem.