spack: Killed a build, now Spack thinks it is installed
I just tried building a package, noticed something was wrong during configure, and killed it with <kbd>Ctrl</kbd>+<kbd>C</kbd>. When I ran spack purge and tried installing it again, I saw:
==> parallel-netcdf is already installed in /blues/gpfs/home/software/spack-0.10.0/opt/spack/linux-centos6-x86_64/intel-17.0.0/parallel-netcdf-1.6.1-fq2r43zk2wcbpl3hrc5h6sfz2dwajbk7
Traceback (most recent call last):
File "/soft/spack-0.10.0/bin/spack", line 198, in <module>
main()
File "/soft/spack-0.10.0/bin/spack", line 175, in main
return_val = command(parser, args)
File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/cmd/install.py", line 346, in install
package.do_install(**kwargs)
File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/package.py", line 1161, in do_install
rec = spack.store.db.get_record(self.spec)
File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/database.py", line 81, in converter
return function(self, spec_like, *args, **kwargs)
File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/database.py", line 554, in get_record
key = self._get_matching_spec_key(spec, **kwargs)
File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/database.py", line 549, in _get_matching_spec_key
raise KeyError("No such spec in database! %s" % spec)
KeyError: 'No such spec in database! parallel-netcdf@1.6.1%intel@17.0.0 cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX"+cxx+fortran+fpic arch=linux-centos6-x86_64^bison@3.0.4%intel@17.0.0 cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX" arch=linux-centos6-x86_64^libpciaccess@0.13.4%intel@17.0.0 cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX" arch=linux-centos6-x86_64^libsigsegv@2.10%gcc@6.1.0 arch=linux-centos6-x86_64^libtool@2.4.6%gcc@6.1.0 arch=linux-centos6-x86_64^m4@1.4.17%gcc@6.1.0+sigsegv arch=linux-centos6-x86_64^mvapich2@2.2%intel@17.0.0 cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX"~debug~gforker+hydra+mrail~nemesis~nemesisib~nemesisibtcp~psm~remshell~slurm~sock arch=linux-centos6-x86_64^pkg-config@0.29.1%gcc@6.1.0+internal_glib arch=linux-centos6-x86_64^util-macros@1.19.0%gcc@6.1.0 arch=linux-centos6-x86_64'
When I ran spack reindex and tried installing again, I saw:
==> parallel-netcdf is already installed in /blues/gpfs/home/software/spack-0.10.0/opt/spack/linux-centos6-x86_64/intel-17.0.0/parallel-netcdf-1.6.1-fq2r43zk2wcbpl3hrc5h6sfz2dwajbk7
Of course, that directory is empty except for .spack/spec.yaml.
About this issue
- Original URL
- State: closed
- Created 10 years ago
- Comments: 30 (30 by maintainers)
@adamjstewart:
There’s a more complicated discussion here on tests that run in the build directory, tests that run from an install directory, etc. and how we should record whether tests pass. I’d like to factor
--run-testsout into some separate methods for different packages and rework thespack testcommand to be able to test packages. Then we could have separate.spack/status/installed,.spack/status/tested, etc. files. It wouldn’t be horrible to store that stuff in the DB either (though i like to keep a backup in the.spackdirectories in case we need to regenerate the DB).Anyway, it sounds like you want unconditional removal by default, with an option to keep. Would it make sense to reuse the
--keep-prefixoption for this? Currently it skips cleanup and keeps the prefix on install failure. I suggest we make it also keep the prefix if it’s left after a prior build failure.I’m self-assigning this. I can start looking at the issue this week (and hopefully a PR by this Friday). If someone else was looking at this let me know.
I’m thinking the most straightforward approach would be to create an additional file in the
.spackmetadata directory in the package prefix to indicate that the install (and post-install hooks) completed (e.g..spack/install_marker, which could be empty). If a command line option is enabled then a package prefix will be removed whenever it does not contain this file.I think it would also be possible to prevent these issues by altering the SIGINT handler during the installation (starting before the point the prefix is created) but it would be harder to do so in a straightforward manner.
@tgamblin @becker33 curious if you have thoughts/preference between the above two strategies.
Almost all of my builds are as batch jobs and I encounter this several times a week. It’s currently my least favorite thing about Spack.
I agree. Now that the basic features of Spack are more solid we need to focus a bit more on polishing the UI, which among other things means not having behaviors like these.
And never fail with a stacktrace, just reassuring messages 😆
Yeah, I think these are at least 3 different problems, all with similar symptoms as a result. I know most of these are crazy, but I don’t want other users to run into these and not know how to solve them.
I think this is a different problem than before. Before, you were probably giving <kbd>Ctrl</kbd>+<kbd>C</kbd> while here:
which is called as part of an exceptional path:
and this is not handled gracefully. I still have to figure out what could be the sequence causing the last problem you posted.
This is just a use case. I really wouldn’t be offended if we cleaned up by default and there was no option to preserve the files. But
--keep-prefixsounds good to me as far as UI goes.