spack: Killed a build, now Spack thinks it is installed

I just tried building a package, noticed something was wrong during configure, and killed it with <kbd>Ctrl</kbd>+<kbd>C</kbd>. When I ran spack purge and tried installing it again, I saw:

==> parallel-netcdf is already installed in /blues/gpfs/home/software/spack-0.10.0/opt/spack/linux-centos6-x86_64/intel-17.0.0/parallel-netcdf-1.6.1-fq2r43zk2wcbpl3hrc5h6sfz2dwajbk7
Traceback (most recent call last):
  File "/soft/spack-0.10.0/bin/spack", line 198, in <module>
    main()
  File "/soft/spack-0.10.0/bin/spack", line 175, in main
    return_val = command(parser, args)
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/cmd/install.py", line 346, in install
    package.do_install(**kwargs)
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/package.py", line 1161, in do_install
    rec = spack.store.db.get_record(self.spec)
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/database.py", line 81, in converter
    return function(self, spec_like, *args, **kwargs)
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/database.py", line 554, in get_record
    key = self._get_matching_spec_key(spec, **kwargs)
  File "/blues/gpfs/home/software/spack-0.10.0/lib/spack/spack/database.py", line 549, in _get_matching_spec_key
    raise KeyError("No such spec in database! %s" % spec)
KeyError: 'No such spec in database! parallel-netcdf@1.6.1%intel@17.0.0 cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX"+cxx+fortran+fpic arch=linux-centos6-x86_64^bison@3.0.4%intel@17.0.0 cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX" arch=linux-centos6-x86_64^libpciaccess@0.13.4%intel@17.0.0 cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX" arch=linux-centos6-x86_64^libsigsegv@2.10%gcc@6.1.0 arch=linux-centos6-x86_64^libtool@2.4.6%gcc@6.1.0 arch=linux-centos6-x86_64^m4@1.4.17%gcc@6.1.0+sigsegv arch=linux-centos6-x86_64^mvapich2@2.2%intel@17.0.0 cflags="-axCORE-AVX2,AVX" cxxflags="-axCORE-AVX2,AVX" fflags="-axCORE-AVX2,AVX"~debug~gforker+hydra+mrail~nemesis~nemesisib~nemesisibtcp~psm~remshell~slurm~sock arch=linux-centos6-x86_64^pkg-config@0.29.1%gcc@6.1.0+internal_glib arch=linux-centos6-x86_64^util-macros@1.19.0%gcc@6.1.0 arch=linux-centos6-x86_64'

When I ran spack reindex and tried installing again, I saw:

==> parallel-netcdf is already installed in /blues/gpfs/home/software/spack-0.10.0/opt/spack/linux-centos6-x86_64/intel-17.0.0/parallel-netcdf-1.6.1-fq2r43zk2wcbpl3hrc5h6sfz2dwajbk7

Of course, that directory is empty except for .spack/spec.yaml.

About this issue

  • Original URL
  • State: closed
  • Created 10 years ago
  • Comments: 30 (30 by maintainers)

Most upvoted comments

@adamjstewart:

The only time I would prefer not to do this is if I run spack install --run-tests, the tests fail, but I don’t really care and just want to install it.

There’s a more complicated discussion here on tests that run in the build directory, tests that run from an install directory, etc. and how we should record whether tests pass. I’d like to factor --run-tests out into some separate methods for different packages and rework the spack test command to be able to test packages. Then we could have separate .spack/status/installed, .spack/status/tested, etc. files. It wouldn’t be horrible to store that stuff in the DB either (though i like to keep a backup in the .spack directories in case we need to regenerate the DB).

Anyway, it sounds like you want unconditional removal by default, with an option to keep. Would it make sense to reuse the --keep-prefix option for this? Currently it skips cleanup and keeps the prefix on install failure. I suggest we make it also keep the prefix if it’s left after a prior build failure.

I’m self-assigning this. I can start looking at the issue this week (and hopefully a PR by this Friday). If someone else was looking at this let me know.

I’m thinking the most straightforward approach would be to create an additional file in the .spack metadata directory in the package prefix to indicate that the install (and post-install hooks) completed (e.g. .spack/install_marker, which could be empty). If a command line option is enabled then a package prefix will be removed whenever it does not contain this file.

I think it would also be possible to prevent these issues by altering the SIGINT handler during the installation (starting before the point the prefix is created) but it would be harder to do so in a straightforward manner.

@tgamblin @becker33 curious if you have thoughts/preference between the above two strategies.

Almost all of my builds are as batch jobs and I encounter this several times a week. It’s currently my least favorite thing about Spack.

I know most of these are crazy, but I don’t want other users to run into these and not know how to solve them.

I agree. Now that the basic features of Spack are more solid we need to focus a bit more on polishing the UI, which among other things means not having behaviors like these.

And never fail with a stacktrace, just reassuring messages 😆

Yeah, I think these are at least 3 different problems, all with similar symptoms as a result. I know most of these are crazy, but I don’t want other users to run into these and not know how to solve them.

I think this is a different problem than before. Before, you were probably giving <kbd>Ctrl</kbd>+<kbd>C</kbd> while here:

if os.path.exists(path):
    try:
        shutil.rmtree(path) # <--
    except exceptions.OSError as e:
        raise RemoveFailedError(spec, path, e)

which is called as part of an exceptional path:

        try:
            # Create the install prefix and fork the build process.
            spack.store.layout.create_install_directory(self.spec)
            # Fork a child to do the actual installation
            spack.build_environment.fork(self, build_process, dirty=dirty)
            # If we installed then we should keep the prefix
            keep_prefix = True if self.last_phase is None else keep_prefix
            # note: PARENT of the build process adds the new package to
            # the database, so that we don't need to re-read from file.
            spack.store.db.add(
                self.spec, spack.store.layout, explicit=explicit
            )
        except directory_layout.InstallDirectoryAlreadyExistsError:
            # Abort install if install directory exists.
            # But do NOT remove it (you'd be overwriting someone else's stuff)
            tty.warn("Keeping existing install prefix in place.")
            raise
        except StopIteration as e:
            # A StopIteration exception means that do_install
            # was asked to stop early from clients
            tty.msg(e.message)
            tty.msg(
                'Package stage directory : {0}'.format(self.stage.source_path)
            )
        finally:
            # Remove the install prefix if anything went wrong during install.
            if not keep_prefix:
                self.remove_prefix()  # <---

and this is not handled gracefully. I still have to figure out what could be the sequence causing the last problem you posted.

Anyway, it sounds like you want unconditional removal by default, with an option to keep.

This is just a use case. I really wouldn’t be offended if we cleaned up by default and there was no option to preserve the files. But --keep-prefix sounds good to me as far as UI goes.