spack: Inconsistent Specs for the Same Package

@tgamblin @alalazo @adamjstewart

I’ve had some problems with the same package being installed twice under different hashcodes, even though NOTHING relevant changed. Certain packages seem to be “hot spots” for this behavior; for example, py-numpy in my case. See:

$ ls -ltd py-numpy-*
drwxr-xr-x. 5 rpfische rpfische 4096 Nov  2 20:48 py-numpy-1.11.1-4x5i7jdcohqmmygrihehgstdrwhrh5xy
drwxr-xr-x. 5 rpfische rpfische 4096 Oct 12 14:59 py-numpy-1.11.1-gvcr5ncrxern5m7g67q5r5d3p3odbkq5
drwxr-xr-x. 5 rpfische rpfische 4096 Oct  6 13:31 py-numpy-1.11.1-5fjabod6muxily63zp3qjdczj6v3yowg

The last change to my Spack was Oct 6, so why does it keep re-installing? Let’s look at the .spack/spec.yaml for each of them:

spec-4x5i7jdcohqmmygrihehgstdrwhrh5xy.yaml.txt spec-gvcr5ncrxern5m7g67q5r5d3p3odbkq5.yaml.txt spec-5fjabod6muxily63zp3qjdczj6v3yowg.yaml.txt

Let’s compare the most recent two (again, no relevant changes to my Spack in that time). First of all, a quick look at spec-gvc... looks wrong, what is this &id001 stuff?

spec:
- py-numpy:
    version: 1.11.1
    compiler:
      name: gcc
      version: 4.9.3
    namespace: builtin
    parameters:
      blas: true
      lapack: true
      cflags: &id001 []
      cppflags: &id002 []
      cxxflags: &id003 []
      fflags: &id004 []
      ldflags: &id005 []
      ldlibs: &id006 []

Second… the dependencies listed in these files are different, even though py-numpy/packages.yaml never changed! For example, spec-gv.. has py-nose, but spec-5f.. does not.

In my work, from time to time I install the same package from different top-level spack install commands. For example, both spack setup icebin and spack install env-modele ( https://github.com/citibeth/spack/blob/efischer/develop/var/spack/repos/builtin/packages/env-modele/package.py ) will result in py-numpy being installed. But Spack is apparently installing them under different hash codes (this would be easier to prove directly with #2224).

My best guess about what’s going on here… when I say spack spec, I get a tree in which every package is listed exactly once. We all know that dependencies form a DAG, not a tree — where a single dependency can appear along multiple paths from the root. Spack converts the DAG to tree in an arbitrary way. The result is that the set of dependencies for the same package will look different in the tree, depending on whether some of its dependencies were “stolen” by other packages elsewhere in the same tree. Thus, the apparent dependencies of a package will differ depending on the top-level package in the DAG.

It had been my implicit assumption that Spack operated on a DAG, and was just showing it as a tree as a matter of practical convenience for the user when displying spack spec. But… it appears that this tree, no the full DAG, is actually what’s used in spec.yaml and in computing hashes. That is clearly buggy behavior, and would explain a lot of the “mysterious” re-installs we’ve been experiencing.

This bug should be fixed; relying on already-installed packages, as oft-talked about, will likely not fix this bug. If we have #2224, then I believe I could prepare a pair of spack spec statements that produce a different hash for the same package, thus nailing down an easy test case. But even without #2224, I’m fairly confident I’ve figured out the root cause of this mysterious behavior.

[For the record… I do all my install commands with “clean” command lines, i.e. no ^xxx stuff. See enclosed packages.yaml that controls the versions and variants of things that get installed: packages.yaml.txt ]

About this issue

  • Original URL
  • State: closed
  • Created 10 years ago
  • Comments: 20 (20 by maintainers)

Most upvoted comments

Ok so it looks to me that the issue here is the YAML anchors. They will adversely affect the hash in ways we don’t want. Ugh. I looked at this previously and I thought I had gotten rid of them by duplicating certain objects in specs. But apparently not. I dug a bit deeper and it looks like we can force YAML to ignore them. So I think that is the fix.

Some clarifications:

  1. Spack operates on a DAG, not a tree. spec.yaml encodes the DAG.
  2. There are actually options for how to draw that tree. Look at spec.traverse(). The options are not on the spec command, but you can choose to have the traversal “cover” unique nodes, edges, or paths. The default is nodes because it draws each node only once, but the edges option might be what you want. There is also spack graph but it’s hard to read for large things.
  3. We currently intentionally do not store build dependencies in the spec.yaml files. This is a little awkward implementation-wise, but it is intended to partially address the concerns in #1325 while we wait for a better concretizer. Not hashing build dependencies basically makes it so that hashes don’t change every time something in the tree starts depending on cmake or automake, and likewise Spack won’t rebuild a packages that transitively depend on things like that.
  4. The DAG hash is computed by generating a canonical version of spec.yaml (basically spec.yaml without the hash fields), running SHA-1 on that, and base32-encoding the result.

(4) means that if we botch the canonical YAML, the hashes can change. The anchors could cause this, so I’ll at least hard-disable them as described in the link above.

It had been my implicit assumption that Spack operated on a DAG, and was just showing it as a tree as a matter of practical convenience for the user when displying spack spec. But… it appears that this tree, no the full DAG, is actually what’s used in spec.yaml and in computing hashes. That is clearly buggy behavior, and would explain a lot of the “mysterious” re-installs we’ve been experiencing.

What makes you think Spack operates on a tree and not on the full DAG?

As far as I can see in spec.yaml you have all the information to fully reconstruct the DAG (including dependencies for every node, and type of dependencies). Also, don’t know if you knew it, but there’s another command similar to spec:

$ spack graph hdf5
o  hdf5
|\
o |  zlib
 /
o  openmpi
o  hwloc
o  libpciaccess

for complex packages it supports dot to output graphs.

@alalazo: I’m not sure about the nose part. But I’ve got a PR to debug this and at least a fix for the handles.

Glad that it looks like a solution is at hand!

BTW… would it be more robust if we just hashed the DAG directly, rather than hashing its YAML representation?

On Thu, Nov 3, 2016 at 2:50 PM, Todd Gamblin notifications@github.com wrote:

Ok so it looks to me that the issue here is the YAML anchors. They will adversely affect the hash in ways we don’t want. Ugh. I looked at this previously and I thought I had gotten rid of them by duplicating certain objects in specs. But apparently not. I dug a bit deeper and it looks like we can force YAML to ignore them http://pyyaml.org/ticket/91. So I think that is the fix.

Some clarifications:

  1. Spack operates on a DAG, not a tree. spec.yaml encodes the DAG.
  2. There are actually options for how to draw that tree. Look at spec.traverse(). The options are not on the spec command, but you can choose to have the traversal “cover” unique nodes, edges, or paths. The default is nodes because it draws each node only once, but the edges option might be what you want. There is also spack graph but it’s hard to read for large things.
  3. We currently intentionally do not store build dependencies in the spec.yaml files. This is a little awkward implementation-wise, but it is intended to partially address the concerns in #1325 https://github.com/LLNL/spack/issues/1325 while we wait for a better concretizer. Not hashing build dependencies basically makes it so that hashes don’t change every time something in the tree starts depending on cmake or automake, and likewise Spack won’t rebuild a packages that transitively depend on things like that.
  4. The DAG hash is computed by generating a canonical version of spec.yaml (basically spec.yaml without the hash fields), running SHA-1 on that, and base32-encoding the result.

(4) means that if we botch the canonical YAML, the hashes can change. The anchors could cause this, so I’ll at least hard-disable them as described in the link above.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LLNL/spack/issues/2229#issuecomment-258238943, or mute the thread https://github.com/notifications/unsubscribe-auth/AB1cdwrET1ets2D1E7xePJ2_cBg_OdKNks5q6iztgaJpZM4KoPU4 .