spack: Inconsistent Specs for the Same Package
@tgamblin @alalazo @adamjstewart
I’ve had some problems with the same package being installed twice under different hashcodes, even though NOTHING relevant changed. Certain packages seem to be “hot spots” for this behavior; for example, py-numpy in my case. See:
$ ls -ltd py-numpy-*
drwxr-xr-x. 5 rpfische rpfische 4096 Nov 2 20:48 py-numpy-1.11.1-4x5i7jdcohqmmygrihehgstdrwhrh5xy
drwxr-xr-x. 5 rpfische rpfische 4096 Oct 12 14:59 py-numpy-1.11.1-gvcr5ncrxern5m7g67q5r5d3p3odbkq5
drwxr-xr-x. 5 rpfische rpfische 4096 Oct 6 13:31 py-numpy-1.11.1-5fjabod6muxily63zp3qjdczj6v3yowg
The last change to my Spack was Oct 6, so why does it keep re-installing? Let’s look at the .spack/spec.yaml for each of them:
spec-4x5i7jdcohqmmygrihehgstdrwhrh5xy.yaml.txt spec-gvcr5ncrxern5m7g67q5r5d3p3odbkq5.yaml.txt spec-5fjabod6muxily63zp3qjdczj6v3yowg.yaml.txt
Let’s compare the most recent two (again, no relevant changes to my Spack in that time). First of all, a quick look at spec-gvc... looks wrong, what is this &id001 stuff?
spec:
- py-numpy:
version: 1.11.1
compiler:
name: gcc
version: 4.9.3
namespace: builtin
parameters:
blas: true
lapack: true
cflags: &id001 []
cppflags: &id002 []
cxxflags: &id003 []
fflags: &id004 []
ldflags: &id005 []
ldlibs: &id006 []
Second… the dependencies listed in these files are different, even though py-numpy/packages.yaml never changed! For example, spec-gv.. has py-nose, but spec-5f.. does not.
In my work, from time to time I install the same package from different top-level spack install commands. For example, both spack setup icebin and spack install env-modele ( https://github.com/citibeth/spack/blob/efischer/develop/var/spack/repos/builtin/packages/env-modele/package.py ) will result in py-numpy being installed. But Spack is apparently installing them under different hash codes (this would be easier to prove directly with #2224).
My best guess about what’s going on here… when I say spack spec, I get a tree in which every package is listed exactly once. We all know that dependencies form a DAG, not a tree — where a single dependency can appear along multiple paths from the root. Spack converts the DAG to tree in an arbitrary way. The result is that the set of dependencies for the same package will look different in the tree, depending on whether some of its dependencies were “stolen” by other packages elsewhere in the same tree. Thus, the apparent dependencies of a package will differ depending on the top-level package in the DAG.
It had been my implicit assumption that Spack operated on a DAG, and was just showing it as a tree as a matter of practical convenience for the user when displying spack spec. But… it appears that this tree, no the full DAG, is actually what’s used in spec.yaml and in computing hashes. That is clearly buggy behavior, and would explain a lot of the “mysterious” re-installs we’ve been experiencing.
This bug should be fixed; relying on already-installed packages, as oft-talked about, will likely not fix this bug. If we have #2224, then I believe I could prepare a pair of spack spec statements that produce a different hash for the same package, thus nailing down an easy test case. But even without #2224, I’m fairly confident I’ve figured out the root cause of this mysterious behavior.
[For the record… I do all my install commands with “clean” command lines, i.e. no ^xxx stuff. See enclosed packages.yaml that controls the versions and variants of things that get installed:
packages.yaml.txt ]
About this issue
- Original URL
- State: closed
- Created 10 years ago
- Comments: 20 (20 by maintainers)
Ok so it looks to me that the issue here is the YAML anchors. They will adversely affect the hash in ways we don’t want. Ugh. I looked at this previously and I thought I had gotten rid of them by duplicating certain objects in specs. But apparently not. I dug a bit deeper and it looks like we can force YAML to ignore them. So I think that is the fix.
Some clarifications:
spec.yamlencodes the DAG.spec.traverse(). The options are not on thespeccommand, but you can choose to have the traversal “cover” unique nodes, edges, or paths. The default isnodesbecause it draws each node only once, but theedgesoption might be what you want. There is also spack graph but it’s hard to read for large things.spec.yamlfiles. This is a little awkward implementation-wise, but it is intended to partially address the concerns in #1325 while we wait for a better concretizer. Not hashing build dependencies basically makes it so that hashes don’t change every time something in the tree starts depending oncmakeorautomake, and likewise Spack won’t rebuild a packages that transitively depend on things like that.spec.yaml(basically spec.yaml without the hash fields), runningSHA-1on that, andbase32-encoding the result.(4) means that if we botch the canonical YAML, the hashes can change. The anchors could cause this, so I’ll at least hard-disable them as described in the link above.
What makes you think
Spackoperates on a tree and not on the full DAG?As far as I can see in
spec.yamlyou have all the information to fully reconstruct the DAG (including dependencies for every node, and type of dependencies). Also, don’t know if you knew it, but there’s another command similar tospec:for complex packages it supports
dotto output graphs.@alalazo: I’m not sure about the nose part. But I’ve got a PR to debug this and at least a fix for the handles.
Glad that it looks like a solution is at hand!
BTW… would it be more robust if we just hashed the DAG directly, rather than hashing its YAML representation?
On Thu, Nov 3, 2016 at 2:50 PM, Todd Gamblin notifications@github.com wrote: