spack: Inconsistent handling of shortened versions

Spack seems to be inconsistently handling shorted versions.

depends_on

✔️ Edited by @haampie 2021-09-22: the issue here is no longer relevant, because depends_on('python@2.7') means the python version is in the closed-open range [2.7, 2.8).

When used in a depends_on statement, if I say:

depends_on('python@2.7')

Spack will install exactly Python 2.7, which will crash because 2.7 doesn’t exist. I would instead expect it to install the latest 2.7.X. This is a long-standing bug documented in #921 and #1035.

As of now, the workaround has been to explicitly turn this into a version range:

depends_on('python@2.7.0:2.7.999')

This works, but is unnecessarily wordy.

when=

✔️ Edited by @haampie 2021-09-22: This issue is fixed, since we can write when='@0.2:0.2.0', and the meaning is consistent with depends_on.

When used in a when= argument, if I say:

depends_on('r@3.4.0:3.4.9', when='@0.2')

this will apply to both versions 0.2 and 0.2.2, as documented in #8950. Similarly, if I say:

patch('foo.patch', when='@3.2')

this patch will be applied to both versions 3.2 and 3.2.1, as documented in #8320 and #8432.

As of now, there is no workaround for this problem. This may be fixed by #8320.

Compiler versions

This one is the worst of all. Let’s say I have a GCC 7.2.0 compiler in my compilers.yaml. There are many other GCC compilers, but this is the only GCC 7 compiler I have. If I say:

$ spack install zlib %gcc@7

Spack will correctly deduce that I want it to use my GCC 7.2.0 compiler, but it will resolve the compiler as %gcc@7 in my spec, and install to the darwin-highsierra-x86_64/gcc-7 directory. The installation succeeds, but if I rerun:

$ spack install foo %gcc@7.2.0 ^zlib

Spack needs to reinstall zlib with the new GCC 7.2.0 compiler. I believe this bug is undocumented. In fact, I think it used to work properly. I used it all the time when I had multiple GCC compilers installed and I wanted to specify one.

The workaround is to use a version range, like %gcc@7:.

Proposal

All of these situations should be handled identically. Otherwise, it’s confusing what behavior to expect. I propose the following behavior.

If a package.py/compilers.yaml contains both 3.2 and 3.2.1, 3.2 should refer to exactly 3.2 only.

If a package.py/compilers.yaml only contains 3.2.1, 3.2 should refer to any 3.2.X version.

@tgamblin @baberlevi @PDoakORNL @cyrush @davydden @goxberry @eschnett

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Reactions: 4
  • Comments: 26 (26 by maintainers)

Most upvoted comments

There are two levels we should think about here.

One is the user facing level, at this point it really isn’t up to us to tell users how they have to think! We consistently see that @1 is expected to match everything from @1.0.0.beta up to (but not including) 2.0. Even though I agree with @davydden that it is problematic that resolution of @1 depends on whether a version('1' does or does not exist, it is too natural to expect it to work and therefore spack should support this. We may be able to train our users to do it our way ™️ (i.e. provide an error and suggest the correct version), but is this really worth it? Or even is it less work than just implementing the matching correctly?

But on the other hand we have how we internally handle versions. Here we are completely free to do as we want (with sensible constructions being easier to maintain, but there is no “user”-catchall argument). Here I would say that we should always use the versions as they are declared in package.py. I.e. all spack-generated folder structure should contain the version from package.py and NOT the user provided strings.

@adamjstewart how is “being able to install versions even if they don’t exist in the package.py.” supposed to work? The one situation that I can think about right now is, that we install a package from one branch of spack and then switch to a branch that does not have the installed version of this package. But in that case the database “knows” the version, so no need to go with the user-provided string for internal use

I was going to suggest creating a constraint like 2019.2:2019.99 (which I thought was analogous to the suggestion in https://github.com/spack/spack/issues/13831#issuecomment-558854712) but after trying that myself I see it doesn’t work.

So the problem you are discussing is more than an inconsistency: it’s a gap in the model’s ability to properly specify constraints (so I think in fact https://github.com/spack/spack/issues/8957#issuecomment-560568209 could be considered separate). I don’t think addressing your issue specifically will be very difficult although it likely won’t address the broader issue here.

@tgamblin @adamjstewart @scheibelp @becker33

Well, I thought about opening a new issue and pretending not to remember this one, just to see if that would get more attention. 😃 But clearly, Adam reported the same issue above.

And to be clear, this is not an issue of interpreting 2019 inside when().

I tried spack install intel-tbb @2019 and the concretizer picks rev exactly 2019, not 2019.8. (This much is correct.) But as Adam reported, the problem is that 2019 satisfies both of these:

patch(..., when='@2019.6:`)
patch(..., when='@:2019.2')

So, rev 2019 is both >= 2019.6 and <= 2019.2.

Basically, this makes it impossible to use version ranges for any package that has variable-length versions. There is no valid order where this makes sense.

IMO, this is a clear bug, seperate from how you want to interpret when='@2019'. (One reason I always specify a full version inside when.)

I was hoping this would be a small and isolated fix and could be done before working out the full details of how to interpret shortened version names.

Btw, does the full fix for this fall under the domain of the new concretizer? Is that what we’re waiting on?

@healther it becomes more annoying when you’re “installing” an external version that may be too old to download/checksum.

Ah okay, I see. Not sure, how useful this use case really is… Changing package.py is a rather low bar to add new versions isn’t it?

I would also be fine with an error, although @tgamblin was a strong proponent of being able to install versions even if they don’t exist in the package.py. If we want to continue to support this, I think Spack should check which versions exist in a package before determining what version is actually requested.

if I ask to install python@2.7, do you really think the best behavior to is to try download exactly 2.7 even though it isn’t in the package?

no, of course not, what I propose is to throw an error.

EDIT: I see the merits of your approach, though.