spack: Wrong source url for older versions of R-packages

Steps to reproduce

When there is a newer version of an R-package published on cran, fetching the older versions does not work in spack anymore. e.g. r-ellipsis

  • newest version in spack is 0.3.1 (before #26965 is merged)
  • newest version on cran 0.3.2
  • running spack checksum r-ellipsis@0.3.1 fails with the error message
» spack checksum r-ellipsis@0.3.1 
==> Found 1 version of r-ellipsis:
  
  0.3.1  https://cloud.r-project.org/src/contrib/ellipsis_0.3.1.tar.gz

==> Something failed on https://cloud.r-project.org/src/contrib/ellipsis_0.3.1.tar.gz, skipping.  (All fetchers failed for spack-stage-z032nuxl)
==> Error: Could not fetch any versions for r-ellipsis

but spack checksum r-ellipsis@0.3.2 works.

Also spack install r-ellipsis0.3.1 works, because it fetches the sources from the spack mirror and not from cran.

Reason: spack tries to get the sources from https://cloud.r-project.org/src/contrib/ellipsis_0.3.1.tar.gz but https://cloud.r-project.org/src/contrib only provides the newest version (here 0.3.2), older versions are migrated to https://cloud.r-project.org/src/contrib/Archive/ellipsis. Thus spack checksum fails for old versions of the package (I checked all old versions in spack: 0.3.1, 0.3.0, 0.2.0.1).

I tried to migrate to using the cran attribute instead of explicit url and list_url but that does not change anything.

All R-packages I tested had this problem regarding older versions, e.g.

  • r-rcpp@1.0.6,
  • r-broom@0.3.1
  • r-lifecycle@0.2.0
  • r-lme4@1.1-27

ping @glennpj as R maintainer

Error message

No response

Information on your system

  • Spack: 0.16.3-5036-7884fc8973
  • Python: 3.8.10
  • Platform: linux-fedora32-haswell
  • Concretizer: clingo

General information

  • I have run spack debug report and reported the version of Spack/Python/Platform
  • I have searched the issues of this repo and believe this is not a duplicate
  • I have run the failing commands in debug mode and reported the output

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 17 (17 by maintainers)

Commits related to this issue

Most upvoted comments

To be honest, I am not sure that spack checksum ... with a version from list_url worked recently. So there may be two problems here.

To be complete, I replaced the cran attribute with explicit url and list_url parameters and see the same behavior as with the cran attribute. That pretty much confirms that this is not an r problem but a checksum problem.

I can confirm that backing out #25831 fixes the case where a version is not specified on the command line.

$ spack checksum -b r-ellipsis
==> Found 8 versions of r-ellipsis:
  
  0.3.2    https://cloud.r-project.org/src/contrib/ellipsis_0.3.2.tar.gz
  0.3.1    https://cloud.r-project.org/src/contrib/Archive/ellipsis/ellipsis_0.3.1.tar.gz
  0.3.0    https://cloud.r-project.org/src/contrib/Archive/ellipsis/ellipsis_0.3.0.tar.gz
  0.2.0.1  https://cloud.r-project.org/src/contrib/Archive/ellipsis/ellipsis_0.2.0.1.tar.gz
  0.2.0    https://cloud.r-project.org/src/contrib/Archive/ellipsis/ellipsis_0.2.0.tar.gz
  0.1.0    https://cloud.r-project.org/src/contrib/Archive/ellipsis/ellipsis_0.1.0.tar.gz
  0.0.2    https://cloud.r-project.org/src/contrib/Archive/ellipsis/ellipsis_0.0.2.tar.gz
  0.0.1    https://cloud.r-project.org/src/contrib/Archive/ellipsis/ellipsis_0.0.1.tar.gz

==> Fetching https://cloud.r-project.org/src/contrib/ellipsis_0.3.2.tar.gz
==> Fetching https://cloud.r-project.org/src/contrib/Archive/ellipsis/ellipsis_0.3.1.tar.gz
==> Fetching https://cloud.r-project.org/src/contrib/Archive/ellipsis/ellipsis_0.3.0.tar.gz
==> Fetching https://cloud.r-project.org/src/contrib/Archive/ellipsis/ellipsis_0.2.0.1.tar.gz
==> Fetching https://cloud.r-project.org/src/contrib/Archive/ellipsis/ellipsis_0.2.0.tar.gz
==> Fetching https://cloud.r-project.org/src/contrib/Archive/ellipsis/ellipsis_0.1.0.tar.gz
==> Fetching https://cloud.r-project.org/src/contrib/Archive/ellipsis/ellipsis_0.0.2.tar.gz
==> Fetching https://cloud.r-project.org/src/contrib/Archive/ellipsis/ellipsis_0.0.1.tar.gz

    version('0.3.2',   sha256='a90266e5eb59c7f419774d5c6d6bd5e09701a26c9218c5933c9bce6765aa1558')
    version('0.3.1',   sha256='4f8a15158dfc27cdc0f7554c7a61e92b02e4d70bfc3d968f01a99da2189b75db')
    version('0.3.0',   sha256='0bf814cb7a1f0ee1f2949bdc98752a0d535f2a9489280dd4d8fcdb10067ee907')
    version('0.2.0.1', sha256='0e6528c5e8016c3617cc1cfcdb5a4bfeb073e0bd5ea76b43e56b0c3208a0a943')
    version('0.2.0',   sha256='2b528689b84047e5eaf67fe9ef1e5a8006f4f12ca6617445c1b6584e703ab341')
    version('0.1.0',   sha256='e6f1f30390182b36664b097b5fbf6e95897a766b1ddfa774d3b507f42e26895f')
    version('0.0.2',   sha256='4e5e2a34842f2e82da0b1abef8c05f35c829e7ec713d764f40c4a48df512965d')
    version('0.0.1',   sha256='6a8ae2f9a62179e94eb3574e467e5011df19632cbeb7d8df6319d3dde830bb0a')

but not when a version is given on the command line.

I guess. an idea might be to implement url_for_version() in lib/spack/spack/build_systems/r.py returning src/contrib/ for the latest version (and check CRAN if it is still the latest…), and src/contrib/Archive/’ + self.cran + ‘/’ for the Archive versions.

No, this used to work. The fix is to ensure that list_url is considered as well as url. Also, it is important to note that this is not an r problem. Other packages that use list_url would also be affected.

To echo what @bernhardkaindl says above, there is also a problem when a version is specified on the command line.

$ spack checksum r-ellipsis@0.3.1
==> Found 1 version of r-ellipsis:
  
  0.3.1  https://cloud.r-project.org/src/contrib/ellipsis_0.3.1.tar.gz

==> Something failed on https://cloud.r-project.org/src/contrib/ellipsis_0.3.1.tar.gz, skipping.  (All fetchers failed for spack-stage-dnog5w_s)

That also triggers using only url and not list_url.

I am trying to catch up on things but will take a look.