spack: Github Archive URL checksums have changed
I haven’t been able to download and test all of the archives, but it looks like we may be affected by homebrew/homebrew-core#18044. See also easybuilders/easybuild-easyconfigs#5151.
Short story: GitHub has changed the way they generate their release “archives”. They’re generated dynamically so many checksums have changed for github.com/.../archive/ URLs. Looks like we may have to download all these tarballs and re-checksum them.
I count 258 packages and 728 tarballs that we’ll need to deal with. Great! 🔥 🔥 🔥
Long story: See the details here:
- homebrew/homebrew-core#18044
- easybuilders/easybuild-easyconfigs#5151
- google/protobuf#3619
- bazelbuild/bazel#3722
- tensorflow/tensorflow#12979
I don’t think it’s that hard to do this but it’s tedious. It’s also not clear to me right now whether their hashes will change back, or whether they might change again. If they’re definitely going to stay changed, I think we should take the plunge, download and checksum these (using https), and use the opportunity to start using SHA256 instead of MD5 if we’re doing this anyway.
Note: The github.com/.../release/... URLs are apparently stable (they respect the file the author uploads, and are not dynamically generated). It might be worth switching to those URLs where possible, but I don’t think it’s possible for all of our projects.
@adamjstewart @alalazo @mamelara @scheibelp @becker33 @lee218llnl @mplegendre
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Reactions: 2
- Comments: 24 (24 by maintainers)
Commits related to this issue
- net/gitlab-agent: Update distinfo It seems that github again delivers different archives. Therefor the checksums changed. It is maybe required to download the files from github.com/.../release/... A... — committed to pfsense/FreeBSD-ports by mfechner 6 years ago
- [Backport MR-770] COMP: Improve source download strategy for Assimp project This commit partially reverts ce3d6a91d (Convert external projects to zips) It avoids check-sum verification failure in th... — committed to jcfr/iMSTK by jcfr 5 years ago
- COMP: Improve source download strategy for Assimp project This commit partially reverts ce3d6a91d (Convert external projects to zips) It avoids check-sum verification failure in the following scenar... — committed to Kitware/iMSTK by jcfr 5 years ago
There is always the
--no-checksumoption for people who know that the archive they have is good. We warn people about it, but this is kind of what it’s there for. We could put some clear guidance in about when to use it when these situations arise. I worry about training people to use it too much, though.That would be great!
@alalazo: I worry about deprecated hashes because I know of no other system that does that. Seems like the solution is always to remove the old file – why not just do that and force a re-download? We could prompt before removing. We could also add an option to force a new download to
spack installand suggest it – wouldn’t those be easier?Mirror will actually do the download part. It just needs a few tweaks.
I think we need a python script to do the job (download/check/update
package.pyif necessary) on the provided subset of all packages in Spack.