spack: Github Archive URL checksums have changed

I haven’t been able to download and test all of the archives, but it looks like we may be affected by homebrew/homebrew-core#18044. See also easybuilders/easybuild-easyconfigs#5151.

Short story: GitHub has changed the way they generate their release “archives”. They’re generated dynamically so many checksums have changed for github.com/.../archive/ URLs. Looks like we may have to download all these tarballs and re-checksum them.

I count 258 packages and 728 tarballs that we’ll need to deal with. Great! 🔥 🔥 🔥

Long story: See the details here:

  • homebrew/homebrew-core#18044
  • easybuilders/easybuild-easyconfigs#5151
  • google/protobuf#3619
  • bazelbuild/bazel#3722
  • tensorflow/tensorflow#12979

I don’t think it’s that hard to do this but it’s tedious. It’s also not clear to me right now whether their hashes will change back, or whether they might change again. If they’re definitely going to stay changed, I think we should take the plunge, download and checksum these (using https), and use the opportunity to start using SHA256 instead of MD5 if we’re doing this anyway.

Note: The github.com/.../release/... URLs are apparently stable (they respect the file the author uploads, and are not dynamically generated). It might be worth switching to those URLs where possible, but I don’t think it’s possible for all of our projects.

@adamjstewart @alalazo @mamelara @scheibelp @becker33 @lee218llnl @mplegendre

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Reactions: 2
  • Comments: 24 (24 by maintainers)

Commits related to this issue

Most upvoted comments

If we don’t record deprecated hashes we could potentially loose a tiny bit on the reproducibility side, and maybe that could be worse for people working on air-gapped systems (if they are stuck with an old tarball and need to wait for some authorization)?

There is always the --no-checksum option for people who know that the archive they have is good. We warn people about it, but this is kind of what it’s there for. We could put some clear guidance in about when to use it when these situations arise. I worry about training people to use it too much, though.

If you think it’s worth it, I can implement the functionality in #5384 .

That would be great!

@alalazo: I worry about deprecated hashes because I know of no other system that does that. Seems like the solution is always to remove the old file – why not just do that and force a re-download? We could prompt before removing. We could also add an option to force a new download to spack install and suggest it – wouldn’t those be easier?

Mirror will actually do the download part. It just needs a few tweaks.

I think we need a python script to do the job (download/check/update package.py if necessary) on the provided subset of all packages in Spack.