pyodide: Dev REPL package checksum verification failure

Currently, I’m getting a checksum verification failure when installing numpy in the dev REPL

The error is,

None of the “sha256” hashes in the integrity attribute match the content of the subresource.

I suspect this is because the dev deployment has a cache duration of 1 hour and we overwrite files, so sometimes packages.json and actual packages might get temporarily out of sync and we would get these errors.

I’m not sure if there is much we can do about it:

As far as I can tell after checking a few commits, package build is mostly reproducible and sha256 of packages remain constant unless there are build-related changes that are not very frequent.
We can reduce the cache duration but then we would pay more in bandwidth if anyone uses dev packages extensively (right now there is no breakout of usage per deployed version)
Otherwise, we could disable sha256 verification on dev deployment but it’s also not ideal

So hopefully this issue will resolve itself in the near future, opened it for future reference.

cc @amitsaha

About this issue

Original URL
State: open
Created 2 years ago
Comments: 18 (15 by maintainers)

Most upvoted comments

But for some CDN suppliers, it’s configurable whether the query parameters will be ignored, something like this: image

Thanks for your suggestions, @KyleXie I appreciate it! I know, but we are relying on JsDelivr CDN does an has an extra caching layer and aggregates multiple CDNs. Serving non immutable artifacts is not a primary use case there.

Another potential solution I can think of would be making the hash value a part of the file name

You are right that could be a solution. The problem with it is,

if we don’t delete old versions with checksums, our storage is going to grow for each commit on main which is not ideal
if we do delete everything , people can end up hitting dev artifacts that no longer exists.

Overall, I would be +1 to close this as “won’t fix”. A good solution would be to have a separate dev bucket maybe with no caching, or deploy each commit on a separate url, but then someone would need to pay for bandwidth (and I don’t think it’s worth it).

This issue is occasionally annoying, but I don’t feel it’s worth spending time on it, while there are more important issues address. Currently existing workarounds are:

it’s possible to starts a REPL from CI artifacts of any commit on main (though admitedly it’s a bit less convient to get that URL)
we should do alpha releases more often so there is a fixed release closer to the current main branch.

rth on Nov 7, 2023

I think it would be good to maybe enable it in the dev REPL by default?

rth on Oct 15, 2022

Hmm, I’m not sure what we should do for this. Maybe we should add an option to disable integrity check and use it for dev consoles? @rth @hoodmane

ryanking13 on Sep 16, 2022