berry: [Bug] Checksum mismatch for package referenced as "git+https://github.com..."
- I’d be willing to implement a fix
☝️ not sure where I’d have to look/start
Describe the bug
-
We are currently trying to migrate to yarn v2.
-
We have one package in our package.json that is defined like this:
"objection": "git+https://github.com/ovos/objection.js.git#1.6.10-mod.0" -
We ran
yarnand it regenerated the lock file -
When we now run
yarn --immutableduring our CI workflow,yarnends up with an error for this package -YN0018checksum doesn’t match. -
I noticed that yarn seems to build a new “archive” for this package (I guess because it’s not packaged and published via a registry), could it be that this package looks slightly different depending on the system and this causes the checksum mismatch? (committed yarn.lock was generated on macos 11.1 whereas the CI flow is running on Debian 10)
-
With yarn v1 and
--frozen-lockfilethis worked fine 🤔
To Reproduce
I created a separate small repo with just this dependency and 2 Github Actions to illustrate the issue: https://github.com/flipace/yarn2-checksum-error-giturl/runs/1749446131?check_suite_focus=true
I allow the yarn step to update the checksums in the lockfile, and as you can see, they are different on the macos and the ubuntu jobs, and when rerunning the job, it always ends up with a different checksum…
Reproduction
await packageJsonAndInstall({
dependencies: {
[`objection`]: `git+https://github.com/ovos/objection.js.git#1.6.10-mod.0`,
},
});
Screenshots
If applicable, add screenshots to help explain your problem.
Environment if relevant (please complete the following information):
- OS: macos 11.1 / debian 10
- Node version: v12.18.3
- Yarn version: 2.4.0
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 41 (10 by maintainers)
Commits related to this issue
- Update package.json and yarn.locks This updates dependencies in the package.json to their latest versions. This also adds the `dependenciesMeta` information to the shared asset repository st3phhays-a... — committed to st3phhays/StephHays by st3phhays 2 years ago
- Update package.json and yarn.lock This updates dependencies in the package.json to their latest versions. This also adds the `dependenciesMeta` information to the shared asset repository st3phhays-as... — committed to st3phhays/StephHays by st3phhays 2 years ago
- Update package.json and yarn.lock This updates dependencies in the package.json to their latest versions. See: https://github.com/yarnpkg/berry/issues/2399 — committed to st3phhays/StephHays by st3phhays 2 years ago
- Update package.json and yarn.lock This updates dependencies in the package.json to their latest versions. See: https://github.com/yarnpkg/berry/issues/2399 — committed to st3phhays/StephHays by st3phhays 2 years ago
- fix: workaround for https://github.com/yarnpkg/berry/issues/2399 — committed to zeal-corp/cfonts by henryatzeal 2 years ago
- Merge pull request #2 from zeal-corp/fix_checksum fix: workaround for https://github.com/yarnpkg/berry/issues/2399 — committed to zeal-corp/cfonts by henryatzeal 2 years ago
- fix: Regenerate the `yarn.lock` on Linux to fix CI build issues Since according to https://github.com/yarnpkg/berry/issues/2399, the OS matters. — committed to AdobeDocs/uxp-xd by pklaschka a year ago
I’m able to reproduce this using yarn 3.0.1. This is the scenario:
preparescript is used to do a build once installedyarn install. Checksums don’t match, and there is an error codeYN0018.Can this issue be re-opened?
We are having the same issue using Yarn
3.2.1. Here is the dependency from ourpackage.json:Here’s the
yarn.lock:We are seeing differences on identical versions of macOS (Monterey
12.4) using Node18.5.0.react-native-track-player-https-50569cc95b-abaff8e3e3.zipabaff8e3e3e33e42a731d2a9256c1a48e3f98ec8583db939a6a99caedbdade29585277d0920a6afeb69c8b9fdcb4f981b09786c2d6ba8193f5025dfb02341f09react-native-track-player-https-50569cc95b-1420b23b25.zip1420b23b250403fefcb4ccfeb22c1529a3ff8dad5a7ac11a10075771297c48c570956a86f8269e7325e2a9bcc4e320c29cc0a3446a21f67a38efd3284dcdf128Upon extracting the zip files and doing a recursive diff, there’s only a single change between the two zip files, at the bottom of the
package.jsonfile:So, for whatever reason, my Yarn
3.2.1is injecting thepackageManagerinto thepackage.jsonthat other Yarn installs are not. I’ve tried--mode=skip-build, but it has the same result. It doesn’t matter if we remove and re-add the package, the behavior stays consistently the same.We have the Yarn
3.2.1release managed in git under.yarn/releases/yarn-3.2.1.cjs, so it’s really confusing how this might be happening. Thepackage.jsonfrom the project doesn’t have thepackageManagerspecified, as they are using Yarn 1:https://github.com/doublesymmetry/react-native-track-player/blob/44b56147c53e833de7b8fe9b6b26b8413d9ae7d9/package.json
I’ve tried installing Yarn via Homebrew, via corepack in Homebrew, and also globally with
npm i -g yarn, and all three install variations (each version1.22.19) have the same behavior.Does anyone have any ideas what might be causing this difference in behavior? @arcanis?
Hi @arcanis I’m also facing this issue.
I think it is a problem with yarn.
The repo in my instance is this one, at the given commit - https://github.com/celo-org/ganache-cli/commit/31b5bbbca5e7d098d2459bc7942eafb77141f442
Referenced like this
Installing locally and in a docker container produce different checksums.
When I diffed the contents of the 2 packages I found this.
One of those files is a
Makefilelooking inside I can see that they have been generated, since they contain paths from the system on which they were built.It seems that yarn is cloning and building these repos including the c code of its dependencies, and since the environments where the repos are being built are different the packages end up with different checksums. I guess this must be a departure from how the yarn 1 cache was structured otherwise it wouldn’t have been possible to use packages with dependencies on c code across different machines.
Since the problem is due to building on different machines I don’t think it can be reproduced by sherlock, I had a go anyway but the sherlock build never completes.
we’re seeing the same issue as @brianlenz where packageManager field in the cache files doesn’t match the project level yarn version (3.2.2 vs 3.2.1). My guess is that this is being added here because yarn is trying to build the package since it is a git package rather than a registry one. Except that happens at a global scope (for global cache) rather than using the project version of yarn that we have and when team members have a different global version of @yarnpkg/core - they get a different built file, and checksums don’t match. This is affecting new members coming on to the project as well as new CI machines, because they are getting latest yarnpkg/core (3.2.2).
Looking at this issue with a fresh eye, I feel like this issue is simply the consequence of us running the full prepack flow on installs since Yarn 2+, whereas Yarn 1 used to just clone and go.
For context, we made this change because git repositories typically contain by design the package sources (for instance TS files) rather than the package artifacts (for instance the transpiled JS files) - meaning that to consume them, they usually need to be built as if you intended to publish them. There isn’t a way to disable this build at the moment, so the fix would be one of those:
Make your build independent from the machine that builds it. In theory this is the approach that I would recommend if not too inconvenient, as packages aren’t meant to have different contents (that’s the whole point of validating checksums!). For instance in your case, perhaps the Makefile shouldn’t be part of the final archive, or perhaps it should use relative links (I haven’t looked in details so I might be wrong in this case, but you see the idea in general).
Help us implement a “don’t build” annotation on the
git+*specifier, which would instruct Yarn to skip the build step and use the repository unmodified. I think there’s been some decent public interest for this feature, so we’d definitely welcome a PR.Hi! 👋
This issue looks stale, and doesn’t feature the
reproduciblelabel - which implies that you didn’t provide a working reproduction using Sherlock. As a result, it’ll be closed in a few days unless a maintainer explicitly vouches for it or you edit your first post to include a formal reproduction (you can use the playground for that).Note that we require Sherlock reproductions for long-lived issues (rather than standalone git repositories or similar) because we’re a small team. Sherlock gives us the ability to check which bugs are still affecting the master branch at any given point, and decreases the amount of code we need to run on our own machines (thus leading to faster bug resolutions). It helps us help you! 😃
If you absolutely cannot reproduce a bug on Sherlock (for example because it’s a Windows-only issue), a maintainer will have to manually add the
upholdedlabel. Thanks for helping us triaging our repository! 🌟Hello! This issue affects us, too, because our developers work on different operating systems. I don’t know why this was marked unreproducible, because it’s very easy to reproduce:
yarn.lockand pushyarn install --immutableConfirmed;
yarn set version 4.0.0-rc.25fixes this for me.@arcanis Any updates on this? This is still a problem for a lot of people, including our team.
We have a dependency that transitively depends on
gluegun@git+https://github.com/edgeandnode/gluegun.git#v4.3.1-pin-colors-dep. It’s not feasible for us to change this dependency, unless we create forks of a bunch of packages all the way to this transitive dependency. Our local and CI builds all keep failing because the checksum is different for everybody.I think adding the ability to selectively ignore the checksum for specific packages can be used as a workaround until the underlying issue is fixed. Right now it’s all or nothing with yarn checksum behavior. Either all checksums are checked, or all are ignored/updated which renders yarn.lock useless.
This is killing so much of our time.
I’m having a similar problem, but not exactly the same:
Different devs are getting different checksums for packages downloaded from git. For example, we have this in our
yarn.lock:I asked one of the devs with the “wrong” checksum to send me the file so I can compare with mine. Turns out the content of the
.zipfiles is different. In particular, thepackage.jsonfile is different. Mine looks like it was “pretty-printed” in some places, while theirs is like the version published in GitHub.Here is an excerpt of the files:
Thanks for the details! I’ll try to find where exactly that thing goes off the rails this week and either close the issue or open a PR with a fix 🤞
It clones the repository, runs
yarn install && yarn packinside it (which in turn runs build scripts), then use the resulting tgz as package. Some steps are skipped when not relevant (for example there are no installs if there are no build scripts to run).for anyone having this issue, I believe this is related https://github.com/yarnpkg/berry/issues/2774#issuecomment-822472844
I’m having what seems to be a related issue, where the checksum of a Github package is changing but without any apparent intervention. It’s referenced as
github:orgname/repo#branchnamein ourpackage.json(notgit+https://...), and the commit it resolves to hasn’t changed.There wasn’t a problem until we upgraded to yarn 3.2.0, at that point the checksum started failing both locally (Mac OS) and on CI (Linux). This didn’t appear to be a cross-platform issue as the same new checksum worked on both platforms, so I assumed there was some change in how the checksum was calculated (though I couldn’t find anything in the changelog).
However, without any further changes from us, that checksum has started failing again - the latest checksum, still on 3.2.0 and with the same commit hash for the Github package, is different both from the checksum under yarn 3.1.1 and the one that was generated when we first updated to 3.2.0. I’ve tried setting
built: falseindependenciesMetaas per @phyllisstein above and re-generating the checksum - it generates exactly the same checksum with or without this setting. We don’t have a build script for this package - we just check in a built version of the code, so I don’t think there is anything to build that would be causing a different checksum.@arcanis I just downloaded the archives of the yarn cache for both systems (you should be able to download them as well here: https://github.com/flipace/yarn2-checksum-error-giturl/actions/runs/506322343).
After inspecting both with Kaleidoscope, I found that there are symbolic links in this repo (eg: https://github.com/ovos/objection.js/blob/master/doc/api/model/README.md) which seem to cause the checksum to be different, depending on the system on which the archive is generated by yarn.
Now I’m not sure whether this can be considered a bug, or it’s just “by design” 😅 (since this worked fine with yarn v1)
(P.S. Also, again - note that, when I rerun this github action workflow - the checksums for this package always seem to differ, even on the same “OS”.)
(P.P.S:
The symlinks in that yarn generated archive are always using absolute paths, so I guess that’s where the differences come from in the generated sha… I guess that the sha for symlinks should either…use the contents of the target file, or the path to which the symlink points to but relative to its current directory instead of absolute?nevermind, i dug into yarns source and the github fetcher, it loads the github .tar.gz, so no idea where this goes wrong then…)