cargo: Performance Regression in `cargo package` on `1.81.0`
Problem
The aws-sdk-rust recently updated our MSRV to 1.81.0. This caused a 3-4x slowdown in our cargo package invocation. We traced the likely culprit of this slowdown to https://github.com/rust-lang/cargo/pull/13960 which removes an if !opts.allow_dirty check around the check_repo_state function causing it to run on ever packaging step.
This causes a problem with the aws-sdk-rust repo. Since it is very large and has a huge history all invocations to git in that repository are slow. Since cargo package is now invoking git on every packaging step in the workspace it slows the whole process down to a crawl.
Steps
- Run
git clone https://github.com/awslabs/aws-sdk-rust.gitto checkout theaws-sdk-rustrepo (this will be a bit slow, it is huge) - Ensure that you are using a cargo version
<1.81 - In the
aws-sdk-rustrepo runcargo package --no-verify --allow-dirty --workspaceobserve the approximate pace at which crates are packaged (you can also use thetimecommand, but this fails locally for me withToo many open files (os error 24)before it completes) - Upgrade your cargo version to
=1.81 - Again run
time cargo package --no-verify --allow-dirty --workspace(this goes too slowly for me to wait for it to fail, but it is very easy to observe the difference in speed)
Possible Solution(s)
Potentially when packaging multiple crates it might be possible to only invoke git once at the beginning of the run? Or potentially add a new option to disable the behavior introduced in https://github.com/rust-lang/cargo/pull/13960 that always generates a .cargo_vcs_info.json
Notes
No response
Version
$ cargo version --verbose
cargo 1.81.0 (2dbb1af80 2024-08-20)
release: 1.81.0
commit-hash: 2dbb1af80a2914475ba76827a312e29cedfa6b2f
commit-date: 2024-08-20
host: aarch64-apple-darwin
libgit2: 1.8.1 (sys:0.19.0 vendored)
libcurl: 8.7.1 (sys:0.4.73+curl-8.8.0 system ssl:(SecureTransport) LibreSSL/3.3.6)
ssl: OpenSSL 1.1.1w 11 Sep 2023
os: Mac OS 14.7.1 [64-bit]
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 17 (15 by maintainers)
Commits related to this issue
- fix(cargo-package): add more traces (#14960) ### What does this PR try to resolve? This helps debug <https://github.com/rust-lang/cargo/issues/14955>. ### How should we test and review this PR? Wh... — committed to rust-lang/cargo by epage 2 years ago
- perf(cargo-package): match certain path prefix with pathspec (#14997) ### What does this PR try to resolve? This revives #14962. See benchmark chart in <https://github.com/rust-lang/cargo/pull/14962... — committed to rust-lang/cargo by epage 2 years ago
Good news: #15534 seems to work as a fix for the regression:
With the current version there are results immediately and it takes a little less than 12m to compile everything. The greatest expense seems to be the fetching of latest packages now.
The changes require the
gitoxideversion to be used, but if someone is interested, there should be no problem in backporting the changes to thegit2version.If anyone else happens to hit this issue, we did find a workaround. In our publishing step we now
rm -rf .git/before runningcargo packageand that has taken a full workspace publish from ~3 hours to ~15 minutes. Would still be nice to have this fixed in thecargo packagecommand itself, but we are back to around the performance we had previously.imo providing
--no-vcs-infoshould be a last resortcrates.ioto have VCS info to check to see if anything has changed between VCS and publish. Improving that situation is the reason we have this performance regression (#13960).imo before we consider alternatives, we should see what the limit is for how much we can optimize what we are already doing and see if that is within an acceptable limit (determined by the Cargo team).
If we do look for more alternatives, some additional ideas include:
--allow-dirty(this seems a bit magical)