cargo: Performance Regression in `cargo package` on `1.81.0`

Problem

The aws-sdk-rust recently updated our MSRV to 1.81.0. This caused a 3-4x slowdown in our cargo package invocation. We traced the likely culprit of this slowdown to https://github.com/rust-lang/cargo/pull/13960 which removes an if !opts.allow_dirty check around the check_repo_state function causing it to run on ever packaging step.

This causes a problem with the aws-sdk-rust repo. Since it is very large and has a huge history all invocations to git in that repository are slow. Since cargo package is now invoking git on every packaging step in the workspace it slows the whole process down to a crawl.

Steps

  1. Run git clone https://github.com/awslabs/aws-sdk-rust.git to checkout the aws-sdk-rust repo (this will be a bit slow, it is huge)
  2. Ensure that you are using a cargo version <1.81
  3. In the aws-sdk-rust repo run cargo package --no-verify --allow-dirty --workspace observe the approximate pace at which crates are packaged (you can also use the time command, but this fails locally for me with Too many open files (os error 24) before it completes)
  4. Upgrade your cargo version to =1.81
  5. Again run time cargo package --no-verify --allow-dirty --workspace (this goes too slowly for me to wait for it to fail, but it is very easy to observe the difference in speed)

Possible Solution(s)

Potentially when packaging multiple crates it might be possible to only invoke git once at the beginning of the run? Or potentially add a new option to disable the behavior introduced in https://github.com/rust-lang/cargo/pull/13960 that always generates a .cargo_vcs_info.json

Notes

No response

Version

$ cargo version --verbose
cargo 1.81.0 (2dbb1af80 2024-08-20)
release: 1.81.0
commit-hash: 2dbb1af80a2914475ba76827a312e29cedfa6b2f
commit-date: 2024-08-20
host: aarch64-apple-darwin
libgit2: 1.8.1 (sys:0.19.0 vendored)
libcurl: 8.7.1 (sys:0.4.73+curl-8.8.0 system ssl:(SecureTransport) LibreSSL/3.3.6)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Mac OS 14.7.1 [64-bit]

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 17 (15 by maintainers)

Commits related to this issue

Most upvoted comments

Good news: #15534 seems to work as a fix for the regression:

With the current version there are results immediately and it takes a little less than 12m to compile everything. The greatest expense seems to be the fetching of latest packages now.

aws-sdk-rust ( main) via 🐍
❯ /Users/byron/dev/github.com/rust-lang/cargo/target/release/cargo package --allow-dirty --no-verify --no-metadata
[..]
   Packaging aws-smithy-observability-otel v0.1.0 (/Users/byron/dev/github.com/awslabs/aws-sdk-rust/sdk/aws-smithy-observability-otel)
    Updating crates.io index
    Packaged 12 files, 83.4KiB (20.6KiB compressed)
   Packaging aws-smithy-types-convert v0.60.9 (/Users/byron/dev/github.com/awslabs/aws-sdk-rust/sdk/aws-smithy-types-convert)
    Updating crates.io index
    Packaged 10 files, 36.9KiB (11.0KiB compressed)
   Packaging aws-smithy-wasm v0.1.4 (/Users/byron/dev/github.com/awslabs/aws-sdk-rust/sdk/aws-smithy-wasm)
    Updating crates.io index
    Packaged 9 files, 40.2KiB (12.7KiB compressed)

aws-sdk-rust ( main) via 🐍 took 11m49s

The changes require the gitoxide version to be used, but if someone is interested, there should be no problem in backporting the changes to the git2 version.

If anyone else happens to hit this issue, we did find a workaround. In our publishing step we now rm -rf .git/ before running cargo package and that has taken a full workspace publish from ~3 hours to ~15 minutes. Would still be nice to have this fixed in the cargo package command itself, but we are back to around the performance we had previously.

imo providing --no-vcs-info should be a last resort

  • This extends the CLI for a fairly limited case where the performance matters and we couldn’t help it in other ways. The larger the CLI, the harder it is for people to find things that are relevant
  • There is a lot of interest in packages published to crates.io to have VCS info to check to see if anything has changed between VCS and publish. Improving that situation is the reason we have this performance regression (#13960).

imo before we consider alternatives, we should see what the limit is for how much we can optimize what we are already doing and see if that is within an acceptable limit (determined by the Cargo team).

If we do look for more alternatives, some additional ideas include:

  • Generate VCS info but skip dirty info with --allow-dirty (this seems a bit magical)
  • Remove dirty info from VCS info: its advisory only anyways