cargo: Tracking the performance of `-Zchecksum-freshness`

This issue is intended for tracking performance related discussion and benchmarks for #14136. #14136 itself is for overall progress reports.

Useful cargo script to summarize number and size of files got checksum'd

#!/usr/bin/env cargo
---cargo
[dependencies]
walkdir = "2"
---

use std::collections::HashMap;
use std::ffi::OsStr;
use std::fs::File;
use std::io::prelude::*;
use std::io::BufReader;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let walker = walkdir::WalkDir::new("target/debug/deps")
        .into_iter()
        .filter_entry(|e| e.file_type().is_dir() || e.path().extension() == Some(OsStr::new("d")));

    let mut map = HashMap::with_capacity(8192);

    for entry in walker {
        let entry = entry?;
        if entry.file_type().is_dir() {
            continue;
        }
        for line in BufReader::new(File::open(entry.path())?).lines() {
            let line = line?;
            let Some(rest) = line.strip_prefix("# checksum:") else {
                continue;
            };
            let mut parts = rest.splitn(3, ' ');
            let (_, size, path) = (parts.next(), parts.next().unwrap(), parts.next().unwrap());
            let size: u64 = size.strip_prefix("file_len:").unwrap().parse()?;
            if let Some(old_size) = map.insert(path.to_string(), size) {
                assert_eq!(old_size, size, "size doesn't match for {path}: {old_size} vs {size}");
            }
        }
    }

    let mut stdout = std::io::stdout().lock();
    writeln!(stdout, "number of files: {}", map.len())?;
    writeln!(stdout, "total file size: {}", map.values().sum::<u64>())?;

    Ok(())
}

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 15 (13 by maintainers)

Most upvoted comments

The main reason I’m excited about this is because I’m running on CI, and all my mtimes change with each run, so my valid cache contents are never used. I think this is an extremely common problem.

If folks decide the overhead of always doing the checksum every time slows down the nothing-to-do case too much, please consider defaulting to an alternative scheme:

  1. do the existing mtime/size/whatever check; if it says the file is unchanged, it’s unchanged.
  2. if it says the file is changed, look more closely with the checksum. There’s really no harm in this; the IO is likely warming the cache for later rather than doubling the cost. And the CPU used for blake3 is insignificant compared to the actual compile.

Would it be possible to have a mode where the checksums are computed only when mtime shows that the file is touched to avoid unnecessary recompiles when nothing actually changed? This would still have the same correctness issues as mtime, but speed up some cases that behave badly with mtime. For example in cg_clif’s test suite this would be really useful to avoid unnecessarily rebuilding crates whose sources are copied and patched before building.

You can use -Zchecksum-freshness already on nightly.

Originally posted at https://github.com/rust-lang/cargo/issues/14136#issuecomment-2430876726.

Here is a result of a performance benchmark that mostly tests the rustc part.

  • It adds aws-sdk-ec2@1.80.0 as a dependency and runs cargo clean && cargo check.
  • rg 'checksum:blake3' target/debug/deps | wc -l shows 8313 matches. It means rustc roughly computes blake3 checksum 8313 times.
  • Checksum computations on Cargo side are mostly ignored because Cargo assumes registry are immutable and won’t do either mtime or checksum check .

Result: On par if you have zero local code.

Command Mean [s] Min [s] Max [s] Relative
cargo +nightly check --frozen 52.986 ± 0.366 52.549 53.681 1.00
cargo +nightly check --frozen -Zchecksum-freshness 53.020 ± 0.315 52.442 53.480 1.00 ± 0.01
Benchmark details

Setup package

cargo new foo --lib
cd foo
cargo add aws-sdk-ec2

Benchmark script

hyperfine --warmup 1 \
  --setup "cargo fetch" \
  --min-runs 20 \
  --prepare "cargo clean" \
  --export-markdown result.md \
  "cargo +nightly check --frozen" \
  "cargo +nightly check --frozen -Zchecksum-freshness"

Platform information

cargo 1.82.0 (8f40fc59f 2024-08-21)
release: 1.82.0
commit-hash: 8f40fc59fb0c8df91c97405785197f3c630304ea
commit-date: 2024-08-21
host: x86_64-unknown-linux-gnu
libgit2: 1.8.1 (sys:0.19.0 vendored)
libcurl: 8.9.0-DEV (sys:0.4.74+curl-8.9.0 vendored ssl:OpenSSL/1.1.1w)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Amazon Linux AMI 2.0.0 [64-bit] (r7a.24xlarge)