cargo: Tracking the performance of `-Zchecksum-freshness`
This issue is intended for tracking performance related discussion and benchmarks for #14136. #14136 itself is for overall progress reports.
Useful cargo script to summarize number and size of files got checksum'd
#!/usr/bin/env cargo
---cargo
[dependencies]
walkdir = "2"
---
use std::collections::HashMap;
use std::ffi::OsStr;
use std::fs::File;
use std::io::prelude::*;
use std::io::BufReader;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let walker = walkdir::WalkDir::new("target/debug/deps")
.into_iter()
.filter_entry(|e| e.file_type().is_dir() || e.path().extension() == Some(OsStr::new("d")));
let mut map = HashMap::with_capacity(8192);
for entry in walker {
let entry = entry?;
if entry.file_type().is_dir() {
continue;
}
for line in BufReader::new(File::open(entry.path())?).lines() {
let line = line?;
let Some(rest) = line.strip_prefix("# checksum:") else {
continue;
};
let mut parts = rest.splitn(3, ' ');
let (_, size, path) = (parts.next(), parts.next().unwrap(), parts.next().unwrap());
let size: u64 = size.strip_prefix("file_len:").unwrap().parse()?;
if let Some(old_size) = map.insert(path.to_string(), size) {
assert_eq!(old_size, size, "size doesn't match for {path}: {old_size} vs {size}");
}
}
}
let mut stdout = std::io::stdout().lock();
writeln!(stdout, "number of files: {}", map.len())?;
writeln!(stdout, "total file size: {}", map.values().sum::<u64>())?;
Ok(())
}
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 15 (13 by maintainers)
The main reason I’m excited about this is because I’m running on CI, and all my mtimes change with each run, so my valid cache contents are never used. I think this is an extremely common problem.
If folks decide the overhead of always doing the checksum every time slows down the nothing-to-do case too much, please consider defaulting to an alternative scheme:
Would it be possible to have a mode where the checksums are computed only when mtime shows that the file is touched to avoid unnecessary recompiles when nothing actually changed? This would still have the same correctness issues as mtime, but speed up some cases that behave badly with mtime. For example in cg_clif’s test suite this would be really useful to avoid unnecessarily rebuilding crates whose sources are copied and patched before building.
You can use
-Zchecksum-freshnessalready on nightly.Originally posted at https://github.com/rust-lang/cargo/issues/14136#issuecomment-2430876726.
Here is a result of a performance benchmark that mostly tests the rustc part.
aws-sdk-ec2@1.80.0as a dependency and runscargo clean && cargo check.rg 'checksum:blake3' target/debug/deps | wc -lshows 8313 matches. It means rustc roughly computes blake3 checksum 8313 times.Result: On par if you have zero local code.
cargo +nightly check --frozencargo +nightly check --frozen -Zchecksum-freshnessBenchmark details
Setup package
Benchmark script
Platform information