cargo: Path to registry index is not cross-platform stable (i.e. hash part of `registry/index/index.crates.io-`)

Hello. I have an issue with cargo-fetch behaving differently than cargo-vendor. In a part of the help section of cargo fetch, it says for the flag --target

Fetch for the given architecture. The default is all architectures. The general format of the triple is <arch><sub>-<vendor>-<sys>-<abi>. Run rustc --print target-list for a list of supported targets. This flag may be specified multiple times.

However, I believe this is untrue since in some architectures, they cannot find their dependencies. I haven’t tried the --target flag yet as a workaround but given that some target architectures cannot find their dependencies is weird to me, since it says that the default is to use “all” target architectures.

Here is a comparison between cargo vendor vs cargo fetch.

You can see here that in https://build.opensuse.org/package/show/home:uncomfyhalomacro:branches:editors:registry/kak-lsp, only aarch64, x86_64, and ppc64le are able to find their dependencies and compile (whether it fails or not). i586, armv7l, and s390x are the architectures that were not able to find their dependencies.

Everything in https://build.opensuse.org/package/show/home:uncomfyhalomacro:branches:editors:vendor/kak-lsp are able to find their dependencies and are able to compile (whether it fails or not).

I would like to know if there is an inaccuracy with the documentation or not.

For context, I’ve been helping out to build a tool to ease the process of vendoring dependencies in openSUSE/SUSE. https://github.com/openSUSE-Rust/obs-service-cargo.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 28 (19 by maintainers)

Commits related to this issue

Most upvoted comments

The cost of changing hash algorithm is it thrashes global caches (index files, git/registry sources) between Rust toolchain versions when we switch. We want to avoid that unless really really needed.

Double checking, It doesn’t thrash the cache (throw out on each switch, like RUSTFLAGS today for rlibs) but keeps the caches between the old and new version separate (like my PR for RUSTFLAGS for rlibs)? This would be a one-time extra download and extra space taken up until the user cleans it up or we stabilize GC. Not great but not the worst.

iirc there are other parts of the project that are seeking out crytograhically secure hashes, making this line up with those, right?

Going to reopen and use this for proposing using stable hash for source URLs.

tl;dr:

  • Propose to switch to cross-platform stable hash from platform-dependent stable hash for source URL
  • Propose to choose BLAKE3 as the hash algorithm of cross-platform stable hash

During the course of designing remap rules for -Ztrim-paths, we found that the source URL hash in Cargo is never platform-dependent. I propose we switch to a cross-platform hash algorithm for source URL hashes. It benefits debugging when sharing source and debug info between different platforms, especially for -Ztrim-paths in the future.

The remaining question is the hash algorithm.

In https://github.com/rust-lang/cargo/pull/14116 we tried out rustc-stable-hash to experiment cross-platform (i.e., endianness and bit-width independen) hash algorithms for stable SourceId hashes, including the default one SipHash and BLAKE3. The benchmarking shows that

  • The performance of stable SipHash is on par with or a bit faster than the original one, while BLAKE3 is 2-3x slower.
  • The actual time spent is around 1.5ms when using BLAKE3 on medium-size projects like cargo for 4000+ hashes on 10-core Apple M1 Pro.

Therefore, I favor and propose BLAKE3 as our stable hash algorithm for these reasons,

  • The real world performance regression is neglectable comparing to other parts in a cargo invocation.
  • BLAKE3 is a cryptographic hash function. While it is not a problem for crates.io itself, it is still possible for other registry or git repo to intentionally make collisions for source URL hashes. It’s better if we take preventive course now than later.

@rfcbot fcp merge


Besides, I’d like to call out some area this stable hash proposal doesn’t cover:

  • File paths on Windows have prefixes like C:\, D:\. When using path, local registry, or directory sources, these prefixes will be included in hash and lead to a different hash output. While I don’t think it is a big issue, w had better keep in mind the “cross-platform” source URL hash is not truly cross-platform in this regard.

Yes. As I said the stable path is best-effort. We don’t guarantee the permanent path of it.

I’ll propose to the team for a stable hash next week (if I remember)

Thanks for your understanding!

Sure! So the plan now is to create

one single index cache for all platforms

not sure when that will happen but for now you can close this issue as that literally answered all my questions!