cargo: Path to registry index is not cross-platform stable (i.e. hash part of `registry/index/index.crates.io-`)
Hello. I have an issue with cargo-fetch behaving differently than cargo-vendor. In a part of the help section of cargo fetch, it says for the flag --target
Fetch for the given architecture. The default is all architectures. The general format of the triple is
<arch><sub>-<vendor>-<sys>-<abi>. Runrustc --print target-listfor a list of supported targets. This flag may be specified multiple times.
However, I believe this is untrue since in some architectures, they cannot find their dependencies. I haven’t tried the --target flag yet as a workaround but given that some target architectures cannot find their dependencies is weird to me, since it says that the default is to use “all” target architectures.
Here is a comparison between cargo vendor vs cargo fetch.
- Vendor https://build.opensuse.org/package/show/home:uncomfyhalomacro:branches:editors:vendor/kak-lsp
- Fetch https://build.opensuse.org/package/show/home:uncomfyhalomacro:branches:editors:registry/kak-lsp
You can see here that in https://build.opensuse.org/package/show/home:uncomfyhalomacro:branches:editors:registry/kak-lsp, only aarch64, x86_64, and ppc64le are able to find their dependencies and compile (whether it fails or not). i586, armv7l, and s390x are the architectures that were not able to find their dependencies.
Everything in https://build.opensuse.org/package/show/home:uncomfyhalomacro:branches:editors:vendor/kak-lsp are able to find their dependencies and are able to compile (whether it fails or not).
I would like to know if there is an inaccuracy with the documentation or not.
For context, I’ve been helping out to build a tool to ease the process of vendoring dependencies in openSUSE/SUSE. https://github.com/openSUSE-Rust/obs-service-cargo.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 28 (19 by maintainers)
Commits related to this issue
- feat(SourceId): use stable hash from rustc-stable-hash This helps `-Ztrim-paths` build a stable cross-platform path for the registry and git sources. Sources files then can be found from the same pat... — committed to weihanglo/cargo by weihanglo 2 years ago
- feat(SourceId): use stable hash from rustc-stable-hash This helps `-Ztrim-paths` build a stable cross-platform path for the registry and git sources. Sources files then can be found from the same pat... — committed to weihanglo/cargo by weihanglo 2 years ago
- feat(SourceId): use stable hash from rustc-stable-hash This helps `-Ztrim-paths` build a stable cross-platform path for the registry and git sources. Sources files then can be found from the same pat... — committed to weihanglo/cargo by weihanglo 2 years ago
- feat(SourceId): use stable hash from rustc-stable-hash This helps `-Ztrim-paths` build a stable cross-platform path for the registry and git sources. Sources files then can be found from the same pat... — committed to weihanglo/cargo by weihanglo 2 years ago
- feat(SourceId): use stable hash from rustc-stable-hash This helps `-Ztrim-paths` build a stable cross-platform path for the registry and git sources. Sources files then can be found from the same pat... — committed to weihanglo/cargo by weihanglo 2 years ago
- feature(SourceId): use stable hash from rustc-stable-hash (#14917) ### What does this PR try to resolve? This helps `-Ztrim-paths` build a stable cross-platform path for the registry and git sources... — committed to rust-lang/cargo by epage 2 years ago
Double checking, It doesn’t thrash the cache (throw out on each switch, like RUSTFLAGS today for rlibs) but keeps the caches between the old and new version separate (like my PR for RUSTFLAGS for rlibs)? This would be a one-time extra download and extra space taken up until the user cleans it up or we stabilize GC. Not great but not the worst.
iirc there are other parts of the project that are seeking out crytograhically secure hashes, making this line up with those, right?
Going to reopen and use this for proposing using stable hash for source URLs.
tl;dr:
During the course of designing remap rules for
-Ztrim-paths, we found that the source URL hash in Cargo is never platform-dependent. I propose we switch to a cross-platform hash algorithm for source URL hashes. It benefits debugging when sharing source and debug info between different platforms, especially for-Ztrim-pathsin the future.The remaining question is the hash algorithm.
In https://github.com/rust-lang/cargo/pull/14116 we tried out
rustc-stable-hashto experiment cross-platform (i.e., endianness and bit-width independen) hash algorithms for stableSourceIdhashes, including the default one SipHash and BLAKE3. The benchmarking shows thatcargofor 4000+ hashes on 10-core Apple M1 Pro.Therefore, I favor and propose BLAKE3 as our stable hash algorithm for these reasons,
@rfcbot fcp merge
Besides, I’d like to call out some area this stable hash proposal doesn’t cover:
C:\,D:\. When using path, local registry, or directory sources, these prefixes will be included in hash and lead to a different hash output. While I don’t think it is a big issue, w had better keep in mind the “cross-platform” source URL hash is not truly cross-platform in this regard.Yes. As I said the stable path is best-effort. We don’t guarantee the permanent path of it.
I’ll propose to the team for a stable hash next week (if I remember)
Thanks for your understanding!
Sure! So the plan now is to create
not sure when that will happen but for now you can close this issue as that literally answered all my questions!
Hi. I tried using the
--targetflag in this commit https://github.com/openSUSE-Rust/obs-service-cargo/commit/a34992cbf696150573cd89e2a0e607b1f2380a45 for my project but it seems it’s still not budging in https://build.opensuse.org/package/show/home:uncomfyhalomacro:branches:editors:registry/kak-lsp