cargo: Cargo fails to run when an *optional* dependency is missing

#!/usr/bin/env bash

set -e

cd /tmp
[ -d cargo-optional-deps ] && rm -rf cargo-optional-deps
cargo new cargo-optional-deps
cd cargo-optional-deps

cargo build
cat >> Cargo.toml <<EOF
optional_dep_does_not_exist = { version = "1.0", optional = true }
EOF
cargo build

Fails with

error: no matching package named optional_dep_does_not_exist found (required by cargo-optional-deps)

I did not expect this because the dependency is not required as per http://doc.crates.io/manifest.html#the-features-section:

optional dependencies, which enhance a package, but are not required; and

I’m running into this behavior while integrating Cargo into a private build system that wants control over dependencies. As such, I need to elide optional dependencies by default in this system’s dependency list.

I’m opening this issue to get some input on:

  1. Would ignoring missing optional dependencies for inactive features be a problem for Cargo?
  2. I tried to look through existing issues - has something like this been discussed before?
  3. An idea about resourcing/priority of the feature and how it fits into the roadmap for Cargo’s integration with existing systems.

About this issue

  • Original URL
  • State: open
  • Created 9 years ago
  • Reactions: 14
  • Comments: 24 (14 by maintainers)

Commits related to this issue

Most upvoted comments

We hit the same issue with an mix of open source and private optional dependencies. We have a public crate A that has an optional dependencies on a private crate B (behind a feature). Cargo is failing to resolve the private crate for anyone who uses A even though it’s behind a feature flag that not enabled.

I’d like to add another “me too” case. I’m working on packaging crates in Fedora. We cannot use crates.io directly, and all crates that are needed during the package build process must be packaged as RPMs. Not all crates are packaged (for various reasons, sometimes lack of resources, sometimes we don’t want to package something), and we want to use cargo build with some optional features disabled and without those crates installed. This is very close to the situation described in the original report for this bug, except that this is a public distro so everything is in the open.

Right now the only workable solution seems to be to edit Cargo.toml files to remove the optional dependencies we don’t want to enable, but that is a terrible hack and makes it much harder to automate the whole process.

We also hit this just now.

We have a few internal crates (as path dependencies) that we’d like to optionally enable when a local dev feature is enabled, but we want those crates to never be required in any capacity by our public users.

We now have to either:

  • patch Cargo.toml locally and add those dependencies in, which is not ergonomic on the dev side.
  • add dummy crates and require the users to work form a workspace with those in, which is also very very ugly

I believe this is a very common use case that I’ve seen and used in most of the projects I’ve worked on in my career so far.

I am also open to any horrible hacks and/or suggestions on how to work around this 😅

Has a decision been made for this issue? I have encountered a case where it is really needed. In fact, one of my dependencies is optional, and needs a crate that is generated locally. I want to be able to build my project without this dependency when its feature isn’t used, and get a compile error if I try to build with this feature and the generated code is missing.

Is there no possible workaround for now?

@cramertj heh I’ve thought we’d hit a problem like this at some point.

A random solution which may help solve this though is to perhaps list the valid targets for a project in a workspace root Cargo.toml. That way Cargo could be smarter and just vendor dependencies for those targets (and alter resolution so it won’t include winapi in the lock file)

So the reason that Cargo needs optional dependencies is that when generating a lock file

I think you’ve hit the nail on the head here. I’m not sure “lock files” are necessary in our system which has its ways of ensuring reproducible builds. Our package template for Rust projects has a .gitignore for Cargo.lock.

but rather you won’t be directly downloading from crates.io

Yup, exactly. Sorry for the ambiguity. We have an “import” process to pull an external dependency into our build system. That process is aware of crates.io.

One thought I had when writing this is that maybe some “shim crates” could work for now?

I’m open to the idea but also skeptical. Right now I don’t care about 32 bit, Windows, etc. There are trees of dependencies I don’t need to worry about if I just remove them. That works great because I can just scrub them from the Cargo.toml and nothing breaks. Optionals are harder because enabling the feature becomes impossible without “unscrubbing” the manifest on demand.

In that sense it may be best to follow the “build system integration” threads and features to see how we can better accommodate your build system

Yeah, I am (trying to). If something in particular is worth investigation, I’d appreciate if you could ping me.

The Servo project is still hitting into this issue. We share a number of crates with Gecko (notable Stylo and Webrender). These crates treat the Mozilla repo as upstream with the Servo project syncing copies to Github. The crates optionally (behind a gecko feature flag) depend on internal Gecko crates (that bind to C++ code in Gecko) that don’t exist outside of the Gecko repository.

This issue means that we have to manually maintain a diff to change those Cargo.toml between the Gecko and Servo copies of the code.

Here’s why the Firefox thing works if anyone’s curious: https://mozilla.logbot.info/servo/20180917#c15322773

LogBot was decommissioned, but here’s that conversation extracted from the archive:
2018-09-17T09:56:19 #servo <emilio> SimonSapin: around? Can I ask you a cargo question?
2018-09-17T09:56:40 #servo <emilio> SimonSapin: so I wanted to unify Gecko's and Servo's style/Cargo.toml
2018-09-17T09:56:51 #servo <emilio> SimonSapin: which right now differ on:
2018-09-17T09:57:36 #servo <emilio> https://www.irccloud.com/pastebin/77W5Ampk/t.diff
2018-09-17T09:58:27 #servo <emilio> SimonSapin: I thought moving the gecko_debug bit somewhere else I'd be able to unify them, since Gecko uses without problems crates that refer to non-existing path that are optional
2018-09-17T09:59:01 #servo <emilio> SimonSapin: but even after that I'm not able to make cargo cope :)
2018-09-17T09:59:08 #servo <emilio> SimonSapin: any idea?
2018-09-17T09:59:13 #servo <emilio> nox: ^ In case you're around
2018-09-17T09:59:40 #servo <SimonSapin> looking
2018-09-17T10:02:33 #servo <emilio> SimonSapin: (in particular, note how gecko builds just fine with `atoms = {path = "../atoms", optional = true}` even though that path is nowhere in m-c). I _assumed_ that that was because of the feature cargo needed to bother looking at the nsstring Cargo.toml, but I'm wrong apparently.
2018-09-17T10:02:42 #servo <SimonSapin> Does Gecko really do that? I think Cargo wants to resolve the full potential dependency graph before doing feature selection
2018-09-17T10:03:26 #servo <emilio> SimonSapin: well it does afaik, but the gecko build may be more complex than just `cargo build` I guess
2018-09-17T10:03:42 #servo <emilio> SimonSapin: maybe I should ask #build instead :)
2018-09-17T10:03:53 #servo <SimonSapin> uh
2018-09-17T10:04:26 #servo <SimonSapin> emilio: I was gonna say, I think I made those changes when removing nsstring bindings from servo/servo, but we could make it an empty crate there instead
2018-09-17T10:04:41 #servo <SimonSapin> but now I wonder how things work out for servo_atoms in gecko
2018-09-17T10:05:11 #servo <emilio> SimonSapin: yeah, that's the other alternative, but I thought there should be a way to make this work... I'll figure out how Gecko does the dependency graph bits.
2018-09-17T10:06:24 #servo <SimonSapin> servo atom is indeed not present in https://searchfox.org/mozilla-central/source/Cargo.lock
2018-09-17T10:23:38 #servo <emilio> SimonSapin: 12:22 <@ted> i have no idea why that works, sorry!
2018-09-17T10:23:47 #servo <SimonSapin> eh
2018-09-17T10:23:55 #servo <emilio> SimonSapin: magic.gif :)
2018-09-17T10:25:07 #servo <emilio> https://github.com/rust-lang/cargo/issues/4544 is open, as expected
2018-09-17T10:25:07 #servo <crowbot> Issue #4544: Cargo fails to run when an *optional* dependency is missing - https://github.com/rust-lang/cargo/issues/4544
2018-09-17T10:25:33 #servo <emilio> oh well
2018-09-17T10:26:16 #servo <emilio> SimonSapin: I may look at what's going on in there if I find some spare time, oh well
2018-09-17T10:26:17 #servo <ted> i would not be surprised to find that it works by accident
2018-09-17T10:26:36 #servo <emilio> ted: yeah, me neither... To bad that now we depend on it heavily :(
2018-09-17T10:27:13 #servo <emilio> ted: do you know what are the biggest differences from a normal `cargo build` in `m-c`? I guess mostly vendored stuff I guess?
2018-09-17T10:27:25 #servo <emilio> s/I guess?//
2018-09-17T10:27:56 #servo <ted> we do write out .cargo/config to do source replacement for vendoring
2018-09-17T10:28:01 #servo <ted> but we also pass a ton of options to cargo
2018-09-17T10:28:14 #servo <ted> https://dxr.mozilla.org/mozilla-central/source/config/rules.mk#957
2018-09-17T10:28:55 #servo <ted> i guess we always run `cargo rustc` not `cargo build`
2018-09-17T10:33:22 #servo <paul> standups: working on making the compositor more embedder friendly: optional at startup, support multiple windows, not bound to the embedder thread.
2018-09-17T10:33:22 #servo <standups> Ok, submitted #59981 for https://www.standu.ps/user/paul/
2018-09-17T10:40:25 #servo <emilio> SimonSapin: ted: Oh, figured out
2018-09-17T10:40:45 #servo <emilio> SimonSapin: ted: (RUST_LOG=cargo is amazing)
2018-09-17T10:41:06 #servo <emilio> SimonSapin: ted: It was erroring in cargo::core::workspace: find_members
2018-09-17T10:41:16 #servo <emilio> SimonSapin: ted: Gecko excludes `servo/` from the workspace
2018-09-17T10:41:23 #servo <ted> aha
2018-09-17T10:41:59 #servo <SimonSapin> uh… so in gecko, the style crate is a path dependency but not a workspace member?
2018-09-17T10:42:14 #servo <SimonSapin> I’m not even sure what that means
2018-09-17T10:42:19 #servo <emilio> SimonSapin: if I add `exclude = ["components/style"]` to servo, it works. Now, I guess we don't really want to do that, but maybe excluding just the non-existing path works well enough
2018-09-17T10:43:08 #servo <emilio> SimonSapin: well, the side effects are nice, imho...
2018-09-17T10:43:28 #servo <ted> SimonSapin: i think it's just because servo defines its own workspace?
2018-09-17T10:43:33 #servo <ted> or maybe because of this, i can't remember
2018-09-17T10:43:42 #servo <SimonSapin> emilio: I mean I don’t know what the other effects are
2018-09-17T10:44:06 #servo <SimonSapin> emilio: does it it work to exclude only support/ ?
2018-09-17T10:45:11 #servo <emilio> SimonSapin: nope, `Unable to update file:///home/emilio/src/moz/servo/support/gecko/nsstring`
2018-09-17T10:46:15 #servo <emilio> SimonSapin: curious what that does to the lockfile, here's the patch:
2018-09-17T10:46:30 #servo <emilio> https://www.irccloud.com/pastebin/4cgrcEGa/t.diff
2018-09-17T10:46:44 #servo <emilio> SimonSapin: I guess it's expected
2018-09-17T10:47:09 #servo <emilio> SimonSapin: (but still, lol)
2018-09-17T10:48:43 #servo <emilio> ted: so Servo used to define its own workspace when we vendored the whole thing, but now we don't, yet we rely on that for everything not to blow up :D
2018-09-17T10:48:58 #servo <ted> heh
2018-09-17T10:49:24 #servo <SimonSapin> I think nested workspaces are ignored, though
2018-09-17T10:51:13 #servo <emilio> SimonSapin: I guess the best way forward for me is to land a cargo patch to not hard-error when broken optional dependencies are found... Does that sound right?
2018-09-17T10:51:36 #servo <emilio> SimonSapin: that should allow us to remove the weird `exclude =` thing as well in Gecko
2018-09-17T10:52:09 #servo <SimonSapin> emilio: I don’t know if the cargo maintainers will want to take that patch, but it may be worth a try
2018-09-17T10:52:38 #servo <SimonSapin> maybe file an issue first in case they don’t, so you don’t do the work if it’s not gonna land
2018-09-17T10:54:37 #servo <emilio> SimonSapin: looks like the patch is trivial to write so I may as well test it and submit it.

I’ve just hit the exact same situation as @siscia, where I have an open source project with an optional proprietary component that adds a few extra features. I’m currently working around it by checking in an empty package to the open source project, and then instructing users to overwrite that with the proprietary code if they need it, but this is pretty bad ergonomically.

Has any further thought been given to what a solution here might look like? I’d be happy to help with implementation if there’s a decision about what should be done. Currently, we’re checking in a huge number of vendored crates into the fuchsia tree because they’re pulled in as optional dependencies of something or other, and I’d like to make it possible to omit them.