cargo: `cargo test --all-targets` does not run doc tests

As per title cargo test --all-targets does not run the doc tests on a crate. This is very surprising behaviour. I found this and it looks like running doc tests is done by rustdoc which is apparently not invoked when doing --all-targets.

About this issue

  • Original URL
  • State: open
  • Created 8 years ago
  • Reactions: 44
  • Comments: 25 (19 by maintainers)

Commits related to this issue

Most upvoted comments

Oh god, I think we’ve been not running doc tests on our CI for a couple of years due to this…

Note that in a future edition we could potentially make --all-targets default to this, which would make this an interim option for current editions.

This just bit my projects — apparently we haven’t been running doctests in CI for months.

I find it very confusing that a bare cargo test includes doctests and excludes examples, but then adding --all-targets adds examples while excluding doctests. While I find the non-additive behavior unintuitive, I don’t claim to understand the nuances involved so perhaps it is my intuition that is flawed.

I’d like to propose amending the --help entry for --all-targets to explicitly mention the “doctests get explicitly excluded” caveat. Right now, that entry merely reads --all-targets Test all targets which I feel encourages users to use that flag as a footgun, just as I did.

If I were to create a PR to add the doctests caveat to the --help entry for --all-targets, is that something that the maintainers would consider merging?

Hmm, I wonder if we can have a project-level switch (maybe in the .cargo/config) for this? cargo c --all-features --all-targets is already pretty verbose and I’m not sure making it more onerous to do the right thing is the best way forward. So for projects that are aware of this and want to eagerly fix the doctests breakage, it would be great if we can opt-in to making --all-targets do (arguably) the right thing without waiting until 2027.

In today’s @rust-lang/cargo meeting, we agreed to try the experiment of adding an --all-targets-including-doc unstable option, initially to cargo test and then to other commands like cargo build and cargo check. We don’t yet know what we should call it, but we can experiment with implementation of it.

Just for some clarification, cargo has a model where there is a distinction between a target and a mode. Doctests are a mode, and that is why --all-targets by itself doesn’t run them. I understand that can be confusing, isn’t obvious from the documentation, and not the mental model some people have. We have talked in the past about making that more explicit (which in part would solve this issue by offering new switches to do both “run tests” and “run doctests”). I can’t find that discussion offhand, but I don’t think it was very deep anyway (just acknowledging that the CLI is limited and there were some ideas around a --mode flag). Perhaps that’s something the testing team could tackle 😉 .

Maybe it is worth to add some context to this issue.

We ran into this because we use cargo test --all-targets on Travis and usually cargo test locally. A not-compiling doc test sneaked into a last-minute update of a PR but the build was fine since we use --all-targets on Travis. Later on, someone complained that we have a broken master because of a non-compiling doc test.

I read some of the discussion in the #6671 and TBH, the whole --doc is a mode would be SUPER confusing. As a user, I expect cargo test to test LESS than cargo test --all-targets since the latter has all in the name. In addition, the documentation (--help) lists doc tests next to all the other targets.

Ah, well, it that case it makes sense to have a new flag for “all targets and docs” 😄 .

This means the command to run absolutely all tests everywhere is cargo test --workspace, not cargo test --workspace --all-targets, right?

It depends on the structure of your packages, but in general there is currently no single command to “test everything”. cargo test is intended to be the usual command. If you also want to test benchmarks and examples, you can either mark them with test=true in Cargo.toml, or run two commands (cargo test --all-targets and cargo test --doc).

This means the command to run absolutely all tests everywhere is cargo test --workspace, not cargo test --workspace --all-targets, right?