cargo: `cargo test --all-targets` does not run doc tests
As per title cargo test --all-targets does not run the doc tests on a crate. This is very surprising behaviour. I found this and it looks like running doc tests is done by rustdoc which is apparently not invoked when doing --all-targets.
About this issue
- Original URL
- State: open
- Created 8 years ago
- Reactions: 44
- Comments: 25 (19 by maintainers)
Commits related to this issue
- *: speed up test build (#7012) Tests has been built slowly lately. That's because we use multiple compile command to build tests for different crates. Things go worse when cdc is split as a compone... — committed to tikv/tikv by BusyJay 7 years ago
- *: speed up test build (#7012) Tests has been built slowly lately. That's because we use multiple compile command to build tests for different crates. Things go worse when cdc is split as a compone... — committed to zhangjinpeng87/tikv by BusyJay 7 years ago
- *: speed up test build (#7012) Tests has been built slowly lately. That's because we use multiple compile command to build tests for different crates. Things go worse when cdc is split as a compone... — committed to c1ay/tikv by BusyJay 7 years ago
- Actually run doc tests in CI https://github.com/rust-lang/cargo/issues/6669 — committed to mlafeldt/codebreaker-rs by mlafeldt 7 years ago
- [ci] Run doctests too. See https://github.com/rust-lang/cargo/issues/6669 — committed to anp/moxie by anp 6 years ago
- [ci] Run doctests too. See https://github.com/rust-lang/cargo/issues/6669 — committed to anp/moxie by anp 6 years ago
- More robust cargo test Running doctests and all-targets (see https://github.com/rust-lang/cargo/issues/6669) Also bump version — committed to pbougue/transit_model by deleted user 6 years ago
- More robust cargo test Running doctests and all-targets (see https://github.com/rust-lang/cargo/issues/6669) — committed to pbougue/transit_model by deleted user 6 years ago
- Do not call `cargo test` with `--all-targets` The `--all-targets` flag for `cargo test` actually DISABLES doctests. Doctests should of course be run, so removing the flag from the CI scrips. There is... — committed to jonasbb/serde_with by jonasbb 6 years ago
- Merge #253 253: Do not call `cargo test` with `--all-targets` r=jonasbb a=jonasbb The `--all-targets` flag for `cargo test` actually DISABLES doctests. Doctests should of course be run, so removing ... — committed to jonasbb/serde_with by bors[bot] 6 years ago
- azure: run doc tests `cargo test --all-targets` will not run the doc tests are documented in this issue: - https://github.com/rust-lang/cargo/issues/6669 — committed to CybercentreCanada/sawp by cccs-sadugas 6 years ago
- chore(tests): Simplify `cargo test` usage We currently use `--all-targets` and `--workspace` in `cargo test` invocations in a few places but: * `--workspace` isn't needed since we don't specify `w... — committed to vectordotdev/vector by jszwedko 6 years ago
- chore(tests): Simplify `cargo test` usage (#7287) We currently use `--all-targets` to include benches (`--bench`), but `--all-targets` does not include `--doc` apparently (https://github.com/rust-lan... — committed to vectordotdev/vector by jszwedko 6 years ago
- Turns out that `--benches` still executes the unit tests Fold back into one step that just executes them all. Notably doc tests still need a separate `cargo test` run due to https://github.com/rust-... — committed to vectordotdev/vector by jszwedko 6 years ago
- Define the Catalogue type to store tiles This is used by n18map::Map and n18game::Game to manage tiles. In the process, I discovered that "cargo test --all-targets" does not run doc tests; see https... — committed to robmoss/rusty_train by robmoss 5 years ago
- build: `--all-targets` quietly skip cargo doc test https://github.com/rust-lang/cargo/issues/6669 — committed to azzamsa/dryip-of-rust by azzamsa 4 years ago
- Add action to run doctest. `cargo test --all-features` does not run doc-tests. For more information see https://github.com/rust-lang/cargo/issues/6669. — committed to Wasabi375/rust-ci-conf by Wasabi375 4 years ago
- Add action to run doctest. `cargo test --all-features` does not run doc-tests. For more information see https://github.com/rust-lang/cargo/issues/6669. — committed to Wasabi375/rust-ci-conf by Wasabi375 4 years ago
- Add action to run doctest. `cargo test --all-features` does not run doc-tests. For more information see https://github.com/rust-lang/cargo/issues/6669. — committed to Wasabi375/rust-ci-conf by Wasabi375 4 years ago
- Add action to run doctest. (#3) `cargo test --all-features` does not run doc-tests. For more information see https://github.com/rust-lang/cargo/issues/6669. — committed to jonhoo/rust-ci-conf by Wasabi375 4 years ago
Oh god, I think we’ve been not running doc tests on our CI for a couple of years due to this…
Note that in a future edition we could potentially make
--all-targetsdefault to this, which would make this an interim option for current editions.This just bit my projects — apparently we haven’t been running doctests in CI for months.
I find it very confusing that a bare
cargo testincludes doctests and excludes examples, but then adding--all-targetsadds examples while excluding doctests. While I find the non-additive behavior unintuitive, I don’t claim to understand the nuances involved so perhaps it is my intuition that is flawed.I’d like to propose amending the
--helpentry for--all-targetsto explicitly mention the “doctests get explicitly excluded” caveat. Right now, that entry merely reads--all-targets Test all targetswhich I feel encourages users to use that flag as a footgun, just as I did.If I were to create a PR to add the doctests caveat to the
--helpentry for--all-targets, is that something that the maintainers would consider merging?Hmm, I wonder if we can have a project-level switch (maybe in the
.cargo/config) for this?cargo c --all-features --all-targetsis already pretty verbose and I’m not sure making it more onerous to do the right thing is the best way forward. So for projects that are aware of this and want to eagerly fix the doctests breakage, it would be great if we can opt-in to making--all-targetsdo (arguably) the right thing without waiting until 2027.In today’s @rust-lang/cargo meeting, we agreed to try the experiment of adding an
--all-targets-including-docunstable option, initially tocargo testand then to other commands likecargo buildandcargo check. We don’t yet know what we should call it, but we can experiment with implementation of it.Just for some clarification, cargo has a model where there is a distinction between a target and a mode. Doctests are a mode, and that is why
--all-targetsby itself doesn’t run them. I understand that can be confusing, isn’t obvious from the documentation, and not the mental model some people have. We have talked in the past about making that more explicit (which in part would solve this issue by offering new switches to do both “run tests” and “run doctests”). I can’t find that discussion offhand, but I don’t think it was very deep anyway (just acknowledging that the CLI is limited and there were some ideas around a--modeflag). Perhaps that’s something the testing team could tackle 😉 .Maybe it is worth to add some context to this issue.
We ran into this because we use
cargo test --all-targetson Travis and usuallycargo testlocally. A not-compiling doc test sneaked into a last-minute update of a PR but the build was fine since we use--all-targetson Travis. Later on, someone complained that we have a brokenmasterbecause of a non-compiling doc test.I read some of the discussion in the #6671 and TBH, the whole
--docis a mode would be SUPER confusing. As a user, I expectcargo testto test LESS thancargo test --all-targetssince the latter hasallin the name. In addition, the documentation (--help) lists doc tests next to all the other targets.Ah, well, it that case it makes sense to have a new flag for “all targets and docs” 😄 .
It depends on the structure of your packages, but in general there is currently no single command to “test everything”.
cargo testis intended to be the usual command. If you also want to test benchmarks and examples, you can either mark them withtest=trueinCargo.toml, or run two commands (cargo test --all-targetsandcargo test --doc).This means the command to run absolutely all tests everywhere is
cargo test --workspace, notcargo test --workspace --all-targets, right?