clap: Regression from 3.1 -> 3.2 in `values_of` (i think)
Please complete the following tasks
- I have searched the discussions
- I have searched the open and rejected issues
Rust Version
rustc 1.60.0 (7737e0b5c 2022-04-04)
Clap Version
3.2.1
Minimal reproducible code
use clap::{Arg, Command};
const INCLUDE: &str = "include";
fn arg_include() -> Arg<'static> {
Arg::new(INCLUDE)
.long(INCLUDE)
.short('i')
.takes_value(true)
.multiple_occurrences(true)
.allow_invalid_utf8(true)
}
fn app() -> Command<'static> {
Command::new("test").arg(arg_include())
}
#[derive(Copy, Clone, PartialEq, Debug)]
pub enum IncludeKind {
Quoted,
Bracketed,
}
#[derive(Clone, PartialEq, Debug)]
pub struct Include {
pub path: String,
pub kind: IncludeKind,
}
#[derive(Debug)]
struct Opt {
include: Vec<Include>,
}
fn parse() -> Opt {
let matches = app().get_matches_from(&["test", "--include", "/path/to/foo"]);
let include = matches
.values_of(INCLUDE)
.unwrap_or_default()
.map(|include| {
if include.starts_with('<') && include.ends_with('>') {
Include {
path: include[1..include.len() - 1].to_owned(),
kind: IncludeKind::Bracketed,
}
} else {
Include {
path: include.to_owned(),
kind: IncludeKind::Quoted,
}
}
})
.collect();
Opt {
include
}
}
fn main() {
dbg!(parse());
}
Steps to reproduce the bug with the above code
Using clap = "3.2"
with cargo run --release
, I get:
$ cargo run --release
thread 'main' panicked at 'Must use `_os` lookups with `Arg::allow_invalid_utf8`', /var/home/walters/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-3.2.1/src/parser/matches/arg_matches.rs:1651:13
stack backtrace:
0: rust_begin_unwind
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:584:5
1: core::panicking::panic_fmt
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/panicking.rs:143:14
2: clap::parser::matches::arg_matches::unwrap_string
3: <clap::parser::matches::arg_matches::OsValues as core::iter::traits::iterator::Iterator>::next
4: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
5: clap_regression::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Yet specifying clap = ">= 3.1, < 3.2"
(resolving to v3.1.18
), the program succeeds.
Actual Behaviour
Panic
Expected Behaviour
Don’t panic
Additional Context
This was seen in https://github.com/dtolnay/cxx/pull/1057
Debug Output
Finished release [optimized] target(s) in 12.03s
Running `target/release/clap-regression`
[ clap::builder::command] Command::_do_parse
[ clap::builder::command] Command::_build: name="test"
[ clap::builder::command] Command::_propagate:test
[ clap::builder::command] Command::_check_help_and_version: test
[ clap::builder::command] Command::_check_help_and_version: Removing generated version
[ clap::builder::command] Command::_propagate_global_args:test
[ clap::builder::command] Command::_derive_display_order:test
[ clap::parser::parser] Parser::get_matches_with
[ clap::parser::parser] Parser::get_matches_with: Begin parsing 'RawOsStr("--include")' ([45, 45, 105, 110, 99, 108, 117, 100, 101])
[ clap::parser::parser] Parser::get_matches_with: Positional counter...1
[ clap::parser::parser] Parser::get_matches_with: Low index multiples...false
[ clap::parser::parser] Parser::possible_subcommand: arg=Ok("--include")
[ clap::parser::parser] Parser::get_matches_with: sc=None
[ clap::parser::parser] Parser::parse_long_arg
[ clap::parser::parser] Parser::parse_long_arg: Does it contain '='...
[ clap::parser::parser] Parser::parse_long_arg: Found valid arg or flag '--include <include>'
[ clap::parser::parser] Parser::parse_long_arg("include"): Found an arg with value 'None'
[ clap::parser::parser] Parser::parse_opt_value; arg=include, val=None, has_eq=false
[ clap::parser::parser] Parser::parse_opt_value; arg.settings=ArgFlags(MULTIPLE_OCC | TAKES_VAL | UTF8_NONE)
[ clap::parser::parser] Parser::parse_opt_value; Checking for val...
[ clap::parser::parser] Parser::parse_opt_value: More arg vals required...
[ clap::parser::parser] Parser::get_matches_with: After parse_long_arg Opt([hash: E81C2B1E12CC3D28])
[ clap::parser::parser] Parser::get_matches_with: Begin parsing 'RawOsStr("/path/to/foo")' ([47, 112, 97, 116, 104, 47, 116, 111, 47, 102, 111, 111])
[ clap::parser::parser] Parser::get_matches_with: Positional counter...1
[ clap::parser::parser] Parser::get_matches_with: Low index multiples...false
[ clap::parser::parser] Parser::split_arg_values; arg=include, val=RawOsStr("/path/to/foo")
[ clap::parser::parser] Parser::split_arg_values; trailing_values=false, DontDelimTrailingVals=false
[ clap::parser::arg_matcher] ArgMatcher::needs_more_vals: o=include, resolved=0, pending=1
[ clap::parser::parser] Parser::resolve_pending: id=[hash: E81C2B1E12CC3D28]
[ clap::parser::parser] Parser::react action=StoreValue, identifier=Some(Long), source=CommandLine
[ clap::parser::parser] Parser::react: cur_idx:=1
[ clap::parser::parser] Parser::remove_overrides: id=[hash: E81C2B1E12CC3D28]
[ clap::parser::arg_matcher] ArgMatcher::start_occurrence_of_arg: id=[hash: E81C2B1E12CC3D28]
[ clap::builder::command] Command::groups_for_arg: id=[hash: E81C2B1E12CC3D28]
[ clap::parser::parser] Parser::push_arg_values: ["/path/to/foo"]
[ clap::parser::parser] Parser::add_single_val_to_arg: cur_idx:=2
[ clap::builder::command] Command::groups_for_arg: id=[hash: E81C2B1E12CC3D28]
[ clap::parser::parser] Parser::add_defaults
[ clap::parser::parser] Parser::add_defaults:iter:help:
[ clap::parser::parser] Parser::add_default_value:iter:help: doesn't have default missing vals
[ clap::parser::parser] Parser::add_default_value: doesn't have conditional defaults
[ clap::parser::parser] Parser::add_default_value:iter:help: doesn't have default vals
[ clap::parser::parser] Parser::add_defaults:iter:include:
[ clap::parser::parser] Parser::add_default_value:iter:include: doesn't have default missing vals
[ clap::parser::parser] Parser::add_default_value: doesn't have conditional defaults
[ clap::parser::parser] Parser::add_default_value:iter:include: doesn't have default vals
[ clap::parser::validator] Validator::validate
[ clap::parser::validator] Validator::validate_conflicts
[ clap::parser::validator] Validator::validate_exclusive
[ clap::parser::validator] Validator::validate_conflicts::iter: id=[hash: E81C2B1E12CC3D28]
[ clap::parser::validator] Conflicts::gather_conflicts: arg=[hash: E81C2B1E12CC3D28]
[ clap::parser::validator] Conflicts::gather_conflicts: conflicts=[]
[ clap::parser::validator] Validator::validate_required: required=ChildGraph([])
[ clap::parser::validator] Validator::gather_requires
[ clap::parser::validator] Validator::gather_requires:iter:[hash: E81C2B1E12CC3D28]
[ clap::parser::validator] Validator::validate_required: is_exclusive_present=false
[ clap::parser::validator] Validator::validate_required_unless
[ clap::parser::validator] Validator::validate_matched_args
[ clap::parser::validator] Validator::validate_matched_args:iter:[hash: E81C2B1E12CC3D28]: vals=Flatten {
inner: FlattenCompat {
iter: Fuse {
iter: Some(
Iter(
[
[
AnyValue {
inner: TypeId {
t: 4040081991788025797,
},
},
],
],
),
),
},
frontiter: None,
backiter: None,
},
}
[ clap::parser::validator] Validator::validate_arg_num_vals
[ clap::parser::validator] Validator::validate_arg_values: arg="include"
[ clap::parser::validator] Validator::validate_arg_num_occurs: "include"=1
[ clap::parser::arg_matcher] ArgMatcher::get_global_values: global_arg_vec=[[hash: 59636393CFFBFE5F]]
thread 'main' panicked at 'Must use `_os` lookups with `Arg::allow_invalid_utf8`', /var/home/walters/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-3.2.1/src/parser/matches/arg_matches.rs:1651:13
stack backtrace:
0: rust_begin_unwind
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:584:5
1: core::panicking::panic_fmt
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/panicking.rs:143:14
2: clap::parser::matches::arg_matches::unwrap_string
3: <clap::parser::matches::arg_matches::Values as core::iter::traits::iterator::Iterator>::next
4: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
5: clap_regression::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 3
- Comments: 15 (7 by maintainers)
Commits related to this issue
- Fix shell completions We had a panic happen on running `py-spy completions bash`, due to https://github.com/clap-rs/clap/issues/3826 . Fixes #520 — committed to benfred/py-spy by benfred 2 years ago
- Fix shell completions (#522) We had a panic happen on running `py-spy completions bash`, due to https://github.com/clap-rs/clap/issues/3826 . Fixes #520 — committed to benfred/py-spy by benfred 2 years ago
I’m just a random unaffiliated developer, but:
@db48x Paragraph 1 of the semver specification says:
The clap 3.0 documentation stated that your code was using the API incorrectly. Those docs are sufficient to meet semver’s requirements, but as a bonus, clap also provided an automatic test to detect that case, which you say you didn’t run. Once you violated the API semantics, clap had no obligation to keep your code working, even if it appeared to work for a while. Even if it wasn’t obvious at the time, this was your bug.
I think folks here understand how painful this fix has been for you, and how surprising. But speaking as an open-source maintainer myself: directing public aggression toward a maintainer just pushes them one step closer to burnout. Please don’t do it. Don’t do it even if you’re convinced they’re wrong. Providing free, high-quality, widely used software to the public is hard. Whenever a maintainer decides their life will be better if they abandon their software and go do something that won’t get them shouted at by random people on the Internet, we all lose out.
I’m sorry this happened to you, and I hope next weekend brings fewer fires. ❤️
As was mentioned before, we generally report development-time errors through debug asserts and a situation where a debug assert would fire but isn’t in a release build is undefined behavior in clap.
Specifically in this case, we called out the need for
allow_invalid_utf8(true)
forvalue_of_os
in the breaking changes in clap 3.0 and called out the need to test for the debug assert in our migration guide from clap 2 to clap 3. We made these changes to more immediately help with problems and with keeping the design open for what we did in clap 3.2.It’d be possible to make the old behavior still work though it isn’t a priority for me. I would be willing to review a PR for it. Note that
value_of
functions will be going away in clap 4.0.