clap: Regression from 3.1 -> 3.2 in `values_of` (i think)

Please complete the following tasks

Rust Version

rustc 1.60.0 (7737e0b5c 2022-04-04)

Clap Version

3.2.1

Minimal reproducible code

use clap::{Arg, Command};

const INCLUDE: &str = "include";

fn arg_include() -> Arg<'static> {
    Arg::new(INCLUDE)
        .long(INCLUDE)
        .short('i')
        .takes_value(true)
        .multiple_occurrences(true)
        .allow_invalid_utf8(true)
}

fn app() -> Command<'static> {
    Command::new("test").arg(arg_include())
}

#[derive(Copy, Clone, PartialEq, Debug)]
pub enum IncludeKind {
    Quoted,
    Bracketed,
}

#[derive(Clone, PartialEq, Debug)]
pub struct Include {
    pub path: String,
    pub kind: IncludeKind,
}

#[derive(Debug)]
struct Opt {
    include: Vec<Include>,
}

fn parse() -> Opt {
    let matches = app().get_matches_from(&["test", "--include", "/path/to/foo"]);
    
    let include = matches
        .values_of(INCLUDE)
        .unwrap_or_default()
        .map(|include| {
            if include.starts_with('<') && include.ends_with('>') {
                Include {
                    path: include[1..include.len() - 1].to_owned(),
                    kind: IncludeKind::Bracketed,
                }
            } else {
                Include {
                    path: include.to_owned(),
                    kind: IncludeKind::Quoted,
                }
            }
        })
        .collect();
    Opt {
        include
    }
}

fn main() {
    dbg!(parse());
    
}

Steps to reproduce the bug with the above code

Using clap = "3.2" with cargo run --release, I get:

$ cargo run --release
thread 'main' panicked at 'Must use `_os` lookups with `Arg::allow_invalid_utf8`', /var/home/walters/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-3.2.1/src/parser/matches/arg_matches.rs:1651:13
stack backtrace:
   0: rust_begin_unwind
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/panicking.rs:143:14
   2: clap::parser::matches::arg_matches::unwrap_string
   3: <clap::parser::matches::arg_matches::OsValues as core::iter::traits::iterator::Iterator>::next
   4: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
   5: clap_regression::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Yet specifying clap = ">= 3.1, < 3.2" (resolving to v3.1.18), the program succeeds.

Actual Behaviour

Panic

Expected Behaviour

Don’t panic

Additional Context

This was seen in https://github.com/dtolnay/cxx/pull/1057

Debug Output

    Finished release [optimized] target(s) in 12.03s
     Running `target/release/clap-regression`
[      clap::builder::command]  Command::_do_parse
[      clap::builder::command]  Command::_build: name="test"
[      clap::builder::command]  Command::_propagate:test
[      clap::builder::command]  Command::_check_help_and_version: test
[      clap::builder::command]  Command::_check_help_and_version: Removing generated version
[      clap::builder::command]  Command::_propagate_global_args:test
[      clap::builder::command]  Command::_derive_display_order:test
[        clap::parser::parser]  Parser::get_matches_with
[        clap::parser::parser]  Parser::get_matches_with: Begin parsing 'RawOsStr("--include")' ([45, 45, 105, 110, 99, 108, 117, 100, 101])
[        clap::parser::parser]  Parser::get_matches_with: Positional counter...1
[        clap::parser::parser]  Parser::get_matches_with: Low index multiples...false
[        clap::parser::parser]  Parser::possible_subcommand: arg=Ok("--include")
[        clap::parser::parser]  Parser::get_matches_with: sc=None
[        clap::parser::parser]  Parser::parse_long_arg
[        clap::parser::parser]  Parser::parse_long_arg: Does it contain '='...
[        clap::parser::parser]  Parser::parse_long_arg: Found valid arg or flag '--include <include>'
[        clap::parser::parser]  Parser::parse_long_arg("include"): Found an arg with value 'None'
[        clap::parser::parser]  Parser::parse_opt_value; arg=include, val=None, has_eq=false
[        clap::parser::parser]  Parser::parse_opt_value; arg.settings=ArgFlags(MULTIPLE_OCC | TAKES_VAL | UTF8_NONE)
[        clap::parser::parser]  Parser::parse_opt_value; Checking for val...
[        clap::parser::parser]  Parser::parse_opt_value: More arg vals required...
[        clap::parser::parser]  Parser::get_matches_with: After parse_long_arg Opt([hash: E81C2B1E12CC3D28])
[        clap::parser::parser]  Parser::get_matches_with: Begin parsing 'RawOsStr("/path/to/foo")' ([47, 112, 97, 116, 104, 47, 116, 111, 47, 102, 111, 111])
[        clap::parser::parser]  Parser::get_matches_with: Positional counter...1
[        clap::parser::parser]  Parser::get_matches_with: Low index multiples...false
[        clap::parser::parser]  Parser::split_arg_values; arg=include, val=RawOsStr("/path/to/foo")
[        clap::parser::parser]  Parser::split_arg_values; trailing_values=false, DontDelimTrailingVals=false
[   clap::parser::arg_matcher]  ArgMatcher::needs_more_vals: o=include, resolved=0, pending=1
[        clap::parser::parser]  Parser::resolve_pending: id=[hash: E81C2B1E12CC3D28]
[        clap::parser::parser]  Parser::react action=StoreValue, identifier=Some(Long), source=CommandLine
[        clap::parser::parser]  Parser::react: cur_idx:=1
[        clap::parser::parser]  Parser::remove_overrides: id=[hash: E81C2B1E12CC3D28]
[   clap::parser::arg_matcher]  ArgMatcher::start_occurrence_of_arg: id=[hash: E81C2B1E12CC3D28]
[      clap::builder::command]  Command::groups_for_arg: id=[hash: E81C2B1E12CC3D28]
[        clap::parser::parser]  Parser::push_arg_values: ["/path/to/foo"]
[        clap::parser::parser]  Parser::add_single_val_to_arg: cur_idx:=2
[      clap::builder::command]  Command::groups_for_arg: id=[hash: E81C2B1E12CC3D28]
[        clap::parser::parser]  Parser::add_defaults
[        clap::parser::parser]  Parser::add_defaults:iter:help:
[        clap::parser::parser]  Parser::add_default_value:iter:help: doesn't have default missing vals
[        clap::parser::parser]  Parser::add_default_value: doesn't have conditional defaults
[        clap::parser::parser]  Parser::add_default_value:iter:help: doesn't have default vals
[        clap::parser::parser]  Parser::add_defaults:iter:include:
[        clap::parser::parser]  Parser::add_default_value:iter:include: doesn't have default missing vals
[        clap::parser::parser]  Parser::add_default_value: doesn't have conditional defaults
[        clap::parser::parser]  Parser::add_default_value:iter:include: doesn't have default vals
[     clap::parser::validator]  Validator::validate
[     clap::parser::validator]  Validator::validate_conflicts
[     clap::parser::validator]  Validator::validate_exclusive
[     clap::parser::validator]  Validator::validate_conflicts::iter: id=[hash: E81C2B1E12CC3D28]
[     clap::parser::validator]  Conflicts::gather_conflicts: arg=[hash: E81C2B1E12CC3D28]
[     clap::parser::validator]  Conflicts::gather_conflicts: conflicts=[]
[     clap::parser::validator]  Validator::validate_required: required=ChildGraph([])
[     clap::parser::validator]  Validator::gather_requires
[     clap::parser::validator]  Validator::gather_requires:iter:[hash: E81C2B1E12CC3D28]
[     clap::parser::validator]  Validator::validate_required: is_exclusive_present=false
[     clap::parser::validator]  Validator::validate_required_unless
[     clap::parser::validator]  Validator::validate_matched_args
[     clap::parser::validator]  Validator::validate_matched_args:iter:[hash: E81C2B1E12CC3D28]: vals=Flatten {
    inner: FlattenCompat {
        iter: Fuse {
            iter: Some(
                Iter(
                    [
                        [
                            AnyValue {
                                inner: TypeId {
                                    t: 4040081991788025797,
                                },
                            },
                        ],
                    ],
                ),
            ),
        },
        frontiter: None,
        backiter: None,
    },
}
[     clap::parser::validator]  Validator::validate_arg_num_vals
[     clap::parser::validator]  Validator::validate_arg_values: arg="include"
[     clap::parser::validator]  Validator::validate_arg_num_occurs: "include"=1
[   clap::parser::arg_matcher]  ArgMatcher::get_global_values: global_arg_vec=[[hash: 59636393CFFBFE5F]]
thread 'main' panicked at 'Must use `_os` lookups with `Arg::allow_invalid_utf8`', /var/home/walters/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-3.2.1/src/parser/matches/arg_matches.rs:1651:13
stack backtrace:
   0: rust_begin_unwind
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/panicking.rs:143:14
   2: clap::parser::matches::arg_matches::unwrap_string
   3: <clap::parser::matches::arg_matches::Values as core::iter::traits::iterator::Iterator>::next
   4: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
   5: clap_regression::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 3
  • Comments: 15 (7 by maintainers)

Commits related to this issue

Most upvoted comments

I’m just a random unaffiliated developer, but:

@db48x Paragraph 1 of the semver specification says:

Software using Semantic Versioning MUST declare a public API. This API could be declared in the code itself or exist strictly in documentation. However it is done, it SHOULD be precise and comprehensive.

The clap 3.0 documentation stated that your code was using the API incorrectly. Those docs are sufficient to meet semver’s requirements, but as a bonus, clap also provided an automatic test to detect that case, which you say you didn’t run. Once you violated the API semantics, clap had no obligation to keep your code working, even if it appeared to work for a while. Even if it wasn’t obvious at the time, this was your bug.

I think folks here understand how painful this fix has been for you, and how surprising. But speaking as an open-source maintainer myself: directing public aggression toward a maintainer just pushes them one step closer to burnout. Please don’t do it. Don’t do it even if you’re convinced they’re wrong. Providing free, high-quality, widely used software to the public is hard. Whenever a maintainer decides their life will be better if they abandon their software and go do something that won’t get them shouted at by random people on the Internet, we all lose out.

I’m sorry this happened to you, and I hope next weekend brings fewer fires. ❤️

As was mentioned before, we generally report development-time errors through debug asserts and a situation where a debug assert would fire but isn’t in a release build is undefined behavior in clap.

Specifically in this case, we called out the need for allow_invalid_utf8(true) for value_of_os in the breaking changes in clap 3.0 and called out the need to test for the debug assert in our migration guide from clap 2 to clap 3. We made these changes to more immediately help with problems and with keeping the design open for what we did in clap 3.2.

It’d be possible to make the old behavior still work though it isn’t a priority for me. I would be willing to review a PR for it. Note that value_of functions will be going away in clap 4.0.