test-case: Result::Err is not interpreted as a failure

In Rust 2018 test can return a Result type, with Result::Err being interpreted as a failure. However with parameterised tests returned errors are treated as passing. While this can be fixed by specifying an expected value, it is unintuitive.

Example:

fn should_equal_4(num: i32) -> Result<(), ()> {
    if num == 4 {
        Ok(())
    } else {
        Err(())
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use test_case::test_case;

    #[test_case(4; "Passing")]
    #[test_case(5; "Failing")]
    fn with_params(num: i32) -> Result<(), ()> {
        should_equal_4(num)?;
        Ok(())
    }

    #[test]
    fn is_4() -> Result<(), ()> {
        should_equal_4(4)?;
        Ok(())
    }

    #[test]
    fn is_5() -> Result<(), ()> {
        should_equal_4(5)?;
        Ok(())
    }
}

Expected: Both tests::with_params::failing() and is_5() fail.

Actual: Only is_5() fails.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 19 (8 by maintainers)

Commits related to this issue

test-case does not handle tests returning Result correctly; see https://github.com/frondeus/test-case/issues/50 — committed to tarka/xcp by tarka 4 years ago
test-case does not handle tests returning Result correctly; see https://github.com/frondeus/test-case/issues/50 — committed to tarka/xcp by tarka 4 years ago
Tweak test_case handling (#88) * handle returning `Result` from test_case when `=>` is absent #50 * leave tested item in scope it was defined in #77 — committed to frondeus/test-case by luke-biel 2 years ago

Most upvoted comments

1.0.x support is old branch with some incompatible changes. @frondeus we probably should create similar branch for 1.2.1 from 1.2.1 tag since master is already at 2.0.0-rc.

I guess we need to come up with some proper branching strat.

luke-biel on Feb 3, 2022

I recommend forking the repository. It is the essence of open source, after all. You don’t have to worry that the issue has been open for so long by forking the repository. Of course, it requires an effort to patch the bug.

And while we made this library open-source and available for everyone, we (usually) are still driven by what we need in our personal and professional projects.

From my experience, I didn’t fix this bug simply because it had no severity. I rarely replaced #[test] with #[test_case], and when I did, I used unwrap() instead of the Result.

So while we value the input from other users, we have a limited amount of time for OSS projects.

That’s why I didn’t make any promises about this fix. If someone else thinks this is a critical issue, I’m more than happy to guide how to fix it and review and merge the PR.

frondeus on Jan 28, 2022

I just ran into this on accident, I wasn’t expecting test_case to not function like the Rust standard #[test] in this situation.

archoversight on Nov 2, 2021