kotest: Add semantics for "expected failures"

pending = true could easily:

• imply a tag (if we want to do that?)
• add PENDING to output (and in Kotest 6 be highlighted properly)
• invert the result

That makes sense:

test("foo").config(pending = true) { 
 //
}

And this is inverted - if it passes it fails and vice versa.

but also offering an alternative use case for this desire.

That’s actually exactly the use-case I had in mind initially.

Partially bumping the thread, but also offering an alternative use case for this desire.

Recently I encountered a bug in a github repo, and I reported the issue in a way that allowed one of their devs to immediately write a unit test, and confirm that my reproduction was good with the latest version of the code. They flagged that test as expecting failure, and merged it into the codebase.

As a user of the framework it felt like a really good user experience to see that my bug was validated, and knew that in future it would not be forgotten about because there was a unit test in there telling them that something was broken and it would never get stale.

edit: here’s the issue in question, see how there was two separate PRs, the first one validated my repro and the second actually fixed it a few weeks later https://github.com/microsoft/playwright/issues/17824

For what it’s worth, my team is hitting the need for this capability as well.

The scenario:

• we have a known code defect
• we don’t have time to actually fix it now
• we can create a test case indicating the failure

…if we had the equivalent of pendingUntilFixed (rspec also does a good job of this) we can note the item to fix in the code, and be reminded of all such pending tech debt items every time we run the test suite.

It is also helpful in that if some other code causes the test to unexpectedly pass, I am notified (by a test suite failure), so I can go investigate and see if something unexpected happened, if it fixed the issue, etc.

Otherwise if I am not notified of the test passing immediately as a test failure, someone later might run the test, see it pass and assume it was properly fixed (which it may not have been).

Hmm. The whole point of “expected to fail” tests is that they do get run, but their failure does not count towards the fail count. We do not want to skip these tests, thus I’m not sure how a custom reason why a test was skipped helps here.