swift-testing: Introduce conditional trait that support "all must be met" and an "any must be met" to enable the test

Motivation

As a test case author, I want to explicitly enable, or disable, a test when:

  • all conditional traits evaluate to true
  • any conditional training evaluate to true

I always prefer explicit vs implicit. Take the following example

@Test(
   .enabled(if: conditionA),
   .enabled(if: conditionB),
)
func myTest() {
   // test implementation
}

It is unclear by ready the code whether the various conditions are || (OR-ed), or whether they are && (AND-ed). A developer must be aware of Swift Testing behaviour to know when the test will be enabled or disabled.

Proposed solution

One option is to introduce a .any and .all traits, that would accept a list of traits, or any number of traits.

Consider the following, which is more explicit. In the first, conditionA && conditionB must be true to enable the test, while the conditionA || conditionB must be true to enable the second test

@Test(
    .all(
        .enabled(if: conditionA),
        .enabled(if: conditionB),
    )
)
func myTestBothConditionsMustBeTrue() {
    // test implementation
}

@Test(
    .any(
        .enabled(if: conditionA),
        .enabled(if: conditionB),
    )
)
func myTestAnyConditionMustBeTrue() {
    // test implementation
}

The .any and .all become especially useful when custom condition traits are added, where using the single .enabled(if: condition) is more challenging.

extension Trait where Self == Testing.ConditionTrait {
    public static var enableOnlyOnMacOS: Self {
        enabled(if: isMacOS(), "Test is only supported on MacOS")
    }

    public static var enableIfRealSigningIdentityTestEnabled: Self {
        enabled(if: isRealSigningIdentityTestEnabled(), "Test only enabled during real signing indentity")
    }

    public static func enableIfEnvVarSet(_ envVarName: String) -> Self {
        let envKey = EnvironmentKey(envVarName)
        return enabled(
            if: isEnvironmentVariableSet(envKey),
            "'\(envVarName)' env var is not set"
        )
    }
}

@Test (
    .all(
        .enableOnlyOnMacOS,
        .enableIfRealSigningIdentityTestEnabled,
        .enableIfEnvVarSet("MY_ENV_VARIABLE"),
    )
)
funct myTest() {
}

Alternatives considered

I’m open to other solution

Additional information

No response

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 28 (23 by maintainers)

Commits related to this issue

Most upvoted comments

If you’re planning to start a proposal in the near future, I will be on leave and probably won’t be able to help much with it. @stmontgomery would you be able to advise @hrayatnia here?

My thinking here was that we would implement || and && operators that would compose multiple instances of ConditionTrait into compound ones (with additional implementation details inside the type to support compound operations without losing track of which specific trait caused a test to be skipped.)

Then you’d be able to write something like:

@Test(.enabled(if: abc123) || .enabled(if: def456))

What makes this somewhat more complicated (regardless of how we spell the operations) is that .disabled(if: x), which is effectively identical to .enabled(if: !x), may not clearly compose. For instance, .disabled(if: x) || .disabled(if: y) equates to .enabled(if: !x) || .enabled(if: !y) whereas you probably meant it to be .enabled(if: !x) && .enabled(if: !y).

To solve that, we may need to teach the operator implementations to treat .enabled and .disabled differently, which is feasible but adds more complexity to the overall implementation.

@bkhouri that would be like:

@Test(
        .skipSwiftCISelfHosted(
            "These packages don't use the latest runtime library, which doesn't work with self-hosted builds."
        ) &&
        .requireUnrestrictedNetworkAccess("Test requires access to https://github.com/") &&
        .skipHostOS(.windows, "Issue #8409 - random.swift:34:8: error: unsupported platform")
)
func testExamplePackageDealer() throws { }

or

@Test(
        .skipSwiftCISelfHosted(
            "These packages don't use the latest runtime library, which doesn't work with self-hosted builds."
        ) ||
        .requireUnrestrictedNetworkAccess("Test requires access to https://github.com/") ||
        .skipHostOS(.windows, "Issue #8409 - random.swift:34:8: error: unsupported platform")
)
func testExamplePackageDealer() throws { }

Right now, I’m working on the behind-the-scenes logic for subconditions and other parts. As soon as that one is finished, I will provide more examples; however, meanwhile, there are some examples of all suggested cases here (sorry, I have the habit of making code collapsible when text is too large; in example segments you find some examples).

There are a lot of ideas being explored here, several which would result in a pretty sophisticated but also potentially complicated API. I’d like to come back to an earlier idea suggested above, about overloading the standard && and || operators. Was there a reason that wasn’t feasible, or wouldn’t solve the need? The comment indicated that for it to be possible, the underlying implementation and representation (e.g. in ConditionTrait.Kind) would need to be modified, but that part seems doable.

As a general guideline, we have tried to make Swift Testing’s APIs easy to learn and understand by even a casual user, and to do that we have leaned on existing Swift APIs, syntax, and concepts wherever possible. I would say we have a strong preference towards using standard Swift language features instead of using custom DSLs. The trait system and built-in traits are probably the area where we adhere to that principle the least strongly, but still, whenever there is an opportunity to use existing things, I think we should try hard to go that route.

But it’s not explicit. I would like to have an equivalent of, where it’s explicit, and where I can select .all, or .any, or whichever the API will allow.

It is a non-goal of the testing library to add multiple ways to accomplish the exact same task. Both AND and OR can already be expressed, trivially, with combinations of .enabled(if:) and .disabled(if:).

What we are missing right now is a way to express AND/OR operations on ConditionTrait instances in a way that can be a) shuffled off into a helper function and b) reports which specific subcondition caused a test to be skipped.

Any changes here should be designed with that specific goal in mind. I don’t think the Testing Workgroup would approve a proposal adding .any for the sake of adding it, without it providing any additional functionality.

@bkhouri That’s already supported and functional though.

Sorry for not being accurate in my previous comment. I didn’t mean that people would introduce new APIs to swift-testing. What I meant was that if swift-testing were to make ConditionTraits.Kind public (which, as it stands, doesn’t seem to be the plan), people outside the library might get creative and try to introduce new methods. The .pending method was purely hypothetical—just an example to illustrate the possibility of using other unary operators for their conditions.

However, since it seems it’s not the case then no need to even entertain the hypotheticals.

The only case can be problematic regardless of infix (custom operator) or prefix(DSL approach) situation is if ConditionTraits.Kind and init become public, and people introduce new methods like .pending (bitwise not/state) and beyond.

As far as I know, we have no plans to make those symbols public, and .pending() is not a proposed API (I’m honestly not sure what it would do?)

Have we considered the how custom conditional traits can be used in the chosen solution?

In general, we wouldn’t advise folks to create their own types that do what ConditionTrait does. If you want to make a fancy label for your bespoke condition, you can write something like:

extension Trait where Self == ConditionTrait {
  static func .skipInJenkins(sourceLocation: SourceLocation = #_sourceLocation) -> Self {
    .disabled(if: inJenkins(), "Running in Jenkins", sourceLocation: sourceLocation)
  }
}

If there’s some functionality that would require a completely custom trait type, I’m not sure what it is.

The implemented solution should consider readability of the conditions, in addition for “swift formatting”.

For example, how do we want to treat a “more complex” condition, say,

@Test(
    .any(
        .any(conditionA, conditionB, conditionC),
        .all(conditionD, conditionE),
        ...
    )
)

Granted, a “condition” like this might mean “the test is too complicated and needs to be broken down”, but the “readability” aspect is still a thing.

Whatever the solution, it would be great if we can an include each traits on it’s own line as the following (which may be syntactically-incorrect) may be less 'readable" than the second

e.g.:

@Test(
   .disabled("This is a comment"") {
       // some code that check environment variable is set
   } && .disabled("this is another comment") {
      // this is another code
   }
)




@Test (
   .all(
      .disabled("This is a comment"") {
          // some code that check environment variable is set
      },
      .disabled("this is another comment") {
         // this is another code
      },
   ) 
)

And whatever the solution, it would be ideal to explicitly indicate every condition that was not met.