Quick: Test filtering does not work with Quick

@jessesquires @phatblat I found out what the issue is, it may only happen on Linux, I don’t have access to a Mac computer to test on macOS.

Basically XCTest takes one argument for the list of tests to execute (which is what SwiftPM --filter uses when invoking your test binary).

The thing is, it’s not one argument = one test case, it’s one argument for all selected test cases, separated by a comma 🙃

So when you have a Quick test that’s called Quick.QuickSpec/my thing, when doing this, also does that, XCTest sees "run 3 tests: “Quick.QuickSpec/my thing”, “when doing this”, “also does that”.

The incriminating code is here: https://github.com/apple/swift-corelibs-xctest/blob/swift-5.7-DEVELOPMENT-SNAPSHOT-2022-06-26-a/Sources/XCTest/Private/ArgumentParser.swift#L69

When removing the split call it fixes the issue for me.

I’m not too sure where to go from there, I guess the real fix would be in XCTest to add quotes or whatever, but it’s useless if SwiftPM does not also add quotes when using --filter… there is also the macOS version of XCTest to consider that I cannot make a PR to.

What do you recommend? Should I make an XCTest issue? I guess it’s normal that commas are not handled properly since test functions are supposed to come from actual function names, but since XCTest allows making them dynamically (through allTestCases) I guess it should be able to handle any string.

In case the answer is “commas are not legal in test cases names”, should we consider a workaround in Quick as well?

If it adds anything, the issue is not stale or outdated: I confirm that filtering still doesn’t work on Swift 5.6 on Linux (I don’t know the associated XCTest version) using swift test --filter. It shows the exact same behavior as outlined in the previous comments.

I played around with this using swift 5.2.2 and the full test names can be found using the --list-tests option.

↪ swift test --list-tests
PinKitTests.PostSpec/post__has_a_description
PinKitTests.PostSpec/post__has_an_extended
PinKitTests.PostSpec/post__has_a_hash
PinKitTests.PostSpec/post__has_a_description_2
PinKitTests.PostSpec/post__has_a_meta
PinKitTests.PostSpec/post__is_shared
PinKitTests.PostSpec/post__has_tags
PinKitTests.PostSpec/post__has_a_time
PinKitTests.PostSpec/post__has_a_toread_flag
PinKitTests.PostsAddSpec/posts_add__has_a_result_code
PinKitTests.PostsAddSpec/posts_add__parsing__can_be_parsed_from_json
PinKitTests.PostsDeleteSpec/posts_delete__has_a_result_code
PinKitTests.PostsDeleteSpec/posts_delete__parsing__can_be_parsed_from_json
phatblat at octodec in ~/dev/swift/PinKit on master [✓]

Providing the full test name (or just the ) to the --filter option seems to do less work because there is a lot less output than when running without the filter. However it doesn’t appear to actually run any tests due to the “Executed 0 tests” output.

↪ swift test --filter PinKitTests.PostsDeleteSpec/posts_delete__parsing__can_be_parsed_from_json
Test Suite 'Selected tests' started at 2020-06-09 13:28:41.011
Test Suite 'PinKitPackageTests.xctest' started at 2020-06-09 13:28:41.012
Test Suite 'PinKitPackageTests.xctest' passed at 2020-06-09 13:28:41.012.
	 Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.000) seconds
Test Suite 'Selected tests' passed at 2020-06-09 13:28:41.012.
	 Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.000) seconds

xcodebuild (v11.5) has the same issue when running a subset of tests. It runs a test suite called ‘Selected tests’ and the summary shows it “Executed 0 tests”.

↪ xcodebuild test -workspace PinKit.xcworkspace -scheme PinKit-Package -only-testing:PinKitTests/PostsDeleteSpec/posts_delete__parsing__can_be_parsed_from_json
Command line invocation:
    /Applications/Xcode-11.5.app/Contents/Developer/usr/bin/xcodebuild test -workspace PinKit.xcworkspace -scheme PinKit-Package "-only-testing:PinKitTests/PostsDeleteSpec/posts_delete__parsing__can_be_parsed_from_json"

User defaults from command line:
    IDETestRunOnlyIdentifiers = (
    "PinKitTests/PostsDeleteSpec/posts_delete__parsing__can_be_parsed_from_json"
)

...
Test Suite 'Selected tests' started at 2020-06-09 13:38:47.655
Test Suite 'PinKitTests.xctest' started at 2020-06-09 13:38:47.656
Test Suite 'PinKitTests.xctest' passed at 2020-06-09 13:38:47.656.
	 Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.000) seconds
Test Suite 'Selected tests' passed at 2020-06-09 13:38:47.656.
	 Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
...

Test session results, code coverage, and logs:
	/Volumes/DerivedData/PinKit-akgpefidgquoryhfyxurznnvcfrr/Logs/Test/Test-PinKit-Package-2020.06.09_13-38-46--0600.xcresult

** TEST SUCCEEDED ** [1.737 sec]

Something in the swift test command is able to see the individual Quick tests. I wonder if that mechanism can be used to help expose the filtered set of “Selected tests” from Quick to the XCTest runtime.

Bug filed with the open source XCTest: https://github.com/apple/swift-corelibs-xctest/issues/458

I also filed FB12489187 for the closed-source version, with contents:

Hello! I maintain the Quick testing framework for Swift and Objective-C. Amongst other things, Quick allows you to use any arbitrary strings as the name of a test. For example, the following code would generate a test named, using xcodebuild test’s filtering syntax, `“SomeSpec”/“can even include /s”:

class SomeSpec: QuickSpec {
    override class func spec() {
        it("can even include /s") {
            // the actual test to run
        }
    }
}

In a recent WWDC lab with XCTest engineers, one of them mentioned that allowing slashes in the test name could potentially cause issues with filtering tests. While this isn’t an issue when selecting tests to run using test gems, it is an issue if use the -only-testing or -skip-testing options in XCTest. That is, when you try to filter test names that contain slashes, xcodebuild test -only-testing will act as if no test filtering were specified.

This was tested using Xcode 15, beta 2 and Xcode 14.3.1 as in the Mac App Store.

Thanks!

I’ll leave this open until it’s fixed.

Hey there! 👋🏼

I’m a new maintainer for this project and I’m trying to get the next release out ASAP and also clean up old issues and old PRs. I’m closing all issues that no longer seem relevant or are very old / stale.

This does not mean this issue is necessarily being rejected. If you are still interested in contributing the changes specified in this issue, please comment to request it be reopened.

We appreciate you opening this issue and acknowledge the time and effort you put in to contribute! You’re awesome. 💯 However, we are all volunteers here with limited capacity working for free. Unfortunately, that means we must close out stale issues and PRs to move the project forward in a reasonable way.

We apologize for the inconvenience and appreciate your understanding! 😎 ✌🏼

Tested with XCode 14.2, -only-testing, -skip-testing work at suite level, not at individual test case level.

Same is true in the XCode UI test navigator.

I can’t install betas on my main work partition, but I’m going to setup a partition with macOS and the beta Xcode to test soon.