fastlane: Test Report is Empty when Tests Execute in Parallel
New Issue Checklist
- Updated fastlane to the latest version
- I read the Contribution Guidelines
- I read docs.fastlane.tools
- I searched for existing GitHub issues
Issue Description
When enabled “Execute in parallel” for Xcode Unit-Tests, the generated report.html is empty while the original .xcresult file is not empty.
Command executed
lane :run_unit_tests do
run_tests(
devices: ["iPhone 14 (16.0)"],
prelaunch_simulator: true,
reset_simulator: true,
code_coverage: true,
result_bundle: true,
output_directory: "output/test_output/unit_tests",
output_types: "html",
scheme: "FastlaneExample",
clean: true,
parallel_testing: true,
concurrent_workers: 2
)
end
Complete output when running fastlane, including the stack trace and command used
[11:40:23]:
[11:40:23]: $ bundle exec fastlane
[11:40:23]:
[11:40:23]: Get started using a Gemfile for fastlane https://docs.fastlane.tools/getting-started/ios/setup/#use-a-gemfile
[11:40:24]: ------------------------------
[11:40:24]: --- Step: default_platform ---
[11:40:24]: ------------------------------
[11:40:24]: Welcome to fastlane! Here's what your app is set up to do:
+--------+--------------------+------------------------------+
| Available lanes to run |
+--------+--------------------+------------------------------+
| Number | Lane Name | Description |
+--------+--------------------+------------------------------+
| 1 | ios run_unit_tests | Run Unit Tests |
| 0 | cancel | No selection, exit fastlane! |
+--------+--------------------+------------------------------+
[11:40:24]: Which number would you like to run?
1
[11:40:24]: Running lane `ios run_unit_tests`. Next time you can do this by directly typing `fastlane ios run_unit_tests` 🚀.
[11:40:24]: Driving the lane 'ios run_unit_tests' 🚀
[11:40:24]: -----------------------
[11:40:24]: --- Step: run_tests ---
[11:40:24]: -----------------------
[11:40:25]: Resolving Swift Package Manager dependencies...
[11:40:25]: $ xcodebuild -resolvePackageDependencies -scheme FastlaneExample -project ./FastlaneExample.xcodeproj
[11:40:25]: ▸ Command line invocation:
[11:40:25]: ▸ /Applications/Xcode-14.0.1.app/Contents/Developer/usr/bin/xcodebuild -resolvePackageDependencies -scheme FastlaneExample -project ./FastlaneExample.xcodeproj
[11:40:25]: ▸ User defaults from command line:
[11:40:25]: ▸ IDEPackageSupportUseBuiltinSCM = YES
[11:40:25]: ▸ --- xcodebuild: WARNING: Using the first of multiple matching destinations:
[11:40:25]: ▸ { platform:macOS, arch:arm64, id:00006001-001409C20245801E }
[11:40:25]: ▸ { platform:macOS, arch:x86_64, id:00006001-001409C20245801E }
[11:40:25]: ▸ { platform:iOS, id:dvtdevice-DVTiPhonePlaceholder-iphoneos:placeholder, name:Any iOS Device }
[11:40:25]: ▸ { platform:iOS Simulator, id:dvtdevice-DVTiOSDeviceSimulatorPlaceholder-iphonesimulator:placeholder, name:Any iOS Simulator Device }
[11:40:25]: ▸ { platform:macOS, name:Any Mac }
[11:40:25]: ▸ { platform:iOS Simulator, id:44AD13FB-1576-4395-AD60-2F0F1FF4A771, OS:16.0, name:iPad (9th generation) }
[11:40:25]: ▸ { platform:iOS Simulator, id:CD9D5AF5-D0F3-4DC1-9742-EC8D59ADB05A, OS:16.0, name:iPad Air (5th generation) }
[11:40:25]: ▸ { platform:iOS Simulator, id:50A7699F-A5D2-4AE2-B10C-089F8A4D549D, OS:16.0, name:iPad Pro (11-inch) (3rd generation) }
[11:40:25]: ▸ { platform:iOS Simulator, id:C701A356-B4CC-48F1-ADAF-C53140F0A80D, OS:16.0, name:iPad Pro (12.9-inch) (5th generation) }
[11:40:25]: ▸ { platform:iOS Simulator, id:F257B2DB-0CB1-4C62-9779-AAA128AF70F9, OS:16.0, name:iPad mini (6th generation) }
[11:40:25]: ▸ { platform:iOS Simulator, id:74F8B615-3882-40A5-89CC-026139BB3D39, OS:16.0, name:iPhone 14 }
[11:40:25]: ▸ { platform:iOS Simulator, id:75FA514D-4C19-4075-A499-0A21A2C3A3CF, OS:16.0, name:iPhone 14 Plus }
[11:40:25]: ▸ { platform:iOS Simulator, id:297DF8AC-AF86-48AA-91E0-BF5D90577649, OS:16.0, name:iPhone 14 Pro }
[11:40:25]: ▸ { platform:iOS Simulator, id:3699AFCB-39FC-4444-9292-E377AD7E718F, OS:16.0, name:iPhone 14 Pro Max }
[11:40:25]: ▸ { platform:iOS Simulator, id:82840798-576E-436C-9CDD-2592F5432FB5, OS:16.0, name:iPhone SE (3rd generation) }
[11:40:25]: ▸ resolved source packages:
[11:40:25]: $ xcodebuild -showBuildSettings -scheme FastlaneExample -project ./FastlaneExample.xcodeproj
--- xcodebuild: WARNING: Using the first of multiple matching destinations:
{ platform:macOS, arch:arm64, id:00006001-001409C20245801E }
{ platform:macOS, arch:x86_64, id:00006001-001409C20245801E }
{ platform:iOS, id:dvtdevice-DVTiPhonePlaceholder-iphoneos:placeholder, name:Any iOS Device }
{ platform:iOS Simulator, id:dvtdevice-DVTiOSDeviceSimulatorPlaceholder-iphonesimulator:placeholder, name:Any iOS Simulator Device }
{ platform:macOS, name:Any Mac }
{ platform:iOS Simulator, id:44AD13FB-1576-4395-AD60-2F0F1FF4A771, OS:16.0, name:iPad (9th generation) }
{ platform:iOS Simulator, id:CD9D5AF5-D0F3-4DC1-9742-EC8D59ADB05A, OS:16.0, name:iPad Air (5th generation) }
{ platform:iOS Simulator, id:50A7699F-A5D2-4AE2-B10C-089F8A4D549D, OS:16.0, name:iPad Pro (11-inch) (3rd generation) }
{ platform:iOS Simulator, id:C701A356-B4CC-48F1-ADAF-C53140F0A80D, OS:16.0, name:iPad Pro (12.9-inch) (5th generation) }
{ platform:iOS Simulator, id:F257B2DB-0CB1-4C62-9779-AAA128AF70F9, OS:16.0, name:iPad mini (6th generation) }
{ platform:iOS Simulator, id:74F8B615-3882-40A5-89CC-026139BB3D39, OS:16.0, name:iPhone 14 }
{ platform:iOS Simulator, id:75FA514D-4C19-4075-A499-0A21A2C3A3CF, OS:16.0, name:iPhone 14 Plus }
{ platform:iOS Simulator, id:297DF8AC-AF86-48AA-91E0-BF5D90577649, OS:16.0, name:iPhone 14 Pro }
{ platform:iOS Simulator, id:3699AFCB-39FC-4444-9292-E377AD7E718F, OS:16.0, name:iPhone 14 Pro Max }
{ platform:iOS Simulator, id:82840798-576E-436C-9CDD-2592F5432FB5, OS:16.0, name:iPhone SE (3rd generation) }
2022-11-01 11:40:26.563 xcodebuild[94652:2551328] Requested but did not find extension point with identifier Xcode.InterfaceBuilderBuildSupport.PlatformDefinition
±-----------------------------------------------±----------------------------------------------------------------------------+
| Summary for scan 2.210.1 |
±-----------------------------------------------±----------------------------------------------------------------------------+
| devices | [“iPhone 14 (16.0)”] |
| prelaunch_simulator | true |
| reset_simulator | true |
| code_coverage | true |
| result_bundle | true |
| output_directory | output/test_output/unit_tests |
| output_types | html |
| scheme | FastlaneExample |
| clean | true |
| parallel_testing | true |
| concurrent_workers | 2 |
| project | ./FastlaneExample.xcodeproj |
| skip_detect_devices | false |
| ensure_devices_found | false |
| force_quit_simulator | false |
| disable_slide_to_type | true |
| reinstall_app | false |
| open_report | false |
| buildlog_path | ~/Library/Logs/scan |
| include_simulator_logs | false |
| xcodebuild_formatter | xcpretty |
| output_remove_retry_attempts | false |
| derived_data_path | /Users/yuantong/Library/Developer/Xcode/DerivedData/FastlaneExample-bpvoeq |
| | jozoypbrcqbpohyhsxcltp |
| should_zip_build_products | false |
| output_xctestrun | false |
| use_clang_report_name | false |
| disable_concurrent_testing | false |
| skip_build | false |
| slack_use_webhook_configured_username_and_icon | false |
| slack_username | fastlane |
| slack_icon_url | https://fastlane.tools/assets/img/fastlane_icon.png |
| skip_slack | false |
| slack_only_on_failure | false |
| xcodebuild_command | env NSUnbufferedIO=YES xcodebuild |
| skip_package_dependencies_resolution | false |
| disable_package_automatic_updates | false |
| use_system_scm | false |
| number_of_retries | 0 |
| fail_build | true |
| xcode_path | /Applications/Xcode-14.0.1.app |
±-----------------------------------------------±----------------------------------------------------------------------------+
[11:40:26]: Resetting iPhone 14
[11:40:27]: Disabling ‘Slide to Type’ iPhone 14
[11:40:27]: $ /usr/libexec/PlistBuddy -c “Add :KeyboardContinuousPathEnabled bool false” /Users/yuantong/Library/Developer/CoreSimulator/Devices/74F8B615-3882-40A5-89CC-026139BB3D39/data/Library/Preferences/com.apple.keyboard.ContinuousPath.plist >/dev/null 2>&1
[11:40:27]: Booting iPhone 14
[11:40:27]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -scheme FastlaneExample -project ./FastlaneExample.xcodeproj -derivedDataPath /Users/yuantong/Library/Developer/Xcode/DerivedData/FastlaneExample-bpvoeqjozoypbrcqbpohyhsxcltp -destination ‘platform=iOS Simulator,id=74F8B615-3882-40A5-89CC-026139BB3D39’ -resultBundlePath ‘output/test_output/unit_tests/FastlaneExample.xcresult’ -parallel-testing-enabled YES -parallel-testing-worker-count 2 -enableCodeCoverage YES clean build test | tee ‘/Users/yuantong/Library/Logs/scan/FastlaneExample-FastlaneExample.log’ | xcpretty --report html --output ‘/Users/yuantong/Developer/FastlaneExample/FastlaneExample/output/test_output/unit_tests/report.html’ --report junit --output ‘/var/folders/sb/4vl7zgtj57gf88jp_rtj4wmc00t6q6/T/junit_report20221101-94642-ogesth’
[11:40:27]: ▸ Loading…
[11:40:28]: ▸ Clean Succeeded
[11:40:29]: ▸ Processing empty-FastlaneExample.plist
[11:40:29]: ▸ Compiling ContentView.swift
[11:40:29]: ▸ Compiling FastlaneExampleApp.swift
[11:40:29]: ▸ Linking FastlaneExample
[11:40:29]: Running Tests: ▸ Touching FastlaneExample.app (in target ‘FastlaneExample’ from project ‘FastlaneExample’)
[11:40:29]: ▸ Build Succeeded
[11:40:29]: ▸ Processing empty-FastlaneExample.plist
[11:40:29]: ▸ Compiling ContentView.swift
[11:40:29]: ▸ Compiling FastlaneExampleApp.swift
[11:40:29]: ▸ Linking FastlaneExample
[11:40:30]: ▸ Processing empty-FastlaneExampleTests.plist
[11:40:30]: ▸ Compiling FastlaneExampleTests2.swift
[11:40:30]: ▸ Compiling FastlaneExampleTests.swift
[11:40:30]: ▸ Linking FastlaneExampleTests
[11:40:30]: Running Tests: ▸ Touching FastlaneExampleTests.xctest (in target ‘FastlaneExampleTests’ from project ‘FastlaneExample’)
[11:41:06]: ▸ 2022-11-01 11:41:06.721 xcodebuild[94719:2551598] [MT] IDETestOperationsObserverDebug: 36.184 elapsed – Testing started completed.
[11:41:06]: ▸ 2022-11-01 11:41:06.721 xcodebuild[94719:2551598] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec – start
[11:41:06]: ▸ 2022-11-01 11:41:06.721 xcodebuild[94719:2551598] [MT] IDETestOperationsObserverDebug: 36.184 sec, +36.184 sec – end
[11:41:06]: ▸ Test Succeeded
±-------------------±–+
| Test Results |
±-------------------±–+
| Number of tests | 2 |
| Number of failures | 0 |
±-------------------±–+
±-----±-----------------±------------+
| fastlane summary |
±-----±-----------------±------------+
| Step | Action | Time (in s) |
±-----±-----------------±------------+
| 1 | default_platform | 0 |
| 2 | run_tests | 45 |
±-----±-----------------±------------+
[11:41:10]: fastlane.tools finished successfully 🎉
[11:41:10]: Shutting down 74F8B615-3882-40A5-89CC-026139BB3D39
Environment
✅ fastlane environment ✅
Stack
Key Value OS 12.6.1 Ruby 3.1.2 Bundler? false Git git version 2.38.1.273.g43a17bfeac-goog Installation Source /opt/homebrew/Cellar/fastlane/2.210.1/libexec/bin/fastlane Host macOS 12.6.1 (21G217) Ruby Lib Dir /opt/homebrew/Cellar/ruby/3.1.2_1/lib OpenSSL Version OpenSSL 1.1.1q 5 Jul 2022 Is contained false Is homebrew true Is installed via Fabric.app false Xcode Path /Applications/Xcode-14.0.1.app/Contents/Developer/ Xcode Version 14.0.1 Swift Version 5.7 System Locale
Variable Value LANG en_US.UTF-8 ✅ LC_ALL LANGUAGE fastlane files:
`./fastlane/Fastfile`
default_platform(:ios) platform :ios do desc "Run Unit Tests" lane :run_unit_tests do run_tests( devices: ["iPhone 14 (16.0)"], prelaunch_simulator: true, reset_simulator: true, code_coverage: true, result_bundle: true, output_directory: "output/test_output/unit_tests", output_types: "html", scheme: "FastlaneExample", clean: true, parallel_testing: true, concurrent_workers: 2 ) end end`./fastlane/Appfile`
# app_identifier("[[APP_IDENTIFIER]]") # The bundle identifier of your app # apple_id("[[APPLE_ID]]") # Your Apple Developer Portal username # For more information about the Appfile, see: # https://docs.fastlane.tools/advanced/#appfilefastlane gems
Gem Version Update-Status fastlane 2.210.1 ✅ Up-To-Date Loaded fastlane plugins:
No plugins Loaded
Loaded gems
Gem Version error_highlight 0.3.0 did_you_mean 1.6.1 atomos 0.1.3 rexml 3.2.5 CFPropertyList 3.0.5 claide 1.1.0 colored2 3.1.2 nanaimo 0.3.0 xcodeproj 1.22.0 rouge 2.0.7 xcpretty 0.3.0 terminal-notifier 2.0.0 unicode-display_width 1.8.0 terminal-table 1.8.0 plist 3.6.0 public_suffix 5.0.0 addressable 2.8.1 multipart-post 2.0.0 word_wrap 1.0.0 optparse 0.1.1 tty-screen 0.8.1 tty-cursor 0.7.1 tty-spinner 0.9.3 artifactory 3.0.15 babosa 1.0.4 colored 1.2 highline 2.0.3 commander 4.6.0 faraday-em_http 1.0.0 faraday-em_synchrony 1.0.0 faraday-excon 1.1.0 faraday-httpclient 1.0.1 faraday-multipart 1.0.4 faraday-net_http 1.0.1 faraday-net_http_persistent 1.2.0 faraday-patron 1.0.0 faraday-rack 1.0.0 faraday-retry 1.0.3 ruby2_keywords 0.0.5 faraday 1.10.2 unf_ext 0.0.8.2 unf 0.1.4 domain_name 0.5.20190701 http-cookie 1.0.5 faraday-cookie_jar 0.0.7 faraday_middleware 1.2.0 fastimage 2.2.6 gh_inspector 1.1.3 mini_magick 4.11.0 naturally 2.2.1 rubyzip 2.3.2 security 0.1.3 xcpretty-travis-formatter 1.0.1 dotenv 2.8.1 bundler 2.3.11 simctl 1.6.8 jwt 2.5.0 emoji_regex 3.2.3 set 1.0.2 excon 0.93.1 json 2.6.2 webrick 1.7.0 httpclient 2.8.3 multi_json 1.15.0 signet 0.17.0 os 1.1.4 memoist 0.16.2 googleauth 1.3.0 mini_mime 1.1.2 retriable 3.1.2 trailblazer-option 0.1.2 declarative 0.0.20 uber 0.1.0 representable 3.2.0 google-apis-core 0.9.1 google-apis-playcustomapp_v1 0.12.0 google-apis-androidpublisher_v3 0.30.0 rake 13.0.6 digest-crc 0.6.4 google-apis-storage_v1 0.19.0 google-apis-iamcredentials_v1 0.16.0 google-cloud-errors 1.3.0 google-cloud-env 1.6.0 google-cloud-core 1.6.0 google-cloud-storage 1.43.0 aws-eventstream 1.2.0 aws-sigv4 1.5.2 aws-partitions 1.654.0 jmespath 1.6.1 aws-sdk-core 3.166.0 aws-sdk-kms 1.59.0 aws-sdk-s3 1.117.1 forwardable 1.3.2 logger 1.5.0 pathname 0.2.0 shellwords 0.1.0 cgi 0.3.1 date 3.2.2 timeout 0.2.0 stringio 3.0.1 securerandom 0.1.1 uri 0.11.0 openssl 3.0.0 digest 3.1.0 io-nonblock 0.1.0 ipaddr 1.2.4 io-wait 0.2.1 zlib 2.1.1 resolv 0.2.1 time 0.2.0 open-uri 0.2.0 mutex_m 0.1.1 net-protocol 0.1.2 net-http 0.2.0 ostruct 0.5.2 english 0.7.1 erb 2.2.3 strscan 3.0.1 abbrev 0.1.0 io-console 0.5.11 tempfile 0.1.2 delegate 0.2.0 fileutils 1.6.0 tmpdir 0.1.2 base64 0.1.1 singleton 0.1.1 open3 0.1.1 nkf 0.1.1 prettyprint 0.1.1 pp 0.3.0 find 0.1.1 yaml 0.2.0 psych 4.0.3 generated on: 2022-11-01
Example
I’ve also created an example project with a fastfile.
- The project has an iOS app.
- The project has 2 unit-tests that runs in parallel.
- Execute
run_unit_testslane from the fastfile. - Observe the
report.htmlunder “output/test_output/unit_tests” is empty whereas theFastlaneExample.xcresultis correct and not empty. FastlaneExample.zip
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 6
- Comments: 17
Same issue here, any workaround?
Not yet. We are just running the tests and later convert the xcresult file to junit reports via an open source library.
Unfortunately no workaround found yet.