webdriverio: [🐛 Bug]: @wdio/junit-reporter in WebDriverIO v8 with CucumberJS shows step counts instead of test counts in JUnit report

Have you read the Contributing Guidelines on issues?

WebdriverIO Version

8.20.4

Node.js Version

18.18.2

Mode

WDIO Testrunner

Which capabilities are you using?

{
browserName: 'chrome',
          'wdio:devtoolsOptions': {
              headless: false
          }

}

What happened?

The JUnit reporter in WebDriverIO with CucumberJS is producing reports that display the count of individual test steps instead of counting complete test scenarios.

What is your expected behavior?

Each test scenario (each “Given,” “When,” “Then” sequence) should be counted as a single test case in the JUnit report, providing a clear and accurate overview of the test coverage.

How to reproduce the bug.

STR:

  1. Clone Offical boilerplate https://github.com/amiya-pattnaik/webdriverIO-with-cucumberBDD
  2. npm install
  3. Enable Juinit reporter in wdio.shared.conf.ts
  4. Run tests npm run test-local
  5. Go to reports folder
  6. Open JUinit report
  7. Observe generate JUnit report at test/reports/junit-results

actual state: The JUnit reporter in WebDriverIO with Cucumber is producing reports that display the count of individual test steps instead of counting complete test scenarios.

<testsuite name="As a user I can log into the secure area" timestamp="2023-11-01T21:55:07" time="2.554" tests="3" failures="0" errors="0" skipped="0">

expected state: Each test scenario (each “Given,” “When,” “Then” sequence) should be counted as a single test case in the JUnit report, providing a clear and accurate overview of the test coverage.

<testsuite name="As a user I can log into the secure area" timestamp="2023-11-01T21:55:07" time="2.554" tests="1" failures="0" errors="0" skipped="0">

Relevant log output

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="As a user I can log into the secure area" timestamp="2023-11-01T21:55:07" time="2.554" tests="3" failures="0" errors="0" skipped="0">
    <properties>
      <property name="specId" value="0"/>
      <property name="suiteName" value="As a user, I can log into the secure area"/>
      <property name="capabilities" value="chrome.118_0_5993_117.mac"/>
      <property name="file" value="file://./test/features/login.feature"/>
    </properties>
    <testcase classname="chrome.118_0_5993_117.mac.login.feature:1:1:_As_a_user,_I_can_log_into_the_secure_area" name="Given I am on the login page" time="1.956">
      <system-out><![CDATA[
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/url - {"url":"https://the-internet.herokuapp.com/login"}
RESULT: {"url":"https://the-internet.herokuapp.com/login"}
]]></system-out>
    </testcase>
    <testcase classname="chrome.118_0_5993_117.mac.login.feature:1:1:_As_a_user,_I_can_log_into_the_secure_area" name="When I login with tomsmith and SuperSecretPassword" time="0.556">
      <system-out><![CDATA[
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element - {"using":"css selector","value":"#username"}
RESULT: {"using":"css selector","value":"#username"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/541E094B7486579404A597B8FBB32F15_element_3/clear - {}
RESULT: {}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/541E094B7486579404A597B8FBB32F15_element_3/value - {"text":"tomsmith"}
RESULT: {"text":"tomsmith"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element - {"using":"css selector","value":"#password"}
RESULT: {"using":"css selector","value":"#password"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/541E094B7486579404A597B8FBB32F15_element_4/clear - {}
RESULT: {}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/541E094B7486579404A597B8FBB32F15_element_4/value - {"text":"SuperSecretPassword!"}
RESULT: {"text":"SuperSecretPassword!"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element - {"using":"css selector","value":"button[type=\"submit\"]"}
RESULT: {"using":"css selector","value":"button[type=\"submit\"]"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/541E094B7486579404A597B8FBB32F15_element_10/click - {}
RESULT: {}
]]></system-out>
    </testcase>
    <testcase classname="chrome.118_0_5993_117.mac.login.feature:1:1:_As_a_user,_I_can_log_into_the_secure_area" name="Then I should see a flash message saying You logged into a secure area" time="0.038">
      <system-out><![CDATA[
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element - {"using":"css selector","value":"#flash"}
RESULT: {"using":"css selector","value":"#flash"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/elements - {"using":"css selector","value":"#flash"}
RESULT: {"using":"css selector","value":"#flash"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element - {"using":"css selector","value":"#flash"}
RESULT: {"using":"css selector","value":"#flash"}
COMMAND: GET /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/01B1D3E994A64795A27A65935ECF9044_element_13/text - {}
RESULT: {}
]]></system-out>
    </testcase>
  </testsuite>
  <testsuite name="As a user I can log into the secure area" timestamp="2023-11-01T21:55:09" time="0.591" tests="3" failures="0" errors="0" skipped="0">
    <properties>
      <property name="specId" value="0"/>
      <property name="suiteName" value="As a user, I can log into the secure area"/>
      <property name="capabilities" value="chrome.118_0_5993_117.mac"/>
      <property name="file" value="file://./test/features/login.feature"/>
    </properties>
    <testcase classname="chrome.118_0_5993_117.mac.login.feature:1:1:_As_a_user,_I_can_log_into_the_secure_area" name="Given I am on the login page" time="0.165">
      <system-out><![CDATA[
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/url - {"url":"https://the-internet.herokuapp.com/login"}
RESULT: {"url":"https://the-internet.herokuapp.com/login"}
]]></system-out>
    </testcase>
    <testcase classname="chrome.118_0_5993_117.mac.login.feature:1:1:_As_a_user,_I_can_log_into_the_secure_area" name="When I login with foobar and barfoo" time="0.393">
      <system-out><![CDATA[
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element - {"using":"css selector","value":"#username"}
RESULT: {"using":"css selector","value":"#username"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/1D272672F6FCE9D8D78046122DDC21BE_element_17/clear - {}
RESULT: {}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/1D272672F6FCE9D8D78046122DDC21BE_element_17/value - {"text":"foobar"}
RESULT: {"text":"foobar"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element - {"using":"css selector","value":"#password"}
RESULT: {"using":"css selector","value":"#password"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/1D272672F6FCE9D8D78046122DDC21BE_element_18/clear - {}
RESULT: {}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/1D272672F6FCE9D8D78046122DDC21BE_element_18/value - {"text":"barfoo"}
RESULT: {"text":"barfoo"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element - {"using":"css selector","value":"button[type=\"submit\"]"}
RESULT: {"using":"css selector","value":"button[type=\"submit\"]"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/1D272672F6FCE9D8D78046122DDC21BE_element_25/click - {}
RESULT: {}
]]></system-out>
    </testcase>
    <testcase classname="chrome.118_0_5993_117.mac.login.feature:1:1:_As_a_user,_I_can_log_into_the_secure_area" name="Then I should see a flash message saying Your username is invalid" time="0.031">
      <system-out><![CDATA[
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element - {"using":"css selector","value":"#flash"}
RESULT: {"using":"css selector","value":"#flash"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/elements - {"using":"css selector","value":"#flash"}
RESULT: {"using":"css selector","value":"#flash"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/element - {"using":"css selector","value":"#flash"}
RESULT: {"using":"css selector","value":"#flash"}
COMMAND: GET /session/a021b7b72ba5c7c898ad991cca6a8d3e/element/E7F58260FE635C38EEEAEB1F0B9FD73C_element_28/text - {}
RESULT: {}
COMMAND: GET /session/a021b7b72ba5c7c898ad991cca6a8d3e/se/log/types - {}
RESULT: {}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/se/log - {"type":"driver"}
COMMAND: POST /session/a021b7b72ba5c7c898ad991cca6a8d3e/se/log - {"type":"browser"}
RESULT: {"type":"driver"}
RESULT: {"type":"browser"}
COMMAND: DELETE /session/a021b7b72ba5c7c898ad991cca6a8d3e - {}
RESULT: {}
]]></system-out>
    </testcase>
  </testsuite>
  <testsuite name="The Internet Guinea Pig Website" timestamp="2023-11-01T21:55:07" time="3.15" tests="0" failures="0" errors="0" skipped="0">
    <properties>
      <property name="specId" value="0"/>
      <property name="suiteName" value="The Internet Guinea Pig Website"/>
      <property name="capabilities" value="chrome.118_0_5993_117.mac"/>
      <property name="file" value="file://./test/features/login.feature"/>
    </properties>
  </testsuite>
</testsuites>

Code of Conduct

  • I agree to follow this project’s Code of Conduct

Is there an existing issue for this?

  • I have searched the existing issues

About this issue

  • Original URL
  • State: open
  • Created 8 months ago
  • Comments: 15 (9 by maintainers)

Most upvoted comments

Thanks for providing these logs, it draws a clear picture. I assume someone has added more detail to the JUnit reporter because they had a need to have more details per step execution. I see that we have controls for this behavior in the Allure reporter.

My suggestion is to allow to define this behavior as reporter configuration and let people choose if they like to define every step as a test case or ever scenario.

Any contributions are appreciated!

+1
christian-bromann on Nov 2, 2023
Read more comments on GitHub
← webdriverio: [🐛 Bug]: `@wdio/devtools-service` support for Selenoid
webdriverio: [🐛 Bug]: Worker starts with downloading the chromedriver from incorrect path what leads to a failure →