webdriverio: [šŸ› Bug]: If use before/suiteSetup hook, then the first test would be duplicated in allure report

Have you read the Contributing Guidelines on issues?

WebdriverIO Version

8.5.8

Node.js Version

16.14.2 (OS X 12.5)

Mode

Standalone Mode

Which capabilities are you using?

capabilities: [{
        maxInstances: 5,
        browserName: 'chrome',
        acceptInsecureCerts: true
    }],

What happened?

If use before hook (or suiteSetup if you are using TDD interface) using mocha test framework, then the first test would be duplicated inside of allure report. If change hook to beforeEach or setup (for TDD) then test wouldnā€™t be duplicated inside of report.

https://imgur.com/SLre3Oc.png - result with before hook https://imgur.com/K8owHqc.png - result with beforeEach

If add steps inside of test, then one test will contain the steps, but another one would be missing them.

What is your expected behavior?

Tests arenā€™t duplicated inside of report

How to reproduce the bug.

Setup a new project

npm init wdio ./path/to/new/project -- --yes

Install allure reporter

npm install @wdio/allure-reporter --save-dev

Install allure cli

npm install -g allure-commandline --save-dev

Add allure to wdio.conf.js that was automatically generated in the root folder:

    reporters: [
        'spec',
        ['allure', {
            outputDir: 'report/allure-results',
            disableWebdriverStepsReporting: true,
            disableWebdriverScreenshotsReporting: true,
        }]],

Open test that was automatically generated in ./test/specs/example.e2e.js and update it to use before hook. You can place await LoginPage.open() into the hook:

import LoginPage from '../pageobjects/login.page.js'
import SecurePage from '../pageobjects/secure.page.js'

describe('My Login application', () => {
    before(async () => {
        await LoginPage.open()
    })
    it('should login with valid credentials', async () => {

        await LoginPage.login('tomsmith', 'SuperSecretPassword!')
        await expect(SecurePage.flashAlert).toBeExisting()
        await expect(SecurePage.flashAlert).toHaveTextContaining(
            'You logged into a secure area!')
    })
})

run tests with

npx wdio wdio.conf.js

After test is passed, open allure report:

allure serve report/allure-results/

Open suites tab and two tests instead of one would be shown

Relevant log output

2023-03-10T18:14:23.735Z INFO chromedriver: Initiate Chromedriver Launcher (v8.1.1)
2023-03-10T18:14:23.736Z INFO @wdio/cli:launcher: Run onPrepare hook
2023-03-10T18:14:23.737Z INFO chromedriver: Start Chromedriver (/Users/artur/WebstormProjects/webdriverIoreproduce/node_modules/chromedriver/lib/chromedriver/chromedriver) with args --port=9515 --url-base=/
2023-03-10T18:14:23.752Z INFO chromedriver: Starting ChromeDriver 111.0.5563.64 (c710e93d5b63b7095afe8c2c17df34408078439d-refs/branch-heads/5563@{#995}) on port 9515
2023-03-10T18:14:23.752Z INFO chromedriver: Only local connections are allowed.
2023-03-10T18:14:23.752Z INFO chromedriver: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
2023-03-10T18:14:23.754Z INFO chromedriver: ChromeDriver was started successfully.
2023-03-10T18:14:23.853Z INFO @wdio/cli:launcher: Run onWorkerStart hook
2023-03-10T18:14:23.854Z INFO @wdio/local-runner: Start worker 0-0 with arg: wdio.conf.js
[0-0] 2023-03-10T18:14:24.376Z INFO @wdio/local-runner: Run worker command: run
[0-0] RUNNING in chrome - file:///test/specs/example.e2e.js
[0-0] 2023-03-10T18:14:24.500Z INFO webdriver: Initiate new session using the WebDriver protocol
[0-0] 2023-03-10T18:14:24.549Z INFO webdriver: [POST] http://localhost:9515/session
[0-0] 2023-03-10T18:14:24.549Z INFO webdriver: DATA {
[0-0]   capabilities: {
[0-0]     alwaysMatch: { browserName: 'chrome', acceptInsecureCerts: true },
[0-0]     firstMatch: [ {} ]
[0-0]   },
[0-0]   desiredCapabilities: { browserName: 'chrome', acceptInsecureCerts: true }
[0-0] }
[0-0] 2023-03-10T18:14:25.619Z INFO webdriver: COMMAND navigateTo("https://the-internet.herokuapp.com/login")
[0-0] 2023-03-10T18:14:25.619Z INFO webdriver: [POST] http://localhost:9515/session/40bc4a1d6fb95319b679ead48d1c3d9a/url
[0-0] 2023-03-10T18:14:25.619Z INFO webdriver: DATA { url: 'https://the-internet.herokuapp.com/login' }
[0-0] 2023-03-10T18:14:27.393Z INFO webdriver: RESULT null
[0-0] 2023-03-10T18:14:27.411Z INFO webdriver: COMMAND findElement("css selector", "#username")
[0-0] 2023-03-10T18:14:27.411Z INFO webdriver: [POST] http://localhost:9515/session/40bc4a1d6fb95319b679ead48d1c3d9a/element
[0-0] 2023-03-10T18:14:27.411Z INFO webdriver: DATA { using: 'css selector', value: '#username' }
[0-0] 2023-03-10T18:14:27.423Z INFO webdriver: RESULT {
[0-0]   'element-6066-11e4-a52e-4f735466cecf': 'acaa5eb5-be6e-4aef-9cd6-c05cb26da3bc'
[0-0] }
[0-0] 2023-03-10T18:14:27.429Z INFO webdriver: COMMAND elementClear("acaa5eb5-be6e-4aef-9cd6-c05cb26da3bc")
[0-0] 2023-03-10T18:14:27.429Z INFO webdriver: [POST] http://localhost:9515/session/40bc4a1d6fb95319b679ead48d1c3d9a/element/acaa5eb5-be6e-4aef-9cd6-c05cb26da3bc/clear
[0-0] 2023-03-10T18:14:27.453Z INFO webdriver: RESULT null
[0-0] 2023-03-10T18:14:27.453Z INFO webdriver: COMMAND elementSendKeys("acaa5eb5-be6e-4aef-9cd6-c05cb26da3bc", "<Screenshot[base64]>")
[0-0] 2023-03-10T18:14:27.453Z INFO webdriver: [POST] http://localhost:9515/session/40bc4a1d6fb95319b679ead48d1c3d9a/element/acaa5eb5-be6e-4aef-9cd6-c05cb26da3bc/value
[0-0] 2023-03-10T18:14:27.453Z INFO webdriver: DATA { text: 'tomsmith' }
[0-0] 2023-03-10T18:14:27.512Z INFO webdriver: RESULT null
[0-0] 2023-03-10T18:14:27.513Z INFO webdriver: COMMAND findElement("css selector", "#password")
[0-0] 2023-03-10T18:14:27.513Z INFO webdriver: [POST] http://localhost:9515/session/40bc4a1d6fb95319b679ead48d1c3d9a/element
[0-0] 2023-03-10T18:14:27.513Z INFO webdriver: DATA { using: 'css selector', value: '#password' }
[0-0] 2023-03-10T18:14:27.521Z INFO webdriver: RESULT {
[0-0]   'element-6066-11e4-a52e-4f735466cecf': '3e5e0275-b468-47b7-b64d-68fc24186c80'
[0-0] }
[0-0] 2023-03-10T18:14:27.524Z INFO webdriver: COMMAND elementClear("3e5e0275-b468-47b7-b64d-68fc24186c80")
[0-0] 2023-03-10T18:14:27.524Z INFO webdriver: [POST] http://localhost:9515/session/40bc4a1d6fb95319b679ead48d1c3d9a/element/3e5e0275-b468-47b7-b64d-68fc24186c80/clear
[0-0] 2023-03-10T18:14:27.545Z INFO webdriver: RESULT null
[0-0] 2023-03-10T18:14:27.545Z INFO webdriver: COMMAND elementSendKeys("3e5e0275-b468-47b7-b64d-68fc24186c80", "SuperSecretPassword!")
[0-0] 2023-03-10T18:14:27.545Z INFO webdriver: [POST] http://localhost:9515/session/40bc4a1d6fb95319b679ead48d1c3d9a/element/3e5e0275-b468-47b7-b64d-68fc24186c80/value
[0-0] 2023-03-10T18:14:27.546Z INFO webdriver: DATA { text: 'SuperSecretPassword!' }
[0-0] 2023-03-10T18:14:27.576Z INFO webdriver: RESULT null
[0-0] 2023-03-10T18:14:27.576Z INFO webdriver: COMMAND findElement("css selector", "button[type="submit"]")
[0-0] 2023-03-10T18:14:27.577Z INFO webdriver: [POST] http://localhost:9515/session/40bc4a1d6fb95319b679ead48d1c3d9a/element
[0-0] 2023-03-10T18:14:27.577Z INFO webdriver: DATA { using: 'css selector', value: 'button[type="submit"]' }
[0-0] 2023-03-10T18:14:27.586Z INFO webdriver: RESULT {
[0-0]   'element-6066-11e4-a52e-4f735466cecf': '2f7c34a6-c512-421a-9fba-c98dabf16f3e'
[0-0] }
[0-0] 2023-03-10T18:14:27.588Z INFO webdriver: COMMAND elementClick("2f7c34a6-c512-421a-9fba-c98dabf16f3e")
[0-0] 2023-03-10T18:14:27.588Z INFO webdriver: [POST] http://localhost:9515/session/40bc4a1d6fb95319b679ead48d1c3d9a/element/2f7c34a6-c512-421a-9fba-c98dabf16f3e/click
[0-0] 2023-03-10T18:14:27.917Z INFO webdriver: RESULT null
[0-0] 2023-03-10T18:14:27.918Z INFO webdriver: COMMAND findElement("css selector", "#flash")
[0-0] 2023-03-10T18:14:27.918Z INFO webdriver: [POST] http://localhost:9515/session/40bc4a1d6fb95319b679ead48d1c3d9a/element
[0-0] 2023-03-10T18:14:27.918Z INFO webdriver: DATA { using: 'css selector', value: '#flash' }
[0-0] 2023-03-10T18:14:27.928Z INFO webdriver: RESULT {
[0-0]   'element-6066-11e4-a52e-4f735466cecf': '91d5edb3-c2c5-4d1d-9b0e-86d528a2847d'
[0-0] }
[0-0] 2023-03-10T18:14:27.930Z INFO webdriver: COMMAND findElements("css selector", "#flash")
[0-0] 2023-03-10T18:14:27.931Z INFO webdriver: [POST] http://localhost:9515/session/40bc4a1d6fb95319b679ead48d1c3d9a/elements
[0-0] 2023-03-10T18:14:27.931Z INFO webdriver: DATA { using: 'css selector', value: '#flash' }
[0-0] 2023-03-10T18:14:27.940Z INFO webdriver: RESULT [
[0-0]   {
[0-0]     'element-6066-11e4-a52e-4f735466cecf': '91d5edb3-c2c5-4d1d-9b0e-86d528a2847d'
[0-0]   }
[0-0] ]
[0-0] 2023-03-10T18:14:27.944Z INFO webdriver: COMMAND findElement("css selector", "#flash")
[0-0] 2023-03-10T18:14:27.944Z INFO webdriver: [POST] http://localhost:9515/session/40bc4a1d6fb95319b679ead48d1c3d9a/element
[0-0] 2023-03-10T18:14:27.944Z INFO webdriver: DATA { using: 'css selector', value: '#flash' }
[0-0] 2023-03-10T18:14:27.950Z INFO webdriver: RESULT {
[0-0]   'element-6066-11e4-a52e-4f735466cecf': '91d5edb3-c2c5-4d1d-9b0e-86d528a2847d'
[0-0] }
[0-0] 2023-03-10T18:14:27.952Z INFO webdriver: COMMAND getElementText("91d5edb3-c2c5-4d1d-9b0e-86d528a2847d")
[0-0] 2023-03-10T18:14:27.953Z INFO webdriver: [GET] http://localhost:9515/session/40bc4a1d6fb95319b679ead48d1c3d9a/element/91d5edb3-c2c5-4d1d-9b0e-86d528a2847d/text
[0-0] 2023-03-10T18:14:27.962Z INFO webdriver: RESULT You logged into a secure area!
[0-0] Ɨ
[0-0] 2023-03-10T18:14:27.965Z INFO webdriver: COMMAND deleteSession()
[0-0] 2023-03-10T18:14:27.965Z INFO webdriver: [DELETE] http://localhost:9515/session/40bc4a1d6fb95319b679ead48d1c3d9a
[0-0] 2023-03-10T18:14:28.021Z INFO webdriver: RESULT null
[0-0] PASSED in chrome - file:///test/specs/example.e2e.js
2023-03-10T18:14:28.136Z INFO @wdio/cli:launcher: Run onWorkerEnd hook
2023-03-10T18:14:28.137Z INFO @wdio/cli:launcher: Run onComplete hook

Code of Conduct

  • I agree to follow this projectā€™s Code of Conduct

Is there an existing issue for this?

  • I have searched the existing issues

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 2
  • Comments: 17 (13 by maintainers)

Most upvoted comments

Had similar issue and adding disableMochaHooks: true helped.

    reporters: [
        'spec',
        ['allure', {
            outputDir: 'report/allure-results',
            disableWebdriverStepsReporting: true,
            disableWebdriverScreenshotsReporting: true,
            disableMochaHooks: true,
        }]],
+3
lukaszczerpak on Mar 29, 2023

@lukaszczerpak yes, I need information from hooks in the report

+2
7artur7 on Mar 30, 2023

Hey @BorisOsipov, Iā€™ve rechecked everything, and the released code is now functioning correctly. The errors mentioned in the previous comment were due to my use of incomplete code modifications.

It is preferred because Allure supports test fixtures. Also, in cases where we can, we report hooks as fixtures, because it is wrong to report them as test cases, because hooks are not test cases and it affects the total number of test cases in different reporters and make it difficult to maintain statistics.

Iā€™ve been experiencing the same issue with my project, so my idea was to collaborate on this change to generate a better report. The current approach is working as intended, and Iā€™ve also updated the test cases accordingly.

What documentation are you referring to? Could you provide an exact link to the lines in the documentation that differ from the current implementation of the reporter? or a quote from the documentation.

Regarding the documentation, I referenced it in a previous comment (https://allurereport.org/docs). Iā€™ve reviewed the documentation, and it aligns with the current implementation of the reporter.

Iā€™ve made all the necessary changes to work with fixtures, and Iā€™d like to create a pull request for a code review. Iā€™ll open a new issue to track these changes. What do you think?

+1
lararojasmr on Oct 19, 2023

@lararojasmr

Currently, thereā€™s an issue with the reporter in WebDriverIO (wdio) as it doesnā€™t adhere to the latest Allure documentation. This results in suboptimal reports when dealing with complex suite executions.

What documentation are you referring to? Could you provide an exact link to the lines in the documentation that differ from the current implementation of the reporter? or a quote from the documentation.

Iā€™m in the process of making necessary changes to align with the updated reporting methodology. As you can observe in the screenshot Iā€™ve shared, the ā€œbeforeā€ sections are not actually tests; they are integral parts of the test cases or suites. This leads to a question: Why is it preferable to report these hooks as test suites?

It is preferred because Allure supports test fixtures. Also, in cases where we can, we report hooks as fixtures, because it is wrong to report them as test cases, because hooks are not test cases and it affects the total number of test cases in different reporters and make it difficult to maintain statistics.

using this option enable (useCucumberStepReporter) Mocha:

We donā€™t support that option for Mocha\Jasmine test runners. It is cucumber specific option only.

Different issues are reproducible with current release: 8.19.0, allure report with useCucumberStepReporter in true.

I created a brand new project via the Getting Started Guide and allowed the generator to create sample feature files - they run correctly. How can we reproduce your problem? Please share the sample project. Also, it will be nice to report it as a separate issue because it is completely unrelated to the original one.

+1
BorisOsipov on Oct 19, 2023

@lararojasmr you want to help us take a stab with a fix for this?

+1
christian-bromann on May 17, 2023

@lukaszczerpak thanks for info. Unfortunately hooks may contain useful info and skipping it doesnā€™t work as a solid solution.

+1
7artur7 on Mar 29, 2023
Read more comments on GitHub
← webdriverio: [šŸ› Bug]: Getting Started does not work on Windows 10
webdriverio: [šŸ› Bug]: Import giving error ERROR @wdio/runner: Error: Cannot find package with paths defined in tsconfig.json →