webdriverio: [đ Bug]: Allure Reporter / Retries not listed anymore.
Have you read the Contributing Guidelines on issues?
- I have read the Contributing Guidelines on issues.
WebdriverIO Version
8.5.6
Node.js Version
18
Mode
WDIO Testrunner
Which capabilities are you using?
"@wdio/allure-reporter": "^8.5.0"
"@wdio/mocha-framework": "^8.5.6",
What happened?
It seems that the allure reporter creates a new step instead of a new entry in the âRetriesâ Tab (if a test fails and retries are configure in wdio). I believe this also leads to unstable builds in jenkins. In previuos version (wdio7 and corresponding allure reporter version) retries were listed in the âRetriesâ Tab and did not made jenkins unstable.
I attached some screenshots for comparison.
What is your expected behavior?
Retries are correctly listed in the retry tab and hopefully the publisher will not consider it as unstable anymore.
How to reproduce the bug.
Let a test fail and look into the generated allure report.
Relevant log output
n.a.
Code of Conduct
- I agree to follow this projectâs Code of Conduct
Is there an existing issue for this?
- I have searched the existing issues
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 24 (22 by maintainers)
Commits related to this issue
- #9993 Remove old implementation — committed to webdriverio/webdriverio by BorisOsipov a year ago
- #9993 The allure historyId should be set correctly by taking into account the suite, test name, and all user-given parameters for the calculation of the historyId. — committed to webdriverio/webdriverio by BorisOsipov a year ago
- #9993 Properly set allure history id (#10176) * #9993 Remove old implementation * #9993 The allure historyId should be set correctly by taking into account the suite, test name, and all user-given... — committed to webdriverio/webdriverio by BorisOsipov a year ago
@praveendvd
I donât remember that any one registered such issue since wdio 4. I prefer not to solve issues that have never happened.
It is not our problem. If the user wants to make a unique test, they can add a parameter with any identifier they want using
addArgument(name, value). It is our responsibility to make it possible by generating the correct historyId from the full test name and parameters user given.@praveendvd I took the changes from https://github.com/webdriverio/webdriverio/pull/10053 The result of âusing hash of suite name and test nameâ is that if you run the tests with 2 browsers (means each test runs in both browsers), the steps from second browser will be listed as retry, because it has same suite and test name. Basically, browser name/version is missing in the hash, I guess.
So, things are different now but not working as they should be. Now you probably get green runs via the jenkins allure publisher, even though tests were failing in one browser.
No, we can close this. If you like to raise a PR with a test for this, feel free. Thanks for the ping.
@praveendvd You donât need âcidâ. You canât use âcidâ in âhistoryIdâ generation.
As @muhserks mentioned, we have to add the browser name to âhistoryIdâ generation, along with all test parameters.
The Allure1 generator generates âhistoryIdâ when it is missing, like this - https://github.com/allure-framework/allure2/blob/master/allure-generator/src/main/java/io/qameta/allure/allure1/Allure1Plugin.java#L494
We should make our âhistoryIdâ generation as close as possible to that implementation.
I am not familiar with the Allure reporter, maybe @epszaw has some insights for #10053
@christian-bromann i will take this up