playwright: [BUG] Test that passed on first retry - defined as flaky and triggers failure of the whole test run
Context:
- Playwright Version: 1.14.1
- Operating System: Windows
- Browser: Chromium
Code Snippet
Help us help you! Put down a short code snippet that illustrates your bug and that we can run and debug locally. For example:
import { PlaywrightTestConfig } from "@playwright/test";
const config: PlaywrightTestConfig = {
retries: 1,
timeout: 150000,
globalTimeout: 3000000,
reporter: [["./my-reporter.ts"], ["list"], ["junit", { outputFile: "reports/test-results.xml" }], ["allure-playwright"]],
use: {
browserName: 'chromium',
trace: 'on',
screenshot: 'on',
viewport: {width: 1400, height: 900},
video: {
mode: "on-first-retry",
size: {
//UNIFY Recommended Resolution
width: 1400,
height: 900,
}
},
contextOptions: {
//viewport: {width: 1500, height: 1000},
},
// Browser options
launchOptions: {
channel: 'chrome', //Changed back to chromium due to the issue with chrome on CI
headless: false,
//args: ['--start-maximized']
},
//slowMo: 0,
},
};
export default config;
Describe the bug
I run several tests in the test run. I defined the retry parameter to 1. A particular test failed on the first run and finished with success on the retry. But the whole test run failed due to the ‘flaky’ definition of the test. As I understand - the successful retry should be an actual status of the test. Am I wrong?
First run - failed.
Retry - finished successfully.
The whole run - failed.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 19 (10 by maintainers)
Hi,
Same for us, as soon as there are “flaky” the exit code is 1
I just want to mention I’m having the same issue in GitLab CI. I didn’t investigate it further yet.
This is still on Playwright 1.15.0, I’ll also try with the latest version.
I also do have tracing enabled in my config:
@dgozman Aside from the console out that comes from the
listreporter, the only other thing that was unusual was theInternal errorlog I pasted in my previous comment.Here is the rest of the output from our CI run where this issue occurred. You can see, no failed tests (just a couple flaky). But still
exit code 1.