cypress: Support mocha test retries / retry test failures

We’re still working on this, but in the meantime I’ve made a plugin you can try out cypress-plugin-retries:

It supports mocha’s this.retries, and it also retries failures in beforeEach

🚨 Do not comment on this issue about issues with the plugin 🚨

Let me know any bugs in the issues of the plugin repo.

We discussed this today as a team. It may be the next feature @chrisbreiding works on after Firefox lands.

Would love to see that feature, also to enable that globally for all tests, e.g. through cypress.json.

You can launch Cypress using the Module API , and effectively collect failed spec files from the run and tell cypress to run just those files. Here’s an example: https://gist.github.com/Bkucera/4ffd05f67034176a00518df251e19f58

Then, instead of cypress run, you use node cypress-retries.js

We’ve internally talked about this before and we’ll be doing something different from the way mocha implements retries.

It will work the same - but will be controlled by different API’s than what mocha provides. I’ll open a proposal later and link it to this issue.

not trying to minimize the importance of getting this in Cypress core. We hear you, it’s up next to be worked on.

But since Github has started collapsing comments in this thread I wanted to make sure people see the workaround plugin posted above:

We’re still working on this, but in the meantime I’ve made a plugin you can try out cypress-plugin-retries:

It supports mocha’s retries, and it also retries failures in beforeEach

🚨 Do not comment on this issue about issues with the plugin 🚨

Let me know any bugs in the issues of the plugin repo.

For anyone still waiting for the lovely Cypress team to get around to this, we came up with a solution we really like. We were using @Bkucera’s handy cypress-retries script, but it didn’t work with our parallelized CI setup and we didn’t like re-running the whole test file just to retry a single failed test, so we wrote this (somewhat hacky) little module:

if (Cypress.env('RETRIES')) {
  let skip = false;
  const _it = window.it;
  const _beforeEach = window.beforeEach;

  window.beforeEach = (...args) => {
    const beforeEachFn = args.pop();
    args.push(() => {
      if (!skip) beforeEachFn();
    });
    _beforeEach(...args);
  };

  window.it = (testName, testFn) => {
    if (!testFn) {
      _it(testName);
      return;
    }

    _it(`${testName} 🔄 1 of 3`, function() {
      skip = true;
      cy.on('fail', () => {
        skip = false;
        this.skip();
      });
      testFn();
    });

    _it(`${testName} 🔄 2 of 3`, function() {
      if (skip) this.skip();
      skip = true;
      cy.on('fail', () => {
        skip = false;
        this.skip();
      });
      testFn();
    });

    _it(`${testName} 🔄 3 of 3`, function() {
      const _skip = skip;
      skip = false;
      if (_skip) this.skip();
      testFn();
    });
  };

  window.it.skip = _it.skip;
  window.it.only = _it.only;
}

This simply monkeypatches it and beforeEach so that every test is tried three times. It obviously doesn’t cover every use case but it works well for us. Just set the env variable CYPRESS_RETRIES=true & stick the code somewhere in your cypress support setup.

Test retries has been released in 5.0.0.

You can refer to our docs on Test Retries for instructions on how to turn on and configure test retries.

There are several options which we think are important to explore depending on your project including configuring globally, per cypress open and cypress run separately, or per test suite / single test.

We think this will be a valuable tool in helping identify flake and have more plans in the future to improve this feature and surface the test retries as well as analytics on flake into our Dashboard.

If you encounter any issues using Test Retries, please open a new issue with a fully reproducible example - do not comment in this thread as this issue is closed.

Is there any ETA on this feature, or estimate of roughly how much work remains? This feature would make our continuous integration pipeline much more forgiving of random temporary network issues and other flakyness. 🙂

would love this feature to come out of the box for cypress: a global setting to retry failed tests x number of times to reduce timeout flakiness.

@brian-mann any news on where this is with the team?

Great feature! Can you already give some insight in when it will be available? 😃

The code for this is done in cypress-io/cypress#3968, but has yet to be released. We’ll update this issue and reference the changelog when it’s released.

+1 for this feature, along with this it would also be nice to allow global and override retries in mocha it block.

@germyjen don’t think this is a top priority when there’s a perfectly working workaround https://github.com/Bkucera/cypress-plugin-retries

It looks like the Cypress team is building out native support for retries, so stay tuned!

@nanoflat I’ve updated the cypress-retries script to work with Parallelization.

The script should be able to work with parallel runs and non parallel runs. Just pass your configuration to DEFAULT_CONFIG

Here’s how it will look on the Dashboard

Some initial work was completed on this, but there is still a good amount of work to be done. For the moment, the release of 4.0 and its features are being prioritized (it has some prerequisite work for this feature).

We recently started using milestones to track sprints (weekly work), which is why this is tagged with a closed milestone. I’m going to remove that milestone, since it’s no longer relevant. When we start working on this again, we’ll tag it with the latest sprint milestone. When this feature is complete, this issue will be closed and when it’s released in a new version of Cypress, we’ll post a message saying so.

@itaykotler-fundbox yes, the Dashboard will mark build as failed, and CI will say pass. Once we implement retries for real, that problem should be solved and Dashboard will pass along with giving you insights about your flakey tests 😄 and we currently have someone working on it.

@vcamposs we are using the script and works great. We adding this to our Cypress.json:

{
...
  "env": {
    "RETRY_FAILED_TESTS": true
  },
"video": false,
...
}

then your script that you copy pasted into your index.js can start like this:

if (Cypress.env("RETRY_FAILED_TESTS")) {
...

Also make sure you don’t use any it.only commands

For anyone still waiting for the lovely Cypress team to get around to this, we came up with a solution we really like. We were using @Bkucera’s handy cypress-retries script, but it didn’t work with our parallelized CI setup and we didn’t like re-running the whole test file just to retry a single failed test, so we wrote this (somewhat hacky) little module:

if (Cypress.env('RETRIES')) {
  let skip = false;
  const _it = window.it;
  const _beforeEach = window.beforeEach;

  window.beforeEach = (...args) => {
    const beforeEachFn = args.pop();
    args.push(() => {
      if (!skip) beforeEachFn();
    });
    _beforeEach(...args);
  };

  window.it = (testName, testFn) => {
    if (!testFn) {
      _it(testName);
      return;
    }

    _it(`${testName} 🔄 1 of 3`, function() {
      skip = true;
      cy.on('fail', () => {
        skip = false;
        this.skip();
      });
      testFn();
    });

    _it(`${testName} 🔄 2 of 3`, function() {
      if (skip) this.skip();
      skip = true;
      cy.on('fail', () => {
        skip = false;
        this.skip();
      });
      testFn();
    });

    _it(`${testName} 🔄 3 of 3`, function() {
      const _skip = skip;
      skip = false;
      if (_skip) this.skip();
      testFn();
    });
  };

  window.it.skip = _it.skip;
  window.it.only = _it.only;
}

This simply monkeypatches it and beforeEach so that every test is tried three times. It obviously doesn’t cover every use case but it works well for us. Just set the env variable CYPRESS_RETRIES=true & stick the code somewhere in your cypress support setup.

Hello, @debrisapron . I’m sorry about a dumb question, but to use your script, i just need put in .js and run switch like “npm run e2e-test CYPRESS_RETRIES=true” ?

I did that but there aren’t any retry.

This feature will be so useful! +1

Hello there, I wanted to chime in and try to give you an understanding that this is currently a Number 1 issue and is actually costing you money:

We just had to switch from our 5000$ a year cypress plan to our own parallel implementation. This is due to the issues described above: You can’t retry a cypress test reliably in a pipeline when using the parallel feature.

For us this even meant that once you retried a cypress test it always went “green”, so we actually deployed failing stuff.

So this issue alone will cose you 5000$ a year now. I would be “SO” happy to pay you for your services, but as it is now, we cannot use your commercial product, even though we want to.

@Bkucera this is not really a workaround, because one can still manually retry a test in e.g. GitLab und force it to pass, even if there are real errors that where introduced with this test.

Furthermore it only retries them automatically. Chances are high that the test failure was introduced by a third party system not being available. The plugin would not do anything for that kind of failure.

@jennifer-shehane this would be such a great feature to help reduce flakiness in our tests. Is there an estimated timeline for this feature?

I’m happy this will be worked on. Currently, I’m using if-then blocks to collect failed tests to be re-run at the end of a spec-file completed run.

@bahmutov @brian-mann I think that with retries by workaround https://github.com/cypress-io/cypress/issues/1313#issuecomment-409342834 with Module API will not work with cypress 3.1.x with the parallel run. The thing that scope of specs will not be same from original run to retry - because some tests will fail and some will pass. So run will fail with https://docs.cypress.io/guides/references/error-messages.html#Cannot-parallelize-tests-across-environments

One of the recent flaky failures we had was the login helper we use to auth pages through an API before visiting pages. Worked hundreds of times without failing, failing once caused the whole CI run to fail.

We’ll look into the module API script for now. Still showing a test failing will help to know when/what is flaky. Sometimes it is a poorly written test, sometimes it is a complex system failing over something seemingly stable.