cypress: Provide a way for custom commands to retry on failed should()

You can make an arbitrary function retry and pass along its return value using a small hack that combines should and invoke

Here’s an example of a custom command that makes sure all h2’s are in alphabetical order, and it retries:

cy.get('h2')
.should(($els) => $els.fn = () => {
  return $els.toArray().map(el => el.innerText)
})
.invoke('fn')
.should(headers => {
  const sortedHeaders = headers.concat().sort()
  expect(headers).deep.eq(sortedHeaders)
})

or

better yet, add a new command that does this called try:

Cypress.Commands.addAll({
  prevSubject: 'optional',
},
{
  try: (subject, fn) => {
    return cy.wrap({ try: () => fn(subject) }, { log: false })
    .invoke('try')

  },
})

and use it like:

cy.get('h2').try(($els) => {
  return $els.toArray().map(el => el.innerText)
})
.should(headers => {
  const sortedHeaders = headers.concat().sort()

  expect(headers).deep.eq(sortedHeaders)
})

without a cypress parent command:

cy.try(() => {
  const h2s = cy.state('document').querySelectorAll('h2')
  return Array.from(h2s).map(el => el.innerText)
}).should(headers => {
  const sortedHeaders = headers.concat().sort()
  expect(headers).deep.eq(sortedHeaders)
})

I’ve been looking at this for a bit now and digging into the source code gave me the command verifyUpcomingAssertions. That’s how you can wait for a should to be resolved.

However, this command is not documented and can, therefore, be a bit iffy to work with at the moment.

The way I was able to figure out how to use it was again by looking to the source of the default commands found here: https://github.com/cypress-io/cypress/tree/develop/packages/driver/src/cy/commands

When you get it working it works great, but expect to spend a lot of time tinkering. It helps to use the same basic structure that’s used in default commands.

I’m at a point where I got it working perfectly except for the log.end() of the first upcoming assertion, the steps remains ‘pending’ (blue with spinner) when it should be ‘passed’ (green).

edit: I got it working now! The basic format should look something like the following. I left out logging, options etc for clarity.

Cypress.Commands.add('aThing', (element, options={}) => {
    /**
     * This function is recursively called untill the timeout passes or the upcomming
     * assertion passes. Keep this function as fast as possible.
     *
     * @return {Promise}
     */
    function resolveAThing() {
        // Resolve a thing
        const aThing = $(element).attr('aThing');

        // Test the upcomming assertion where aThing is the value used to assert.
        return cy.verifyUpcomingAssertions(aThing, options, {
            // When the upcoming assertion failes first onFail is called
            // onFail: () => {},
            // When onFail resolves onRetry is called
            onRetry: resolveAThing,
        });
    }

    return resolveAThing();
});

This was release in 12.0.0

Hello! I just wanted to let you all know that we’re adding this in Cypress 12.0.0. It turns out that “ability to retry commands” was pretty central to resolving #7306 , and as part of that effort, we’re exposing the addQuery interface publicly.

This will be going out in a couple of weeks with Cypress 12, but if you want a bit of a preview, here’s the PR we have open with cypress-testing-library to update them to use the new API: https://github.com/testing-library/cypress-testing-library/pull/238/files

As an easier way to get started, here’s a preview of the API docs: https://deploy-preview-4835--cypress-docs.netlify.app/api/cypress-api/custom-queries, and the re-written guide on retry-ability that discusses queries vs. other commands: https://deploy-preview-4835--cypress-docs.netlify.app/guides/core-concepts/retry-ability#Commands-Queries-and-Assertions

The docs are still in review, but I’d welcome any comments or questions on them if people want to read it / try out the pre-release builds of Cypress 12 (latest as of now: https://github.com/cypress-io/cypress/commit/b9d053e46777683a9c502a5e970ef3ae26c563d5#comments).

Behold, the birth of the module cypress-commands!

I’ve made the extension on the then command and published it in a repo (and on npm) where I will add more commands in the future. I have a few commands laying on the shelf I could add.

For more details see the repo at https://github.com/Lakitna/cypress-commands

Cypress 12.0.0 is going out today, which should address these needs. If there’s still anything lacking once you take a look at Cypress.Commands.addQuery, please feel free to open a new issue and we can start some new discussions and look at improvements!

Hey @Lakitna, can you add your commands library to our docs on the plugins page?

Thank you @Lakitna 🙏 An(other) example using cy.verifyUpcomingAssertions was just what I needed. I think cy.retry is a reserved command though (at least in v3.2.0) because if I try to add it, unrelated parts of my code start blowing up.

But I tinkered a bit, got it working & was able to remove the race condition (ie a cy.wait) I was relying on earlier. Hope the following example is useful for others who are in a similar situation:

// cy.resolve(fn).should(blah) will re-run the promise-returning fn until
// the value it resolves to passes the assertion
Cypress.Commands.add('resolve', { prevSubject: 'optional' }, (subject, fn, opts={}) => {
  const resolve = () => {
    fn(subject).then(res => cy.verifyUpcomingAssertions(res, opts, { onRetry: resolve }))
  }
  return resolve();
});

// an example function that returns a Cypress.Promise
const getBalance = () => {
  return cy.wrap(new Cypress.Promise((resolve, reject) => {
    cy.get('h1').children('span').invoke('text').then(whole => {
      cy.get('h3').children('span').invoke('text').then(fraction => {
        cy.log(`Got balance: ${whole}${fraction}`)
        resolve(`${whole}${fraction}`)
      })
    })
  }))
}

describe('Test', () => {
  it(`Should wait until balance is non-zero`, () => {
    cy.resolve(getBalance).should('not.contain', '0.00')
  })
})

You are correct, retry is an existing, undocumented cy command (see screenshot).

I would personally fall back to thentry until I made a way to overwrite then with a retry option.

What’s the status on this? I’ve got a two-part cy.get that I’d like to retry until it resolves to a non-zero value. My first attempt:

const getBalance = () => {
  return cy.wrap(new Cypress.Promise((resolve, reject) => {
    cy.get('h1').children('span').invoke('text').then(whole => {
      cy.get('h3').children('span').invoke('text').then(fraction => {
        cy.log(`Got balance: ${whole}${fraction}`)
        resolve(`${whole}${fraction}`)
      })
    })
  }))
}

describe('Test', () => {
  it(`Should wait until balance is non-zero`, () => {
    // cy.wait(2000)
    getBalance().should('not.equal', '0.00')
  })
})

The above runs getBalance() once and then waits on the return value to pass the assertion. It fails via timeout even though my UI eventually has a non-zero balance because getBalance() isn’t rerun when it’s return value fails the assertion.

I can get the above test to pass by uncommenting the cy.wait(2000) but I’d rather not introduce any potential race conditions.

My second attempt, inspired by the conversation between @Lakitna and @Bkucera above.

Cypress.Commands.addAll({ prevSubject: 'optional' }, {
  retry: (subject, fn) => {
    return cy.wrap({ retry: () => fn(subject) }, { log: false }).invoke('retry')
  },
})

describe('Test', () => {
  cy.retry(getBalance).should('not.equal', '0.00')
})

The above errors out with CypressError: Cypress detected that you returned a promise from a command while also invoking one or more cy commands in that promise. and I’ve tinkered with it for a while but can’t figure out how to avoid this error while still testing what I want to test…

The custom queries we are trying to support aren’t wrappers around jQuery attribute selectors; some of them use multiple DOM traversals to match labels to inputs, for example. cy.get isn’t a workaround unfortunately. See linked issues.

This workaround works:

cy.getByTestId = (id) => cy.get(`[data-test-id="${id}"]`);