cypress: Assertion failure in async test case does not fail test overall

What is the point of having testing tool that doesn’t even assert properly? This should have P1

Can someone guide me to related code?

I believe most of these failures are due to the assertion failing AFTER the test has successfully finished, read https://www.cypress.io/blog/2020/01/16/when-can-the-test-stop/

Very helpful blog post @bahmutov! This approach solved my issue:

it('waits for window confirm to happen using variable', () => {
  cy.visit('index.html')
  let called

  cy.on('window:confirm', (message) => {
    expect(message).to.equal('Are you sure?')
    called = true
  })

  cy.get('#click').click()
  // test automatically waits for the variable "called"
  cy.wrap(null).should(() => {
    expect(called).to.be.true
  })
})

The same story here.

@jennifer-shehane btw why we should not ping you guys? This is a major issue inside Cypress, caused a lot of people. So that’s the only way to say you “Hey, we’re still here and need help”. As a result, make some impact on reordering among your roadmap prioritizations. Based on how much folks ask for. 😦

Otherwise, it stuck in your backlog for a while…

Thanks.

A workaround would be to just not use async test cases.

So this issue is due a bug where Cypress doesn’t support async/await tests bodies or tests that return promises. This wouldn’t be too hard to fix, but we need to prioritize it. We’ll need to rangle all the issues that are affected by this

Please refrain from commenting asking for updates 🙏

This issue is still in the ‘ready for work’ stage, which means no work has been done on this issue as of today, so we do not have an estimate on when this will be delivered.

If there are any updates the status label on the issue will change, a linked PR will be opened, or a comment will be made by someone on the Cypress team. Otherwise, there are no updates.

@ryan-snyder Was able to solve my problem with a cy.wrap() … looks something like this

cy.wrap(
   AWS.listAttachedRolePolicies('Admin').then((list) => {
         expect(5).to.be.null;
   })
);

I believe most of these failures are due to the assertion failing AFTER the test has successfully finished, read https://www.cypress.io/blog/2020/01/16/when-can-the-test-stop/

Not sure if related but this test is also failling with native promises mixed with cypress promises with this code:

it('It fails on promise but it passes', () => {
  new Cypress.Promise((resolve, reject) => {
    Promise.reject(new Error('Error from native promise')).catch(err => {
      reject(err);
      expect(true).to.be.false;
    });
  });
});

It produces:

it also passes while failing on:

it('It fails on promise but it passes', () => {
  new Cypress.Promise((resolve, reject) => {
    Promise.reject(new Error('Error from native promise')).catch(err => {
      reject(err);
    });
  }).catch(err => {
    expect(true).to.be.false;
  });
});

@Bkucera and I have been over this a bunch of times before and it’s something on our end that we need to fix. It has to do with the interop between promises being returned to the test, and Cypress also knowing that commands were enqueued.

We can generally always figure out how to do the right thing, but it will involve doing things like adding a .then() and additionally monkey patching Promise.prototype.then to know when the promise chain is settled, and to know whether or not cypress commands were enqueued within each .then() block.

We need to do things like intelligently throw if you’ve enqueued cypress commands inside of a native promise .then() but not returned cy, or we need to automatically figure this out and prevent the .then() from resolving until the cypress command chain has resolved inside of there. I’m in favor of the latter.

@ryan-snyder its a cypress issue. Mocha supports this, but we do some nasty hacks on top of mocha and likely didn’t wire something up properly, such as awaiting a promise if it’s returned.

Temperorary workaround for me is to wrap the promise in a Cypress command. Note that errors in catch will just time out the test that calls this command. In my case, in my commands js:

Cypress.Commands.add('resetSomeData', function() {
    return new Cypress.Promise((resolve, reject) => {
      graphqlFetchObject
        .request("mutation { doSomething }")
        .catch(function(err) {
          let didDeleteData = false;
          expect(didDeleteData).to.be.true;
          reject(err);
        })
        .then(function(resp) {
          resolve(resp);
        });
    }).then((resp) => {
        expect(resp.didItHappen).to.be.true;
    });
})

I’m looking into a fix for this issue; it’s required work on the way to https://github.com/cypress-io/cypress/issues/1417.

An example to demonstrate the most basic case:

it('passes', async () => {
  cy.wrap(true).should('eq', false)
})

The promise resolves immediately, but there are still items in the Cypress command queue. We should clearly be waiting for both the test function to resolve and the command queue to be empty.

Notably, it works if you wrap it in cy.then():

it('fails, as it should', () => {
  cy.then(async () => {
    cy.wrap(true).should('eq', false)
    await 1
  })
})

meaning that Cypress has already ‘solved the hard part’ - we know how to do the right thing (because the right thing happens when we wrap our original function, unchanged, in cy.then()), we’re just not doing it when the top-level function is async / returns a promise.

I’ve just confirmed that this is an issue in version 9.5.0. I accidentally left an async on my test function that I didn’t really need and spent much of this afternoon trying to debug why my tests kept passing even when I had obviously broken assertions in them. After finding this I had to go through our existing test cases to fix other async tests that would succeed no matter what.

We are fairly new to Cypress but the fact that this issue has been open for 3.5 years is making me reconsider whether Cypress is reliable enough for us to be using.

This is still a bug in 8.6.0

@Bkucera Sounds good. I’ll probably do a little of poking around on my own just because I want to get more familar with the codebase. If you need me to do anything, ping me

So I’ve started to take a look at this. Was able to get the test not passing but I’m not able to trigger a test failure. With the async function, mocha triggers the test finish before it’s actually finished. Was able to get around that by checking if all the commands are done, but If I do that, mocha never triggers the test finish event at all. Sorry the issue I linked was async describe block. Doesn’t seem to be an issue with an async it block. Still seems to be something with mocha unless I’m missing where cypress triggers the test end event