cypress: run-ct returns with exit code 0 when tests fail - breaking CI detection of failed tests

Current behavior

This is a pretty big bug. All CI jobs running with run-ct will report success even if tests fail due to this bug.

In this screenshot, you are seeing the output of yarn cypress:component:cli --spec=packages/SearchInput/src/component_tests/ClearButton.test.tsx which runs this command:

    "cypress:component:cli": "cypress run-ct --project integration-tests",

Note that at the bottom, we see Done in 69.93s but no error. This Done is reported by yarn. When a test fails, it should look more like this (I added expect(false).toBe(true) to a jest test to simulate a failure):

cypress.json:

{
  "baseUrl": "http://localhost:6009",
  "retries": {
    "runMode": 2,
    "openMode": 0
  },
  "integrationFolder": "../packages",
  "testFiles": "**/integration_tests/*.test.tsx",
  "component": {
    "componentFolder": "../packages",
    "testFiles": "**/component_tests/*.test.tsx"
  }
}

Desired behavior

when a test fails, the shell error code should be non-zero. cypress run does this correctly.

Test code to reproduce

write a failing component test
run it with cypress run-ct
watch to see that the exit code is 0

Cypress Version

8.3.0

Other

tested with 7.6.0 as well, upgraded to be sure it was not already fixed.

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 1
Comments: 31 (12 by maintainers)

Most upvoted comments

I wonder if somewhere in the release and test pipeline a tool like https://github.com/bahmutov/cypress-expect could be used to verify the failing tests and the exit code

bahmutov on Aug 25, 2021

excellent, thank you, I’ll report back shortly, probably a few hours once these meetings are dun

cellog on Aug 25, 2021

Can you run the DEBUG logs for the cypress:cli during cypress run? This will print the exact exit code that Cypress is issuing at the end of the tests. Please provide the entire set of logs. A reproducible example would be helpful also.

DEBUG=cypress:cli* cypress run

DEBUG=cypress:cli* cypress run-ct

jennifer-shehane on Aug 25, 2021

@cellog your post says

“cypress:cli”: “cypress run --project integration-tests”,

Should this be "cypress:cli": "cypress run-ct --project integration-tests"? Assuming yes, since your issue is about run-ct.

lmiller1990 on Aug 25, 2021

tagging @JessicaSachs because you responded to a component test 5 days ago as well, hope you can route it to the right person!

cellog on Aug 23, 2021