github-action: When tests fail during `github-action@v3` the step does not emit failure exit code

I’m executing parallelized tests recorded to the Cypress Dashboard using github-action@v3 Functionally everything works perfectly and as expected with the single exception that when tests fail the github-action@v3 step in my workflow appear with a grey checkmark instead of the expected red X. Thus that workflow run appears to have passed with a green checkmark when reviewing my Github workflow run history.

WORKS:

  • The Cypress test are successfully executed in parallel
  • Cypress successfully adds PR comments when the test execution has completed
  • The correct pass/fail results are present in the PR comment
  • The PR comment links to the correct Cypress Dashboard run
  • github-action@v3 logs all have the correct pass/fail results
  • github-action@v3 logs all link to the correct Cypress Dashboard run

DOES NOT WORK:

  • github-action@v3 step looks like it passed even though it should have failed because there are failed tests

Workflow run showing cypress (chrome, 6) with a green checkmark, and github-action@v3 with a checkmark.

Screen Shot 2022-07-20 at 8 02 13 PM

Expanding github-action@v3 and it clearly includes failed tests but does not include a failure exit for the step.

Screen Shot 2022-07-20 at 7 52 20 PM

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (6 by maintainers)

Most upvoted comments

I am using cypress/included:12.0.0.

I’m running this command from the Dockerfile after I install some packages I need on the box:

CMD cypress run --browser chrome --env api_key=${API_KEY}

The command I’m running in Github Actions is: docker compose up --exit-code-from cypress

@admah Sorry for the delay, I’m confirming the way we were using Cypress Module API was the root cause of my issues.

Unsure how to make this better, but some comments:

  • The documentation exists and provides all of the code necessary to avoid this issue 🥇
  • But I obviously didn’t realize the impact in relation to my CI jobs
  • Maybe call out how CI jobs require handling of results to correctly fail so people like me have less of a chance to fall into this scenario?