cypress: "run" is much slower than "open"

This seems like a bug - the headless version of the tests run 4x - 5x slower than the headed ones. (cypress run is about 4x-5x slower than the same tests run via cypress open)

Current behavior:

I have a test suite with 77 tests in it currently.

In local dev I have been using the open command, per the docs. The GUI is awesome, thank you for this. The ability to select one tests suite at a time to test on the fly is great. And when I run all tests at once, they run fun fairly quick. All 77 of them run in about 1:15. This is acceptable, especially considering all of the debugging tools that are available in this mode. But cypress open isn’t the problem.

We’re working on getting a CI in place. In preparation for this, I ran the tests via cypress run as this is how I would expect our CI server to operate. I was (honestly) shocked to find that they were running 4x-5x slower in the headless electron browser. Tests routinely take at least 4:10 to complete in this mode. I’ve dug through other issues, google, docs, everything I could find and I have not found a way to speed this up (aside from running in parallel… which I don’t feel addresses the real problem here).

Desired behavior:

Tests run at last as fast vis cypress run as they do for cypress open.

Steps to reproduce: (app code and test code)

…This may be a little tough. The bug is not a specific ‘this blew up’ but more a ‘this really seems like wrong behavior, at the top level’. But if you feel that I can provide something specific, please ask.

Versions

Cypress 3.1.3 Node 10.12.0 Yarn 1.9.4 MacOS X High Sierra Chrom 71 (cypress open executing in this browser)

config:

{
  "integrationFolder": "src/app/components/_pages/", // we keep tests with their corresponding component
  "testFiles": "**/*.int.spec.js", // as a result, we had to easily denote cypress tests
  "baseUrl": "http://localhost:3000/",
  "video": false,
  "chromeWebSecurity": false
}

Thank you!

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 19
Comments: 38 (10 by maintainers)

Most upvoted comments

@tarponjargon @lugus @sergiomap Switching to Chrome solved this issue for me. Chrome runs about 4x faster than Electron on my machine.

cypress run --headless --browser chrome

blumk on Dec 14, 2019

After digging into some source codes of cypress, could we introduce an env like autoRunAll for open, and check this env at specs.jsx file when it gui ready, If it’s true, then auto trigger run all Specs.

I thinks this can be an option to reduce run time on CI caused by browser relaunching.

maple-leaf on Apr 11, 2019

Cypress run is much slower than cypress open. Is anyone looking into this? Testcafe doesn’t have this problem.

devdelpiero on Mar 8, 2019

Same for us, much more slower, resulting some tests are not passing with cypress run.

lugus on Sep 8, 2019

cypress run --headed

This makes it ok for me. It does prompt electron but it automatically closes when the tests are finished. Tests drop from 50 to 1 second even with the video recording.

Problem is possibly on not getting the native video.

guilhermetod on Aug 11, 2020

Yes, we’re being quite lazy 😉 I will try to put together a test repo

tarponjargon on Sep 10, 2019

I’m experiencing the same issue, cypress open is way much faster than cypress run. If I’m lucky, the speed is decreased by 2. However, most of the time it ends up with timeout. Actually, I can reproduce the issue with cypress open by starting a screen record when the tests are running. I don’t know why but the HTTP requests take more time.

BTW, I’m using a MacBook Pro with macOS 10.14.5.

getkode on Jun 14, 2019

@ShawnMercado can you give us a reproduction and open a new issue? We’re currently investigating various performance issues.

Wish I had time, maybe in a couple weeks. What I’m working on is complicated and under NDA so I’d have to build a full-but-vanilla version to have a good chance of a meaningful reproduction.

Sorry I know that’s not helpful 😕

ShawnMercado on Aug 19, 2021

@JZebra No, we run all specfiles separately during cypress run with the support/index.js file running before each file in order to enable the use of parallelization.

We anticipate removing the ‘Run all tests’ functionality for several reasons, so recommend moving your workflow to the way cypress run runs. https://github.com/cypress-io/cypress/issues/1586

jennifer-shehane on Feb 19, 2020

Right now for me each cypress it() function takes ~40 seconds to complete when running on our CI pipeline (as opposed to ~2 when running local). I think the vms just need to be beefier, but I don’t know what specs to use. Can anyone suggest specs for vms running in CI for using electron?

MattMcFarland on Feb 14, 2020

@cake-jnorman @fr0 @devdelpiero @andreasmihm

Can you confirm if you are comparing the 2 situations below?

cypress run
cypress open then clicking ‘Run All Tests’

Explanation of why it is slower (this is expected behavior)

cypress run relaunches the browser in between each spec file while ‘Run All Tests’ button does not. There is an issue to address this discrepancy here: https://github.com/cypress-io/cypress/issues/1586 This would account for the difference in running time.

If this is the case, then we will close this issue and suggest you add your thoughts and 👍 to this issue: https://github.com/cypress-io/cypress/issues/2951

jennifer-shehane on Jul 11, 2019

same issue here, my test suite runs (both on local machine):

headless -config video=false : 02:21 -headed (with Electron) : 01:35

andreasmihm on Mar 8, 2019