cypress: CT Performance Improvements: v10 slower than v9
Edit from @lmiller1990: this is getting worked on: https://github.com/cypress-io/cypress/pull/23104. Component Testing performance will be improved, initially released under an experimental flag to let use continue making incrementally performance updates. See latest comments for latest updates.
Edit: Try using the new experimentalSingleTabRunMode flag in your component property in cypress.config. It is up to 40% faster for larger suites. Read more in the docs here.
Current behavior
After upgrading to the latest version, we have an issue with slowness when we run cypress run --component. The tests themselves are not slow, but it seems there is some delay between specs runs as you can see in this video 👇 .
https://user-images.githubusercontent.com/6830426/174029644-aa31da35-ee35-4a04-8474-b0eab9a72869.mp4
We were not able to detect the exact issue, but we would appreciate any help 🙏
I was trying to run the tests with DEBUG and this might be potentially the issue 👇
Desired behavior
The tests should run smoothly without any delay.
Test code to reproduce
We use cypress on public repo https://github.com/toptal/picasso.
We have cypress 10 on the current master.
yarn
yarn test:integration
for debugging it is faster to run
# these need to be run only once
yarn
yarn build:package
yarn test:setup cypress run --component
Cypress Version
10.1.0
Other
We use cypress with Happo for visual testing. The first idea was that it might be the problem since Happo didn’t adapt to v10 yet, but it is not the case, I have tried to remove it from the repo and it didn’t help.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 10
- Comments: 39 (23 by maintainers)
Summary
Ok, I looked into it. It turns out creating a new tab (and associated processes) does impact the performance. Here’s a branch with the 9.7.0 behavior, which is one tab -> run all specs in there, as opposed to the current implementation which is to create a new tab per spec.
There’s a build of this branch you can install for linux x64, get it here. There’s likely some bugs, but
npx cypress run --componentseems to work fine - if you are running linux, you could give it a try. I’ll get a more reliable build up for windows/mac soon.Note: these numbers are from 1 time runs - not the most scientific. To be more accurate, I’d run them all many times and take the average, but these are good enough to validate that opening a new tab is the primary reason we are slower.
Toy repo
For the run time: (+18% Cypress 9.7.0, +37% Cypress 10.3.0)
These numbers are a bit exaggerated, since I’m running 1 test per spec, which is not common. Normally, a single spec file will have 10s or 100s of tests. Either way, this experiment implies a ~300ms cost for creating a new tab, which is significant.
picasso repo https://github.com/toptal/picasso (run locally)
It’s 43% faster! I guess this mostly accounts for the 2x slowdown described.
Cypress internal component tests (local, no parallel, no video)
Run locally. I turned off video, which makes everything slower in a linear fashion, so it shouldn’t impact the relative differences. 30% is a solid improvement.
Discussion
I think it’s death by many a thousand paper cuts - just opening a new tab is not a single performance killer, but it looks like we do some cleanup between specs, and reset a bunch of things. I need to go deeper, but I suspect at least some of these things are absolutely necessary to guarantee spec isolation and determinism in E2E, which is why E2E has done the “new browser per spec” things since forever. I’ll look more into this, and if it’s necessary for component testing.
As shown on my toy repo with 100 of the space spec, as as predicted, the actual test execution time is identical, since your production code and specs hasn’t changed changed.
Next Steps
Thanks for the patience and assistance here everyone! I think we are on the right track. I’ll be focused on this until we have the performance issue solved.
There is definitely a slowdown here. Running your
picassotests, I see that:yarn test:setup cypress run --componentyarn test:setup cypress run-ctIt appears that with 10.x we do some cleanup work/reset some browser state between spec executions that is the bulk of the added time. This time increases linearly with the number of specs included with a given run. The debug log output from the 10.1.0 run is provided below.
Thanks again for the report. I’m going to direct this issue to the CT team to investigate mitigating factors.
10.1.0 debug output between spec run start and actually executing tests in the spec (click me)
Hi team, it’s finally merged, it’ll be in 10.6. Finally! Excited for this feature.
Surprise off cycle release… 10.6 is now out 🎉
@lmiller1990 I have tested it out in Picasso and it works perfectly! 🚀 🎉
I really enjoyed following this issue and checking how you are handling it, especially since you share your thought process when debugging!
Looking forward to this being released.
Thank you for your work! Keep rocking! 💪 🙇
Another update - this is getting close, binaries available for testing: https://github.com/cypress-io/cypress/commit/293a0f489bbbeae9ecc6d76053fa24eef77fc2de#comments
To use it, you need to enable it. It’s initially going out as an experimental feature. Example.
Hello folks, an update - trying to get this through: https://github.com/cypress-io/cypress/pull/23104. It’s awaiting internal review. It will likely have some edge cases on initial release - thus the experimental flag, but exactly what those limitations will be will clear when it lands.
We are facing a similar problem with component testing.
In our case we have a monorepo with a structure like:
custom-ui-framework. app-a (depends on custom-ui-framework). app-b (depends on custom-ui-framework). We are using:
cypress 10.3.1 + @cypress/vue2 1.0.1 vue 2.6.14. vite 2.9.14. We can notice that when running component tests on any package, all the components from the custom-ui-framework are being fetched, no matter if they are being used or not by the tests.
I attach the cypress and vite config just in case it is useful to you.
cypress.config.ts
vite.config.ts
Hello all, it looks like doing a significant change to how we run specs is going to be a more longer term project that ties in with the “Run All Specs” feature discussion. This project touches similar issues (how to run many specs quickly).
For a short term fix, it’s been suggested we have an opt in flag to replicate the functionality of CT (Component Testing) in 9.x, which is to run all specs in a single tab, sequentially. This would come with the caveats that are listed in the first post of Run All Specs. This would be for CT only - E2E never worked like this.
As for how to opt into this, It would likely be listed as experimental, so you’d opt into it - so something like
experimentalComponentSingleSession. This will also allow for future experimentation, like usinglocation.reloador an alternative strategy, such as those listed above.I will post more updates as they come. Thanks for the patience everyone!
@TomasSlama thank you for the report, and thanks all for the corroborating comments. I have the
picassotests up and running and will report back my findings.Ran the Darwin Arm64 build 3x on our two projects with and without the
experimentalSingleTabRunModeflag. They saw an average 41% and 51% improvement to the total test duration times 🎉Hi @lmiller1990! I just checked and when I set
supportFile: falseit does not import everything.I just investigated a bit more and the root cause of the problem was that in our
support/component.tsfile, in our custommountdefinition we were registering the router as in the documentation. As a result, all of the dependencies of all our views in the app where being fetched onmount.Since for the component testing we don’t need the real routes (at least not most of the times) we have just changed it to use a mock router by default, e.g.:
Thanks for your input!
The pre-release builds are only available for a few days. Maybe it’s expired. I can make a new one if needed. At this point, it’s pretty clear what’s slowing us down, though, I don’t see a ton of value in getting more testing in at this point.
Making these changes isn’t super easy, since there’s a lot of complexity here, but I’m bringing this up internally as a priority. I’ll post updates here w.r.t what we are doing, when it can expect to land, etc.
Re: ARM builds - yes they absolutely fly, although I am still on Intel, I’d like to make Intel fly, too.
@TomasSlama cool, thanks for the support and help testing!!
@wopian can you confirm your table info a bit? When you say 10.3.0 you are referring to my build (not the live 10.3.0 on npm?)
Those are some huge improvements, good to see!
Just to clarify a bit more here - unfortunately it’s not as simple as “just ship these changes”. Opening a new tab clears a lot of things and better isolates the specs, which is a feature of any good test runner. The original goal of this change was to be more in line with E2E - you can image it’d be confusing if CT and E2E diverge. I will find out exactly what we do better each spec and find out how to do it in a faster way than what we do now.
My next step is for me to dig deeper and isolate more closely what’s really killing performance - I think it’s more than just “creating a new tab”, but also things like teardown/setup/rendering the runner UI.
The end goal is to improve performance for both CT and E2E. The spec execution flow is at the core of Cypress. It needs a bit more investigation, but we are on the right track with this thread. We take this kind of regression (performance, and more importantly determinism and reliability) very seriously. Just to clarify, this is a performance regression for Component Testing - this change made E2E faster, since it use to spin up a new browser for each spec, now it just does a new tab.
I am out next week, but after that I’m going to look into a more detailed technical breakdown, then we can actually start moving forward with some updates. The Component Testing “alpha” label was there for reason, and I’d like to move out of “beta” sooner than than later, but that means exercising great caution around mission critical things like performance and spec isolation.
I will continue posting my updates here, I think it’s fine to take a little time as long as we are clear with communication. This is still my main focus and will be until it’s we are back to a comparable, if not faster, state than the 9.x Component Testing alpha.
@lmiller1990 I have tested on Picasso and the results are from 112s 👉 62s. Great news 🎉.
Thank you very much for your effort 🙇
@mverdaguer what does your
support/component.tslook like? Vite should only fetch dependencies that are imported in some fashion. We are just a thin layer on top of Vite - although it’s definitely possible we’ve got a bug. I don’t see any obvious problem in your configuration.If you do
supportFile: false, does that stop the fetching? Just one idea to help debug. I suspect your issue isn’t directly related - are you experience this only after going from v9 -> v10? If so, it could be - if not, I’d recommend a new issue titled something like “Vite loading unrelated dependencies”, since it sounds like that’s closer to what you are experiencing.Yes this issue is just for component tests.
The Vite Dev Server integration is only used for Component Tests, though - E2E does not use a dev server, so I don’t think Vite has anything to do with your problem.
Definitely worth opening a new issue, try to include:
cypressfiles (likecypress.json,cypress/plugins/index.jsInteresting. New tab seems to be a no-go, but
location.reload()(or some other single tab strategy) looks positive. I will link the repo and numbers in a moment.The second benchmark doesn’t reload, so it’s not really an option. The third,
location.reload()appears about 6x faster. Link.Since we need to change the href, too, we might do
document.location = .....?spec=newSpecfor a similar behavior.Basic summary of above:
Some specific exploration to avoid re-parsing
cypress_runner.js, which takes a non trivial amount of time, was was conducted. Note that this was done in isolation to the runner - a minimal web page that does nothing but fetch and parsecypress_runner.jsover and over was used for the benchmarks below. You can see the results here and the basic code is here. The README has instructions on running the benchmarks yourself.document.location.reloadordocument.location.href = <next_spec_url>in the future. This might be faster than the current approach, since the browser does not re-parsecypress_runner.jsin the same tab, but it does re-parse it when opening a new tab.Here are some basic numbers:
document.reload()