We’re still working through what changed with Electron 12, but I found our first sign of real progress today. It appears that something with screencasting has changed with Electron 12/Chrome 89. Please note that we use the screencasting API for start/end screenshots, so this runs regardless of whether you are recording videos.

Numbers

All numbers are from the upgrade commit, b52ac98, with the Electron version swapped.

Numbers after the first break are the averages. Numbers after the second break are the percent differences of the averages.

Standard

Electron 11.3.0

real	user	sys
150.59	73.17	11.40
151.56	73.80	11.38
153.74	74.26	11.94
--------	--------	--------
151.96	73.74	11.57

Electron 12.0.0

real	user	sys
223.73	163.98	46.80
225.18	164.95	47.62
233.01	170.77	49.31
--------	--------	--------
227.31	166.56	47.91
--------	--------	--------
39.73%	77.25%	122.19%

Removed _maybeRecordVideo

Removed from packages/server/lib/browsers/electron.js

Electron 11.3.0

real	user	sys
162.45	59.69	6.99
162.62	59.45	7.33
165.31	59.27	7.26
--------	--------	-------
163.46	59.47	7.19

Electron 12.0.0

real	user	sys
165.49	64.30	9.72
165.36	66.39	9.84
165.12	65.75	10.18
--------	--------	--------
165.32	65.48	9.91
--------	--------	--------
1.13%	9.62%	31.81%

So in all we still see a slowdown without the screencasting calls, but it’s much less of a slowdown overall, which is a start. These numbers are also replicated on Chrome 89, by removing the same screencasting calls.

Anyone tried the latest 7.1.0 version?

Just for clarity, 7.1.0 does not include any changes we expect would influence performance.

Ha, I bet that’s what you said about version 7.0.0 😆

I apologize for the delay. If you haven’t seen my testing in https://github.com/cypress-io/cypress/issues/15779#issuecomment-815269145, take a look. After a bunch of playing with it, I arrived at the following system for automatically bisecting, assuming we’re looking for differences in Chrome/Electron CPU time. This gist contains the final run scripts.

This was run on an AWS t2.xlarge (this is important), configured in the following way:

wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo apt install ./google-chrome-stable_current_amd64.deb
sudo apt install build-essential unzip

git clone https://github.com/cypress-io/cypress-example-kitchensink.git
cd cypress-example-kitchensink
yarn

cd ../
git clone https://github.com/cypress-io/cypress.git

The process was initiated via:

git bisect start 0399a6e58e226b860a41acf2847ee43adb8ad41e 73317218230319d45689bbad3ce46e7ab2312e18
git bisect run ./bisect.sh

considering 0399a6e58e226b860a41acf2847ee43adb8ad41e (tag 7.0.0) as known bad and 73317218230319d45689bbad3ce46e7ab2312e18 (tag 6.8.0) as known good.

This process turned up b52ac98a6944bc831221ccb730f89c6cc92a4573 as the first bad commit. This commit is notable as it both updates to a new major version of Electron (12.0, from 11.3.0) and a new major version of Node (14.16.0, from 12.18.3). However, it doesn’t seem likely that upgrading Electron would change Chrome performance whatsoever.

Commit	Chrome real	Chrome user	Chrome sys	Electron real	Electron user	Electron sys
3fc5f03 (good)	119.90	101.23	25.86	111.85	49.45	9.13
f6a5d1e (bad)	112.19	100.00	25.46	107.51	75.89	21.13
e76edd7 (good)	102.69	36.74	5.55	100.84	39.95	6.44
f31013d (bad)	109.51	96. 38	24.04	106.62	75.27	20.78
7e9da23 (good)	112.28	92.88	24.48	107.10	48.36	9.24
bc4d5ea (good)	108.89	91.79	23.99	103.88	47.09	9.34
b52ac98 (bad)	109.05	96.30	24.99	110.26	77.43	21.02
c449775 (good)	109.71	93.54	25.01	103.92	48.06	8.87

Looking at the numbers, that does indeed seem to be the case, as Chrome numbers did not change significantly other than one run being significantly faster than the others. This also points to another potential problem. As seen in https://github.com/cypress-io/cypress/issues/15779#issuecomment-815269145, both Chrome and Electron were noted to slow down with each version upgrade.

1. I speculate that Electron (the default for cypress run, which is what will be used in CI) has been slowed at least by the Electron upgrades, which is the primary cause of the visibility in #15779 and #15853. This problem is exacerbated on limited CI machines.
2. Chrome has a more niche slowdown, specifically correlated with strength of the machine. On ample resources, Chrome runs fine, but when constrained, the differences between 6.6, 6.7, 6.8, and 7.0 slow it down in some way.

TLDR: We will be looking into the Electron upgrade performance, and I am going to re-bisect on a slower AWS instance to compare the results. The issue with Chrome performance has not yet been identified.

The code for this is done in cypress-io/cypress#16113, but has yet to be released. We’ll update this issue and reference the changelog when it’s released.

I can’t see any obvious regression in the times it takes to run the cypress tests themselves - but I don’t think that is uncommon. Cypress uses itself for testing, but all of their tests are fast because they test on tiny pages with very little js and additional resources. Where as real-world cypress usage is on pages that are resource heavy.

My own app is a very heavy SPA with 3mb of javascript and we get a 7x speed regression on low spec CI machines and no regression on fast multi-core developer machines.

Am trying to narrow down the commit in 7.0 that fails… but it take a long time for me as our process is quite slow. Will continue tomorrow.

https://github.com/cypress-io/cypress/commit/6f9e80a952b1a980b563ed42a310de32025bfe5c - slow - bad

https://github.com/cypress-io/cypress/commit/232bd873685f1f3277004173a177bc76ece0e6e5 - ok - good

@GC-Mark we did, same result and time as 7.0 and 7.0.1

Anyone tried the latest 7.1.0 version?

Just for clarity, 7.1.0 does not include any changes we expect would influence performance.

Yeah, I can confirm this as well - our tests are much slower under 7.0. In our case, it’s closer to 50% slower, although we have much longer tests overall. This is only happening in CI (Github Actions) for us, local runs cypress run have no measurable difference. Rolling back to 6.9.1 fixes the slowness for us.

@mmonteiroc https://github.com/cypress-io/cypress/issues/15853#issuecomment-824229219 2 bugs - 1. Slowdown with video 2. Always doing video

Hey ! This issue is non-related to the video recording… At our side we never recorded videos with cypress and we encountered this issue when upgrading to 7.1.0 among other issues that we opened… @flotwig

Do you have any instructions of how to generate a metric like this using Cypress?

Hey I don’t want to pollute this thread with offtopic stuff but the TLDR is to pass values from cypress (eg: elapsed time) to a statsd server (search statsd-client) which then feeds to a prometheus data source in grafana

Just throwing my hat in the ring too - we’re also noticing a major performance and reliability hit with 7.x when upgrading from 6.6.0.

Below is a site performance metric I use cypress to capture - this particular metric is the time from test execution start until an essentially static page is rendered, so most of it is cypress downloading react framework javascript files and loading them into the browser (the whole test is basically just a cy.visit()). I use this test as a bellweather/coal mine canary so if other metrics show wild results and this one does too then we know it could be something infrastructure related rather than our product. Interestingly, this simple test really highlights the performance hit that 7.x has caused.

On 6.6.0, before the first annotation, the average was 3.6 seconds. When I upgraded to 7.0.1, you can clearly see the page loads skyrocket, averaging around 13 seconds or almost 4x slower, and regularly timing out (16.7sec average if the substantial amount of timeouts are included). Other tests in our suite are now flaky as hell because simple visit requests are regularly timing out before the load event even starts (even at 60s!).

For reference to the other stuff in the graph, the second annotation is an upgrade to 7.1.0, showing no improvement. The last annotation is a downgrade back to 6.6.0 today, and the page load times immediately dropped to what they were previously.

I can confirm this issue. We had to downgrade to 6.6.0 since our tests in Jenkins slowed down considerably.

Same, jumped from 6.2.0 to 7.1.0, all suites take double the time to finish

Same here switching from version 6.8.0 to 7.1.0 directly

I confirm that, too. Tests with both 7.0.1 and 7.1.0 take at least double the time as compared to 6.9.1.

Not sure how this relates, but noticed extremely high CPU usage for my Mac, when running test from local terminal, on 7.1.0 version. But I gotta double check other apps in Mac. There wasn’t such high CPU usage thing in earlier versions. Not sure now.

We don’t use intercept or any stubbing and still have a big slowdown

I believe https://github.com/cypress-io/cypress/issues/15841 might play a role here. Depending on the testing setup (locally/with backend API running/CI) those incorrect network requests might require a timeout, which might be the cause of the slowdown.

We are observing the same thing. On my workstation it’s as fast as ever, but on our CI machine (with fewer resources) it’s taking 3x as long. Maybe it’s using more memory?