cypress: Cypress 4.0.1 - Electron is 1.5x slower in a test spec

Current behavior:

In our project, we are updating Cypress from 3.5 to 4.0.1. See PR https://github.com/okTurtles/group-income-simple/pull/829

The same test spec “Large Group” takes 2:29m to run instead of 1:36m. The other specs take the same time. See run 284 - @3.5 vs run 285 - @4.01.

Desired behavior:

All specs should take the same time.

Test code to reproduce

  1. Clone our project and checkout the PR (or master to see v 3.5.0)
  2. Install dependencies npm install
  3. Run the project grunt dev
  4. In another terminal run cypress npm run cy:open
  5. Run the test step “Group Large” on Electron.

The test in the spec taking too long is located at test/cypress/integration/group-large.spec.js. See file here

  it(`A group with ${groupLength} members shows correctly the pledging month overview widget`, () => {
    for (let i = 2; i <= groupLength; i++) {
      // The member joins the group through an invitation...
      cy.giAcceptGroupInvite(invitationLinks.anyone, {
        username: `user${i}-${userId}`,
        groupName,
        bypassUI: true,
        actionBeforeLogout: () => {
          if (i > 3) {
            cy.getByDT('contributionsLink').click()
          }
          cy.giAddRandomIncome()
        }
      })
    }

    cy.giLogin(`user1-${userId}`)
    cy.giAddRandomIncome()
    cy.get('.graph-bar')
      .should('have.length', groupLength)
    cy.giLogout()
  })

If anything else is missing, please let me know.

Versions

  • Cypress 3.5 and 4.0.1
  • Browser Electron 72 and 78 (maybe that’s the reason?)
  • OS Ubuntu 14.04

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 22
  • Comments: 27 (3 by maintainers)

Most upvoted comments

We’re investigating some performance issues related to the rendering of the Command Log such as https://github.com/cypress-io/cypress/issues/6783 where I think the solution to that may also solve this issue.

Since there was never a reproducible example provided in this thread - we will have no way of actually testing if any solution we provide solves these however. But we’ll followup in here when a PR is open for that.

There are some cases where hiding the Command Log does improve performance, and this is what we’re investigating - what in the rendering of the component is causing the slowdown and how we can improve that.

I have now been testing v5.0.0. and I do not see any significant improvement in performance to what has been discussed in this thread… Has anyone have any clue to how could it be possible to have the same performance in all these new versions matching the performance in the old versions that we have been talking about… I would very much like to be able to update to recent version but I cannot compromise the overall performance doing it, any tip will be very much appreciated, thanks!!

Here’s an example of the very same test run on one of our features with Electron, one with v3.4.1 (111.32s) and the other with v5.0.0 (232.50s), notice the different is quite significant:

Cypress 3.4.1

Screenshot 2020-08-20 at 12 31 53

Cypress 5.0.0

Screenshot 2020-08-20 at 12 11 19

Hi @jennifer-shehane, do you know if there are any updates from the Cypress Developers? We are currently executing more than 6000 tests nightly and this would quite significantly slow down our execution.

We would be happy to provide more info information or help fix this topic.

Is there any future plan to fix the performance issue?.. In our organisation we have not updated since 3.4.1 as now it would not make much sense to move to 3.7 which seems to have still a good timings, but cypress 4.7 has just been released and there’s no sign of the performance issue being addressed.

Building off of this thread: https://github.com/cypress-io/cypress/issues/2970#issuecomment-589082328

we tried setting overflow:hidden to the command log only when the test is run in CI (via checking environment variables), and this has yielded large performance benefits when running tests through CI.

const sidebar = window.parent.document.querySelector('.reporter-wrap .container');
if (sidebar) {
    sidebar.setAttribute('style', 'overflow: hidden;');
}

On cypress@3.4.1, our test suite took about 1.75hrs. On cypress@5.1.0 without these changes, the test suite timed out at 3hrs+ On cypress@5.1.0 with these changes, we were back down to 1.5hrs

Of course this isn’t entirely robust as the UI can change in future versions, and it doesn’t necessarily fix the underlying issue, but it enabled us to upgrade cypress from 3.4.1 to 5.1.0 without inheriting the large performance hits described in this thread. One caveat is that in some recordings, it looked like the runner did not continually keep scrolling the command log, but in most instances it did.

For local development, we did something similar, where we added a button to the UI that toggles the command log. Here’s an incomplete snippet, but it should convey the general approach:

        const icon = document.createElement('i');
        icon.setAttribute('class', 'fas fa-list');

        const button = document.createElement('button');
        button.setAttribute('class', 'toggle-command-log');
        button.appendChild(icon);


        button.onclick = toggleCommandLog;
        button.setAttribute('aria-label', 'Toggle Command Log');

        const newCommandLogExpandButton = document.createElement('span');
        newCommandLogExpandButton.appendChild(button);

        commandLogButtons.appendChild(newCommandLogExpandButton);

We tossed both of these snippets into a before hook. Hopefully others find this work-around handy!

Same in our organisation. We’re still running on 3.7.0 because of the performance drop.

We’ve ran same 5 jobs in TeamCity on the same agent for 3.7.0 and 4.7.0 (following Migration Guide) and had following results: comparison

Hi, I tried it with cypress 5.0.0 on macOS Catalina 10.15.2.

Running with Electron 78 headless and Cypress 3.8.3 takes 05:23 minutes Running with Electron 83 headless and Cypress 5.0.0 takes 12:04 minutes

We would like to update the package for quite some time now, any updates here dear Cypress team?

current version has lots of bugs

We have the same issue. However, we noticed the speed tests degradation from version 3.8.2 to 3.8.3 (15 minutes slower). We thought that maybe the speed improves in version 4 but it is not.

Versions

  • Cypress 3.8.2 to 4.0.1
  • Electron 78 (headless)
  • Doker image: cypress/base:10

Hi @jennifer-shehane or any other Cypress developer. Is there any news about this? Seems that more than a few organisations are impacted by this performance issue.

Is there something you can say about when you going to look into this?

Same here. Since we are using Cypress to test an application which controls phone calls, we unfortunately rely on timing (calls only ring for 10 seconds for example). Upgrading to 4+ (even 4.7) brakes half of our existing test.

Our dev machine + CI is also windows based, and tests are running at least 3-5 times slower. On our end it does not matter if we run the tests with Chrome, Firefox or Electron. Tests which took previously 8s to complete, now take ~30s.

Happy to share a debug log or other information.

tried with Cypress 4.5 today, and the performance is still extremely slow. each XHR request my client is trying to make takes FOREVER to resolve and the entire runner is just frozen up. Our organization is still on 3.8.0 because of this. I dont know if i can get a working example to use but i’ll see.

B.t.w. Going to a Linux machine is not an option at the moment 😉

Don’t bother, it’s the same thing.