cypress: Recover from renderer / browser crashes
Related to #348.
It is actually possible for Cypress to implement strategies when the renderer (or browser process) crashes during a test run - something like recoverFromRendererCrashes: true
by default.
There is already a mechanism for Cypress to “reload” mid-run, rebuild the state of every previous run test, skip over previously run tests, and continue with the next one in line.
In fact this is exactly what cy.visit
already does under the hood.
We can utilize this same process upon a renderer / browser process crashing to continue on with the run.
So it may look something like this:
(Running Tests)
✓ test 1 - foo
✓ test 2 - bar
✓ test 3 - baz
Oh noes the renderer process crashed... we will attempt to recover
...Restarting tests at 'test 4 - quux'
✓ test 4 - quux
✓ test 5 - ipsum
Taking this a step further, we are starting to see several patterns emerge with how and why renderer processes crash - it is almost always related to extremely long test runs in a memory starved environment (such as Docker).
It may even be a good idea for us to always preemptively “break up” headless runs by spec file.
In other words, we could have an option like restartBrowserBetweenSpecFiles: true
which would automatically kill the renderer / browser process before moving on to a different spec file (but still rebuild the state of the UI correctly, and still have a single contiguous video recording).
To the user it would look like nothing is really different, but internally the renderer process would be killed and then restarted.
This would forcefully purge primed memory from the process, which could keep environments like docker
from ever crashing to begin with.
Depends on: #6170
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Reactions: 107
- Comments: 93 (12 by maintainers)
Commits related to this issue
- fix: mixing api and component tests (#349) — committed to cypress-io/cypress by bahmutov 4 years ago
We’ve started hitting this fairly frequently now too
We are hitting this problem as well. Not using Docker. Unfortunately, this issue makes cypress way too unreliable for automated tests.
I am also facing this issue consistently using headless Electron in CI. I’m on 3.8.3.
any update on this issue? we are experiencing it on version 11.2.0. Thanks!
Please fix this. I’m not able to set ipc=host in my ci/cd pipeline
Hi, we’re also experiencing this issue in Kubernetes (using Jenkins as our CI engine). Would be happy to provide additional information if helpful.
I’ve become very impatient waiting for the Cypress folks to fix these crashing issues. In the meantime, I’ve created a very similar API using selenium and am having no memory issues. There’s no recording of tests, but at least it’s reliable. Here’s a code snippet for you if you want to try it out.
Also experiencing this issue. Tried both solutions provided by @sesamechicken with no luck.
Happening during Netlify build w/ the netlify-plugin-cypress plugin. I am just starting to add Cypress to my project and only have a single simple test to run, so the issue shouldn’t be many tests exhausting memory.
I am facing the same issue as well, I tried optimizing my tests and following the best practices guide: https://docs.cypress.io/guides/references/best-practices.html. I can see improvements but still crashes the CI most of the time.
And just to add - there is open pull request that adds video recording to Chrome https://github.com/cypress-io/cypress/pull/4791 which is THE main thing stopping people from using Chrome on CI
If you’re seeing consistent crashes and would like this implemented, please leave a note in the issue.
Yes, please.A few days ago I started facing the same issue regardless no changes were made. It’s running on Travis without docker and against a separate app that is not installed in the same code base. What interesting, that switching to
--browser chrome
seems to help with it, so looks like it is related to the electron no matter if it is headless or not - in both cases it’s failing. However, with chrome, you lose the video recording. Any progress on this topic? @brian-mann@Neoxrus86 I just have package.json:
and then in my github actions I mount the folder in with all of my tests and run the npm command above:
in my case I dont have an entrypoint.sh and run tests on
test.cypress
because I have a multitenant SAAS app that has subdomains so I add a host in there that I can work off.We have noticed often Electron crashing while running tests, but switching to use Chrome browser solved the problem with everything else being equal.
@sesamechicken It’s happening on
6.3.0
. I plan to upgrade to6.4.0
next week and will report back thenalso just got it while running cypress github-actions
Also facing this issue. No changes to the code base. It just started to happen on all of our Jenkins jobs. Please advise!
Please fix this or provide work-around for different environments. In my case I’m running Cypress using Jenkins and pipelines where I do not have access to flags.
@EvanHerman (and anyone else on this thread): FWIW, since switching to Chrome (from Electron) and setting some flags we have not seen a crash in CI for almost 2 months now.
See https://github.com/cypress-io/cypress/issues/350#issuecomment-503231128 for details.
I’ve just given up and switched to testcafe.
Get Outlook for Androidhttps://aka.ms/ghei36
From: neboryte notifications@github.com Sent: Saturday, July 13, 2019 4:23:08 PM To: cypress-io/cypress Cc: Bogdan Calapod; Manual Subject: Re: [cypress-io/cypress] Recover from renderer / browser crashes (#349)
We are hitting this problem as well. Not using Docker.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcypress-io%2Fcypress%2Fissues%2F349%3Femail_source%3Dnotifications%26email_token%3DABJVVR2NXFIZX4XDY7TLEY3P7HJLZA5CNFSM4CZ6PJ42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ3RTVA%23issuecomment-511121876&data=02|01||ca1a4a9cdfca40b14b8308d707953f8c|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|636986209898185694&sdata=SL3JL0DVIokmpWemSUEeWDBky6azp9PQwxar9RVq8bg%3D&reserved=0, or mute the threadhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABJVVR24JRVUTGEWYRPGWALP7HJLZANCNFSM4CZ6PJ4Q&data=02|01||ca1a4a9cdfca40b14b8308d707953f8c|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|636986209898195699&sdata=YUj%2BJggwZ626ht5SYL8ZNqb0O6%2BZtuFV3Ch78togaHg%3D&reserved=0.
@RockChild Downgrading to 3.3.0 (or even 3.2.0) has not resolved this issue for us.
Similar to you we just started seeing this on or around May 27. No idea what has changed, and we have tried just about everything to fix this. It is gradually getting worse, with almost 100% crash rate today (when it started a few weeks ago it was closer to 5-10%).
Only happening on CircleCI.
/dev/shm
is 30GB there. No pattern to where the tests fail. Nothing interesting when usingDEBUG=cypress:*
.I switched to
cypress/browsers:chrome69
, changed the package version to3.3.0
and, with the following build step config indrone.io
, it seems that the renderer doesn’t crash anymore:Later edit - it just crashed this morning, so it seems that this is not it. Isn’t there any way to auto-restart the test if it crashes ?
@jbinto yeah, looks like it started crashing after upgrade to 3.3.1, so I’ll try to downgrade to 3.3.0. Thanks for your insights!
I’ve recently started running into the issue, as our codebase starts to acquire more dependencies. It’s intermittent and unpredictable. Sometimes I get a passing test, sometimes it fails the moment it begins.
After more experimentation, I’ve found that using the cypress/browsers:chrome69 image instead of the cypress/base:10 made the issue go away. This issue is likely to be tied to an older version of electron being unable to handle a larger codebase, and I think more effort should go into updating electron.
Hi cypress team!
We are also getting this error when we use
cypress run
as well ascypress open
We noticed that it happens more when we use
cy.wait
. We can consistently reproduce it when we usecy.wait
with a value greater than20000
. This is on our circle-ci linux containers fyi.