cypress: Cypress hung up trying to run a spec - won't resolve
Current behavior
Can’t reproduce this locally - but intermittently when running specs via circleci (so far 50% of the time) - one of the specs (never the same one), spins causing circleci to timeout. When I look at the run in the Cypress dashboard it shows all the rest of the specs have passed except for that one and it’s just constantly saying it is running - but shows no test results. https://dashboard.cypress.io/projects/zb7814/runs/1686/overview is an example. When I rerun the tests they all pass again so it looks like there’s an issue with the cypress runner. This just started happening when we upgraded to 8 - specifically 8.2
The error from circleci is:
ERR_TIMED_OUT (-7) loading '<specUrl>'
Error: ERR_TIMED_OUT (-7) loading '<specUrl>
at rejectAndCleanup (electron/js2c/browser_init.js:217:1457)
at Object.failListener (electron/js2c/browser_init.js:217:1670)
at Object.emit (events.js:315:20)
After saying it was still running for several hours on the Cypress dashboard - it finally stopped and showed this error message: "The run started, but never completed. This can happen when the run is canceled from CI or when Cypress crashes during running tests.
Check your CircleCI #557d91f7-1bed-4c21-9f3c-9a4850bc64ad logs for more information. Archive this run to remove it from the runs list and analytics."
Desired behavior
Cypress not to get hung up
Test code to reproduce
Since it fails on a different spec each time, it doesn’t seem to be related to actual test code
Cypress Version
8.2.0
Other
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 37
- Comments: 89 (22 by maintainers)
Commits related to this issue
- #31: Remove video recording to hopefully help with this issue: https://github.com/cypress-io/cypress/issues/17627 — committed to wrporter/checkit by deleted user 2 years ago
- Disable video recording in end-to-end tests This is a workaround attempt for an issue with Cypress not successfully completing in Github Actions: https://github.com/cypress-io/cypress/issues/17627 — committed to mttschltz/life by mttschltz 2 years ago
- try adding videoUploadOnPasses set to false based on https://github.com/cypress-io/cypress/issues/17627 — committed to flexion/ef-cms by rosiehsmith 2 years ago
- try adding videoUploadOnPasses set to false based on https://github.com/cypress-io/cypress/issues/17627 — committed to flexion/ef-cms by rosiehsmith 2 years ago
Nope, everyone is doing a workaround. Most of the people go back to latest 7.x version. As we switched from 6.x to 8.5.0 currently and spend a lot effort to get there, we decided to turn off video captuirng as soon as it is fixed. It’s a very bad solution as we need a second run with only failed test and video capturing on and pray that this is fast enough that it don’t break.
I hope the people from Cypress will give that more priority soon. Again: I can offer my help to run a development version on our setup and share logs afterwards.
FYI: Upgraded to Cypress 9.2.0, still same issue
We have a recreation! https://app.circleci.com/pipelines/github/mjhenkes/jcontent/36/workflows/98ba8b18-e3e1-4b0d-b584-6866e025d6e6/jobs/81
Still occurs with Cypress 9.0.0
Given this issue is still marked “needs more info” since Aug 7th, what more information is required to get this issue moving with the Cypress team? Just had a GitHub actions pipeline run for 52 minutes after the official cypress action hung 😱
FYI; for others who find this issue, I’ve elected to mitigate it by adding
timeout-minutesto the action step … not ideal but better than getting charged another another hour of runner minutes! 😆FWIW, I’ve switched to using Chrome instead of the default electron browser as a workaround.
I’ve narrowed this issue to having been introduced with version 8.6.0 based on my CI runs https://app.circleci.com/pipelines/github/mjhenkes/jcontent?branch=master 8.5.0 -> 8.6.0: https://github.com/cypress-io/cypress/compare/v8.5.0...v8.6.0
Next I’ll narrow down which commit in 8.6.0 specifically causes the hang.
@mjhenkes The challenge with that particular issue is that it’s very very difficult to get to the root of what’s happening and get a reproducible scenario.
We ran into a bunch of dead ends when we tried to debug the issue on our codebase, I hope it’s not becoming the same here.
Anyway, I did went and ran a bunch of tests, trying to pinpoint the culprit commit, here are the results (link to the run by replacing the last digits in this URL: https://app.circleci.com/pipelines/github/Jahia/jcontent/1662):
First, it seems the commits I pointed out in my previous comment were not the cause of the issue, I initially thought that a change in our module codebase was the source of the problem, but based on the above, it might actually not be the case.
If you look at the table above, if I take the Cypress codebase from commit #753c5e5 I get a pass on all 24 runs when testing either a version of the module from May 30th (commit: #e0e8e56) or the current master. If I take the Cypress codebase from master, then I get 35 runs failing with 3 different versions of the module.
The interesting thing is that there’s actually one single change in the Cypress codebase between, in this PR: https://github.com/Jahia/jcontent/pull/518/files#diff-db39095f32428d360c52df9785fd8a41cf1e04e2391549cc47ca7fc2ad71b38d and that’s a change to testing codebase’s yarn.lock (though it might be painful to debug).
I have a series of further runs on a branch reverting the yarn.lock to the version from Jun 10th, I’ll review the results tomorrow (will be available here: https://app.circleci.com/pipelines/github/Jahia/jcontent?branch=revert-yarnlock&filter=all)
Still seeing on 9.5.4 using docker on around 1 in 3 builds, preventing us upgrading from 7.x
@jennifer-shehane this has been marked as needs information for a while, what info is needed and how can we get it?
it still happens with Cypress
8.7.0. So it’s not solved yet. We just use a workaround that we turn off the video capturing during run of the full suite. if it fails we execute only the failing ones. but that’s not a solution. I was just silent as there was no progress and it seemed Cypress engineers were busy.Surely some of the staff at cypress could weigh in on what they are running on their enterprise platform for their paying users? Im amazed how cypress themselves dont battle with this issue on their own platform because every single team ive worked with the biggest complaint is the amount of crashes and failures when running cypress in CI
@mjhenkes, A few days ago, this issue started to happen for the nightly runs in one of our public repositories (my previous message was on a private repo), so hopefully, it will help.
The Runs on CircleCI: https://app.circleci.com/pipelines/github/Jahia/jcontent?branch=master&filter=all The GitHub repo: https://github.com/Jahia/jcontent (the tests: https://github.com/Jahia/jcontent/tree/master/tests/cypress)
How to read/understand the runs:
jahia/jahia-discovery:8) and the latest version of our snapshot (jahia/jahia-discovery-dev:8-SNAPSHOT)If we summarize the past few runs:
If you compare the two profiles for run 1585 for test
createContent.spec.ts, you can see that although it does output1 passing (55s)on both, the log line aboutmochawesomeand the summary table are not displayed when the test is hanging. It might point in the direction of something happening after the1 passing...but beforemochawesomeis triggered.There have been changes in our codebase that could have served as a trigger for these failures:
When this issue happens, the video file gets created but is corrupted, if that helps, I repaired is using
synctree/untruncdocker image.https://user-images.githubusercontent.com/5667028/174668164-f75355c3-e550-4729-a238-2b3fbb377e2b.mp4
My guess is that something we’re doing now in the UI is what’s causing Cypress to hang.
Let me know if you need any additional details.
Note that we’re in the middle of migrating away from CircleCI, so in the next few days, we will lose run history for this codebase.
I have a PR out to fix this hang: https://github.com/cypress-io/cypress/pull/23001
A big thanks to @Fgerthoffert. Without your help getting a reproducible example, I wouldn’t have gotten anywhere.
This specific hang happened on Cypress versions 8.6.0 and greater when using electron and recording video.
I believe there are more hangs reported in this thread than just the one I am fixing. Help me get a reproducible example and I’ll see what I can do. I may close this issue with this fix and use new issues for additional hangs.
If you cannot upgrade to Cypress 10.4.0, you’ll have to lock into cypress 8.5.0 to avoid this hang. If there are enough of you that cannot upgrade, we can see about getting a 8.x or 9.x patch out, but I’d prefer it if we could avoid that.
@stychu, you may be right, once the PR is merged you can test the pre-release or wait for 10.4.0 to get released next week.
It seems this change is causing the hang: https://github.com/cypress-io/cypress/pull/18392
See my CI run evidence here: https://app.circleci.com/pipelines/github/mjhenkes/jcontent?branch=master All green builds up till i add in this change, then instantly red.
Now to dig into exactly what this change is causing.
Following on my previous comment, we eventually tracked this down to a change in our code which added Heap Analytics (https://heap.io) for evaluation purposes. Removing the Heap script fixes the stalling. Not sure if this useful for anyone else or for the cypress team, but that was the root cause for us.
Faced with the same issue, Cypress 9.1.1
Actually… I think I might have spotted something. If debug is enabled for
cypress:server:browsers:electron,debuggets called with a truncated, cloned version of the response. If it’s not enabled, it gets called with the full version. No clue why that might be causing issue, but it at least gives us a clue as to why this is impossible to replicate in debug mode.If I’m right about that, adding the debug string I suggested above will make it impossible to reproduce the issue (and hence, could be a more targeted temporary fix than
DEBUG=*).We have seen this issue as well and it disturbed our CI flow in Github Actions. We had one day last week where every PR, even on multiple reruns, was hitting this. Before that, it happened only occasionally for a period of about a week and half leading up.
A few things we tried:
uses: cypress-io/github-action@v2in our github workflow) - did not appear to help.DEBUG: 'cypress:*') - appeared to resolve the issue. Will update if the issue happens again but for several days now we haven’t seen a recurrence. This follows what @baviereteam and @gbparada28 reported. Did confirm the issue seemed to stop but that removing the DEBUG environment again causes the error again.Disabling videos did not work for us.
Seeing a lot more occurrences of this error after upgrading from 8.2.0 to 8.3.0. Affected almost every run.
I have noticed the same problem on my attempt to upgrade to 8.2.0, on 100% of my 8.2.0 builds on Jenkins (but not always on the same spec). The time between starting Cypress and the failure happening isn’t consistent (ranges from 5 to 15 minutes). No problems on 8.0.0. The same test run also executes correctly with 8.2.0 on my own (Windows 10) computer. However, when I enable debug mode (with
DEBUG=*), I can’t reproduce it anymore and the whole test run passes.Is there something else I can look into to help identify the cause of this problem?
They don’t battle with many more important issues: like Safari, runner performance, adequately preserve local storage and cookies, cy.origin, iframe, native events etc
I really have no idea why they prioritized component testing (you can see a lot of opened Pull requests with that stuff), since cypress has beed forced as e2e testing tool
Some updates here. I’ve been able to get a semi reliable recreation, about 1 failure for every 4 builds. I was also able to get debug logs (attached are a successful and unsuccessful run)
The pertinent sections start at line 17506 for the successful run and 17553 of the unsuccessful run
Successful
Failed
Note the lack GET for
http://jahia:8080/jahia/content-editor/en/create/59d45566-d61f-4ae9-9a07-b1517fdcac3d/jnt:bigTextIn the failed logs. It’s hard to say what exactly is causing this, but it is the most telling difference in the logs.Logs failed.txt logs succeeded.txt
So this is fixed in 10 version , isn’t it?
@mjhenkes, my bad I forgot to replace the snapshot image (which is private) with the release. This should work: https://github.com/mjhenkes/jcontent/pull/2
@mjhenkes To get the environment running on your CircleCI org:
manual-runworkflow. It’s currently usingQA_ENVIRONMENT. Contexts can be created by clicking the COG on the left bar, then “Contexts”.DOCKERHUB_LOGIN&DOCKERHUB_PASSWORD. Even though we’re accessing public images, the orb is doing a docker login, which will make the workflow fail if credentials are incorrect.The following other environment variables are detailed in the orb and I’m not sure if it will fail if the orb cannot find it. If it does, you should be fine populating them with empty values.
The test container will be rebuilt with every change you make.
Let me know if that works.
@Fgerthoffert, First off, thank you for the effort you’ve put in narrowing down the issue, I had suspected the yarn lock change once I saw it. At this point, what would help the most is if you could help me get my fork of your repo building in CI such that I can start debugging what dependencies in the yarn lock start causing this issue.
https://github.com/mjhenkes/jcontent https://app.circleci.com/pipelines/github/mjhenkes/jcontent?filter=all
I tried setting up circle, but it seems i’m missing some ENV variables, any guidance there would be great.
Hi all,
Adding this to
cypress.jsondefinitely fixed the issue for us:"videoUploadOnPasses": false,We haven’t had any timeouts for a month now. So I’m pretty sure it’s related to video uploads timing out… FYI we run 5 machines in parallel.
Hey all, I’m sorry that you’re all seeing this problem. Would anyone be able to provide a project where this is recreate-able for us to debug? A minimally recreate-able project would be nice, but i’ll take what I can get.
FWIW, we wound up sticking with Electron and switching to
DEBUG="cypress:*", and that seems to have fixed things for us (for now).Cypress 9.4.1 I don’t know what kind of miracle happened, but the parameters { “video”: true, “videoUploadOnPasses”: false, }
in cypress.json solved the problem ERR_TIMED_OUT (-7). I hope forever.
Not a lot of fun having this happen. Try setting a timeout on your cypress step to mitigate
Experiencing this issue in v8.5.0 / both Electron & Chrome within gitlab pipelines (docker-in-docker).
Things we’ve tried:
This feels like some sort of a memory issue (or memory leak in the test runner / mocha / etc) since it’s not a specific to a suite or test, but rather, seems like it’s hitting a limit somewhere. I wish there was a better way to reproduce this, but unfortunately, there’s not.
You have either the chance to switch to Chrome or turn off video capturing
A workaround, not a solution. At least I like the Chrome approach already (with Docker and the prepared Cypres image it’s not pain to set up) as I don’t know that much customers which use Electron.
the switch to Chrome works (3 successful runs in a row) for at least version
8.7.0and docker imagecypress/browsers:node14.17.0-chrome91-ff89… with newer images I wasn’t successful but for a different reasonDownside: duration of the whole test suite is now 45 to 60 min. longer than before (compared to Electron with video capturing)
Thanks, @rebel-l. So where you’re getting hung up specifically is here (so unsurprisingly, we’re getting hung up waiting for the initial page to load). I think we can disregard what I said above - that specific debug string obviously isn’t the magic cure.
At this point, I’m at a loss. There’s a timeout error being caught by Electron itself after ~30 seconds. In my testing trying to find a reproducible test case, I’ve been unable to make that happen (only sporadically, in CI, with our full test suite and configuration, and with debug off). I think this needs further investigation from the Cypress team, who will need a reproducible test case.
Ooh, nice capture, @rebel-l! Just from a quick inspection, it seems like it’s getting hung opening Electron - it’s making it to here, but getting hung up somewhere in here. It seems useful if you were able to rerun the same tests using:
(just adding debugging of
cypress:server:browsers:electron). That might let us narrow this down to a particular operation.We found a workaround. Could not reproduce the problem anymore after disabling video recording (setting
"video": falsein cypress.json). But unfortunately we have no more info than this about the problem.We’ve also started seeing this very sporadically since upgrading from 8.0.0 -> 8.2.0. I’ve seen it 2x in ~50 runs, so it’s definitely not something I can reliably replicate yet.