cypress: Recover from renderer / browser crashes

Related to #348.

It is actually possible for Cypress to implement strategies when the renderer (or browser process) crashes during a test run - something like recoverFromRendererCrashes: true by default.

There is already a mechanism for Cypress to “reload” mid-run, rebuild the state of every previous run test, skip over previously run tests, and continue with the next one in line.

In fact this is exactly what cy.visit already does under the hood.

We can utilize this same process upon a renderer / browser process crashing to continue on with the run.

So it may look something like this:

(Running Tests)

✓ test 1 - foo
✓ test 2 - bar
✓ test 3 - baz

Oh noes the renderer process crashed... we will attempt to recover

...Restarting tests at 'test 4 - quux'

✓ test 4 - quux
✓ test 5 - ipsum

Taking this a step further, we are starting to see several patterns emerge with how and why renderer processes crash - it is almost always related to extremely long test runs in a memory starved environment (such as Docker).

It may even be a good idea for us to always preemptively “break up” headless runs by spec file.

In other words, we could have an option like restartBrowserBetweenSpecFiles: true which would automatically kill the renderer / browser process before moving on to a different spec file (but still rebuild the state of the UI correctly, and still have a single contiguous video recording).

To the user it would look like nothing is really different, but internally the renderer process would be killed and then restarted.

This would forcefully purge primed memory from the process, which could keep environments like docker from ever crashing to begin with.

Depends on: #6170

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Reactions: 107
  • Comments: 93 (12 by maintainers)

Commits related to this issue

Most upvoted comments

We’ve started hitting this fairly frequently now too

We are hitting this problem as well. Not using Docker. Unfortunately, this issue makes cypress way too unreliable for automated tests.

I am also facing this issue consistently using headless Electron in CI. I’m on 3.8.3.

any update on this issue? we are experiencing it on version 11.2.0. Thanks!

Please fix this. I’m not able to set ipc=host in my ci/cd pipeline

Hi, we’re also experiencing this issue in Kubernetes (using Jenkins as our CI engine). Would be happy to provide additional information if helpful.

I’ve become very impatient waiting for the Cypress folks to fix these crashing issues. In the meantime, I’ve created a very similar API using selenium and am having no memory issues. There’s no recording of tests, but at least it’s reliable. Here’s a code snippet for you if you want to try it out.

import { Builder, ThenableWebDriver, By, WebElement, Key, Condition } from "selenium-webdriver"

/**
 * Wrap any promised coming from the Selenium driver so that we can
 * get stack traces that point to our code.
 */
async function wrapError<T>(p: Promise<T>) {
	const e = new Error()
	e["__wrapError"] = true
	try {
		const result = await p
		// Wait just a little bit in case the browser is about to navigate
		// or something.
		await new Promise(resolve => setTimeout(resolve, 20))
		return result
	} catch (error) {
		if (error["__wrapError"]) {
			throw error
		}
		e.message = error.message
		throw e
	}
}

async function waitFor(
	driver: ThenableWebDriver,
	fn: () => Promise<boolean | object>,
	timeout = 2000
) {
	await driver.wait(
		new Condition("wait", async () => {
			try {
				const result = await fn()
				return Boolean(result)
			} catch (error) {
				return false
			}
		}),
		timeout
	)
}

class Element {
	private promise: Promise<WebElement>
	then: Promise<WebElement>["then"]
	catch: Promise<WebElement>["catch"]

	constructor(
		public driver: ThenableWebDriver,
		promise: Promise<WebElement> | WebElement
	) {
		this.promise = Promise.resolve(promise)
		this.then = this.promise.then.bind(this.promise)
		this.catch = this.promise.catch.bind(this.promise)
	}

	/** Map in the monadic sense. */
	map(fn: (elm: WebElement) => Promise<WebElement | undefined | void>) {
		return new Element(
			this.driver,
			wrapError(
				this.promise.then(async elm => {
					const result = await fn(elm)
					if (result) {
						return result
					} else {
						return elm
					}
				})
			)
		)
	}

	waitFor(fn: (elm: WebElement) => Promise<boolean | object>) {
		return this.map(elm => waitFor(this.driver, () => fn(elm)))
	}

	mapWait(fn: (elm: WebElement) => Promise<WebElement>) {
		return this.waitFor(fn).map(fn)
	}

	click() {
		return this.map(elm => elm.click())
	}

	clear() {
		return this.map(elm => elm.clear())
	}

	type(text: string) {
		return this.map(elm => elm.sendKeys(text))
	}

	enter() {
		return this.map(elm => elm.sendKeys(Key.RETURN))
	}

	backspace() {
		return this.map(elm => elm.sendKeys(Key.BACK_SPACE))
	}

	find(selector: string) {
		return this.mapWait(elm => {
			return elm.findElement(By.css(selector))
		})
	}

	findAll(selector: string) {
		return new Elements(
			this.driver,
			this.promise.then(elm => {
				return waitFor(this.driver, () =>
					elm.findElements(By.css(selector))
				).then(() => {
					return elm.findElements(By.css(selector))
				})
			})
		)
	}

	contains(text: string) {
		return this.mapWait(elm => {
			// TODO: escape text.
			// https://stackoverflow.com/questions/12323403
			return elm.findElement(By.xpath(`//*[contains(text(), '${text}')]`))
		})
	}

	clickText(text: string) {
		return this.contains(text).click()
	}
}

class Elements {
	private promise: Promise<Array<WebElement>>
	then: Promise<Array<WebElement>>["then"]
	catch: Promise<Array<WebElement>>["catch"]

	constructor(
		public driver: ThenableWebDriver,
		promise: Promise<Array<WebElement>> | Array<WebElement>
	) {
		this.promise = Promise.resolve(promise)
		this.then = this.promise.then.bind(this.promise)
		this.catch = this.promise.catch.bind(this.promise)
	}

	/** Map in the monadic sense. */
	map(
		fn: (
			elm: Array<WebElement>
		) => Promise<Array<WebElement> | undefined | void>
	) {
		return new Elements(
			this.driver,
			wrapError(
				this.promise.then(async elms => {
					const result = await fn(elms)
					if (Array.isArray(result)) {
						return result
					} else {
						return elms
					}
				})
			)
		)
	}

	waitFor(fn: (elm: Array<WebElement>) => Promise<boolean | object>) {
		return this.map(elm => waitFor(this.driver, () => fn(elm)))
	}

	mapWait(fn: (elm: Array<WebElement>) => Promise<Array<WebElement>>) {
		return this.waitFor(fn).map(fn)
	}

	clickAll() {
		return this.map(async elms => {
			await Promise.all(elms.map(elm => elm.click()))
		})
	}

	atIndex(index: number) {
		return new Element(
			this.driver,
			wrapError(
				this.promise.then(elms => {
					const elm = elms[index]
					if (!elm) {
						throw new Error("Element not found!")
					}
					return elm
				})
			)
		)
	}
}

export class Browser {
	private promise: Promise<void>
	then: Promise<void>["then"]
	catch: Promise<void>["catch"]

	constructor(public driver: ThenableWebDriver, promise?: Promise<void>) {
		this.promise = Promise.resolve(promise)
		this.then = this.promise.then.bind(this.promise)
		this.catch = this.promise.catch.bind(this.promise)
	}

	visit(route: string) {
		return new Browser(
			this.driver,
			wrapError(
				this.promise.then(async () => {
					await this.driver.get(route)
				})
			)
		)
	}

	rerender() {
		return new Browser(this.driver, wrapError(rerender(this.driver)))
	}

	flushTransactions() {
		return new Browser(this.driver, wrapError(flushTransactions(this.driver)))
	}

	find(selector: string) {
		return new Element(
			this.driver,
			wrapError(
				this.promise
					.then(() => {
						return waitFor(this.driver, async () =>
							this.driver.findElement(By.css(selector))
						)
					})
					.then(() => {
						return this.driver.findElement(By.css(selector))
					})
			)
		)
	}

	getClassName(className: string) {
		return this.find("." + className)
	}

	getTitle() {
		return this.driver.getTitle()
	}

	waitFor(fn: () => Promise<boolean>, timeout = 2000) {
		return new Browser(this.driver, waitFor(this.driver, fn))
	}

	waitToLeave(url: string) {
		return new Browser(
			this.driver,
			wrapError(
				waitFor(
					this.driver,
					async () => {
						const currentUrl = await this.driver.getCurrentUrl()
						return url !== currentUrl
					},
					10000
				)
			)
		)
	}

	waitForRoute(url: string) {
		return new Browser(
			this.driver,
			wrapError(
				waitFor(
					this.driver,
					async () => {
						const currentUrl = await this.driver.getCurrentUrl()
						return url === currentUrl
					},
					10000
				)
			)
		)
	}
}

Also experiencing this issue. Tried both solutions provided by @sesamechicken with no luck.

Happening during Netlify build w/ the netlify-plugin-cypress plugin. I am just starting to add Cypress to my project and only have a single simple test to run, so the issue shouldn’t be many tests exhausting memory.

I am facing the same issue as well, I tried optimizing my tests and following the best practices guide: https://docs.cypress.io/guides/references/best-practices.html. I can see improvements but still crashes the CI most of the time.

And just to add - there is open pull request that adds video recording to Chrome https://github.com/cypress-io/cypress/pull/4791 which is THE main thing stopping people from using Chrome on CI

If you’re seeing consistent crashes and would like this implemented, please leave a note in the issue. Yes, please.

A few days ago I started facing the same issue regardless no changes were made. It’s running on Travis without docker and against a separate app that is not installed in the same code base. What interesting, that switching to --browser chrome seems to help with it, so looks like it is related to the electron no matter if it is headless or not - in both cases it’s failing. However, with chrome, you lose the video recording. Any progress on this topic? @brian-mann

@Neoxrus86 I just have package.json:

  "scripts": {
    "cy:run": "cypress run --browser chrome"
  }

and then in my github actions I mount the folder in with all of my tests and run the npm command above:

docker run --add-host="test.cypress:172.17.0.1" --network host --env-file <(env | grep cypress_) -v $(pwd):/app {{MY_CYPRESS_IMAGE_NAME}} /bin/bash -c "cd app/cypress && npm i && npm run cy:run"

in my case I dont have an entrypoint.sh and run tests on test.cypress because I have a multitenant SAAS app that has subdomains so I add a host in there that I can work off.

We have noticed often Electron crashing while running tests, but switching to use Chrome browser solved the problem with everything else being equal.

@sesamechicken It’s happening on 6.3.0. I plan to upgrade to 6.4.0 next week and will report back then

also just got it while running cypress github-actions

Also facing this issue. No changes to the code base. It just started to happen on all of our Jenkins jobs. Please advise!

Please fix this or provide work-around for different environments. In my case I’m running Cypress using Jenkins and pipelines where I do not have access to flags.

@EvanHerman (and anyone else on this thread): FWIW, since switching to Chrome (from Electron) and setting some flags we have not seen a crash in CI for almost 2 months now.

See https://github.com/cypress-io/cypress/issues/350#issuecomment-503231128 for details.

@RockChild Downgrading to 3.3.0 (or even 3.2.0) has not resolved this issue for us.

Similar to you we just started seeing this on or around May 27. No idea what has changed, and we have tried just about everything to fix this. It is gradually getting worse, with almost 100% crash rate today (when it started a few weeks ago it was closer to 5-10%).

Only happening on CircleCI. /dev/shm is 30GB there. No pattern to where the tests fail. Nothing interesting when using DEBUG=cypress:*.

I switched to cypress/browsers:chrome69, changed the package version to 3.3.0 and, with the following build step config in drone.io, it seems that the renderer doesn’t crash anymore:

steps:
  - name: dev-tests
    image: cypress/browsers:chrome69
    shm_size: 4096000000
    mem_limit: 1000000000
    commands:
      - npm ci
      - $(npm bin)/cypress verify
      - $(npm bin)/cypress run

Later edit - it just crashed this morning, so it seems that this is not it. Isn’t there any way to auto-restart the test if it crashes ?

@jbinto yeah, looks like it started crashing after upgrade to 3.3.1, so I’ll try to downgrade to 3.3.0. Thanks for your insights!

I’ve recently started running into the issue, as our codebase starts to acquire more dependencies. It’s intermittent and unpredictable. Sometimes I get a passing test, sometimes it fails the moment it begins.

After more experimentation, I’ve found that using the cypress/browsers:chrome69 image instead of the cypress/base:10 made the issue go away. This issue is likely to be tied to an older version of electron being unable to handle a larger codebase, and I think more effort should go into updating electron.

Hi cypress team!

We are also getting this error when we use cypress run as well as cypress open

We noticed that it happens more when we use cy.wait. We can consistently reproduce it when we use cy.wait with a value greater than 20000. This is on our circle-ci linux containers fyi.