BackstopJS: Intermittent Failures

Hey all, We’re hitting intermittent bumps in our chromy-based backstop suite.

On a semi-regular basis we get false failures of this: compare | Chromy error: Error. See scenario – xxx xxx xxx xxx

Accompanied by a report that looks like this: screen shot 2017-11-07 at 2 39 41 pm

We’ve played with asyncCapture/compare limit, played with delay, but no dice. Sometimes it can be resolved by pkill Chrome, but generally we’re struggling to find the special sauce to solve them for good.

First question - have you encountered these intermittent failures? Second question - Is there a chromy/backstop flag we can set to investigate the error or capture some more context around this Chromy error?

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 38 (21 by maintainers)

Most upvoted comments

Alright, good news guys. That seemed to be our issue. We added wait(<selector>) to all of our DOM interactions and we’ve run our suite 15+ times with "asyncCaptureLimit": 20. All clear!

Here are the guts of our solution if you’re interested.

We have a helper script in which we introduced two functions for click() and type():

let chromy
exports.init = (chromyReference) => {
    chromy = chromyReference
    return this
}
exports.click = (selector) => {
    chromy.wait(selector).click(selector)
    return this
}
exports.type = (selector, text) => {
    chromy.wait(selector).type(selector, text)
    return this
}

And then in our onReadyScript:

const interactions = require('../interactions.js')
module.exports = function (chromy, scenario, vp) {
	interactions
		.init(chromy, vp)
		.click('.some-selector')
		.type('.another-selector', 'text to type')
}

A few chromy/backstop tweaks helped us debug this work:

backstop: index.js:65 - log uncaught errors (this is generally helpful to catch errors in onReadyScript too)
chromy: document.js - in reject(new WaitTimeoutError('...')) - we added the selector on which we were waiting. In a few of our scenarios we discovered we had bogus selectors that were never working in the first place, so our test was timing out.

Hope this helps others with the same issue!

joseph-stano on Nov 14, 2017

@kiran-redhat – thank you for tracking this down! I completely agree with your idea. We should add a log.error at line 319 which also includes the scenario label+viewport. This would make debugging MUCH easier for engineers!

garris on Nov 10, 2017

@garris Regarding @joseph-stano OP, I modified backstop source in node_modules like you mentioned and received the following error for every one of our tests that result in the white diff for all 3 images as in the OP screenshot.

{ Error: An error has occurred evaluating the script in the browser.TypeError: Cannot read property 'focus' of null
    at <anonymous>:2:102
    at <anonymous>:2:112
    at EvaluateError.ExtendableBuiltin (/Users/*****/code/*****/node_modules/backstopjs/node_modules/chromy/dist/error.js:19:28)
    at new EvaluateError (/Users/*****/code/*****/node_modules/backstopjs/node_modules/chromy/dist/error.js:133:131)
    at Chromy._evaluateWithReplaces$ (/Users/*****/code/*****/node_modules/backstopjs/node_modules/chromy/dist/document.js:448:21)
    at tryCatch (/Users/*****/code/*****/node_modules/backstopjs/node_modules/regenerator-runtime/runtime.js:65:40)
    at GeneratorFunctionPrototype.invoke [as _invoke] (/Users/*****/code/*****/node_modules/backstopjs/node_modules/regenerator-runtime/runtime.js:299:22)
    at GeneratorFunctionPrototype.prototype.(anonymous function) [as next] (/Users/*****/code/*****/node_modules/backstopjs/node_modules/regenerator-runtime/runtime.js:117:21)
    at tryCatch (/Users/*****/code/*****/node_modules/backstopjs/node_modules/regenerator-runtime/runtime.js:65:40)
    at invoke (/Users/*****/code/*****/node_modules/backstopjs/node_modules/regenerator-runtime/runtime.js:155:20)
    at /Users/*****/code/*****/node_modules/backstopjs/node_modules/regenerator-runtime/runtime.js:165:13
  object:
   { type: 'object',
     subtype: 'error',
     className: 'TypeError',
     description: 'TypeError: Cannot read property \'focus\' of null\n    at <anonymous>:2:102\n    at <anonymous>:2:112',
     objectId: '{"injectedScriptId":2,"id":1}'
   } 
}

Do you have any tips on how we can learn more about what file anonymous is, or perhaps what injectedScriptId is pointing to? I’ve removed all instances of focus from our code base and still receive the errors consistently.

Also, I set asyncCaptureLimit to 1. Instead of getting a third of the tests erroring, I only got one that errored. It errored with the same error message directly above.

ackmanx on Nov 9, 2017