testcafe: Visual regression testing

@DevExpress/testcafe Let’s discuss functionality of layout testing

I suggest to use the following approach. Perhaps it looks too over engineered, but it is pretty flexible at my sight.

Screenshot provider

I suggest to provide possibility to use any comparing screenshots library. We do two options out-of-the-box - per-pixel comparison and perceptual hash comparison.

To avoid working with huge binary images we can calculate perceptual hash. In this case we don’t create screenshot file and compare only hash value. To build the diff images use should run the local tests in per-pixel mode.

Also we can provide possibility of taking screenshot in black and white mode or ultra contrast mode. I think it can be passed to the chosen library as layoutTestOptions.

So we create:

1. Pixel perfect provider with options

{
    mode: 'color' | 'black_and_white',
    threshold: Number	
}

1. Perceptual hash with options

{
    threshold: Number 
    // or
    exactMatching: true | false 
    //it will be depend of implementation
}

Run options

For the run function we add options for screenshot comparing:

{
    etalonsPath: <relative_or_absolute_path>,
    layoutTestProvider: <provider_name>,
    layoutTestOptions: {...},
    updateEtalons: true | false
}

Accordingly, we add the similar options to the CLI interface: --etalons-path, --layout-provider, --update-etalons.

Test API

We provide two different ways to testing layout.

1. We look for an etalon automatically and user should just call t.checkLayout() method.

checkLayout() is searching for a screenshot for test in according to index of using in the test code body, starting with 0.

test('testName', async t => {
    await t
        .click('#el')
        .checkLayout() //looking for .../<testName>/<workerName>/1.png        
        .type('.inputClass', 'value')
        .checkLayout() //looking for .../<testName>/<workerName>/2.png
        ......
        .click(...)
        .checkLayout() //looking for .../<testName>/<workerName>/<N>.png
});

1. We provide testControler.Image constructor and .equalLayout() assertion. That means user should decide on their own how to store artifacts and etalons, we just use comparing logic from provider.

E.g.:

 test('testName', async t => {
     await t.expect(new t.Image(<image_path>)).equalLayout();
 });

Also we should resize browser window up to screenshot size.

Screenshots storage

Every provider implements mechanism of storing artifacts. For per-pixel provider we store screenshots similarly with current screenshots directory. Also we should create difference files in that directory.

Etalons will be got from path that specified in etalonsPath for programmatically API or from --etalons-path parameter for CLI. For hash comparison we write key-value pairs to the .json file.

{
    '<testName1><workerName1><etalonId1>': '3c3e0e1a3a1e1e2e',
    ...
    '<testNameN><workerNameN><etalonIdN>': 'ac3e0e1a3a1e1e2F'
}

Updating of etalons

As soon as the first screenshot will be different from etalon we run web application with GUI interface for managing etalons. Service will run only if was passed --update-etalons to CLI or updateEtalons to the runOptions in programmatically API. In this case we output in report only path to the difference file.

As alternative we can just output paths to the artifacts, etalons and difference files, but it does not look convenient.