playwright: [BUG] Screenshot comparison fails in Chrome because of font render differences on two machines with same specs

Context:

Playwright Version: 1.27.1
Operating System: macOS Ventura 13.1
Node.js version: 16.18.0
Browser: 109.0.5414.87, …/Library/Caches/ms-playwright/chromium-1041/chrome-mac/Chromium.app/Contents/MacOS/Chromium

Code Snippet

import { expect, test } from '@playwright/test';
import { getComponentTestPageUrl } from '../../helper/utils';

test('Button', async ({ page, headless }) => {
	await page.goto(getComponentTestPageUrl('Button'));
	await expect(page).toHaveScreenshot();
});

Describe the bug

We experience a very strange bug when using screenshot comparison on Chome. My colleague and me both use Playwright on macOS and when he updates the snapshots the test succeeds for him, but not for me, and the other way round. The snapshot comparison works fine on both machines with the devices Desktop Safari and iPhone 12.

We checked that everything is even on the same version: Node.js, macOS, the browser specs and so on.

Take a look at the snapshot diff: snapshot-comparison-diff

{
    name: 'Chromium',
    use: {
        ...devices['Desktop Chrome'],
        channel: 'chrome',
    },
},

About this issue

Original URL
State: closed
Created a year ago
Reactions: 1
Comments: 17

Most upvoted comments

I had the same problem a while ago with my own pixelmatch-based compare method. In my case the CI job (Docker based) was creating slightly different screenshots than dev machine(s) did. You can try setting {threshold: 0.7}. It helps in some cases. Sometimes the whole div is slightly off - then it won’t help. But… I ended up creating a separate CI job with branch name as a param, to record the screenshots and commit them back to git. I enable picture comparison only in CI because I know there is always the same docker image used for producing and for comparing. Default is to skip picture comparison so local test runs always pass. Downside is that I need to push to git to check if screenshot-tests are OK. Nice thing is that when the PRs do not pass I have a separate “updated screenshots by CI agent” commit where I can see what exactly in UI have changed in this particular PR. I am aware that my solution is a workaround but it works quite stable for over a year now. Proper solution here IMO would be an AI based compare but I didn’t have time to look into it (yet). Looking forward to see how will this issue be solved and how others have solved it.

PawelWesolowski on Jan 16, 2023

We are facing a similar issue. What I have observed is when we are running the test on a external monitor vs macbook screen the snapshots are having similar mismatch

pratheeshp007 on Jan 20, 2023