playwright: [BUG] Same test, different runs causes different heights with a weird compression effect
Context:
- Playwright Version: 1.27.1
- Operating System: MacOS 13.0
- Node.js version: 16.15.0
- Browser: All
- Extra: Also CI using MacOS 12 (Github Actions)
Describe the bug
Subsequent screenshots (on different test runs) of the same page are coming out with different heights. This is mainly due to a weird “compression” (like smashing) effect on one of the screenshots.
These two example come from the same code, same server (NextJS locally) running, just yarn test two times in a row:
I’m sorry for the long image, but that may be part of the problem (although short pages sometimes have this problem too).
If you compare the images using something like https://www.diffchecker.com/image-diff/ you can see the weird effect.
Is this a known bug? The differences are even greater once we run the tests on CI (more screenshots are different), but it’s flaky, so sometimes it happens and sometimes everything is good.
EDIT: Just to clarify, by compression I mean that the pixels themselves are smashed, so everything is shorter, which in turn causes the heights of the screenshots to differ, like on of them was “smashed”, which is wrong, e.g. causes the square logo to turn into a rectangle, etc
Thanks in advance!
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 20
- Comments: 42 (4 by maintainers)
Any update on this issue?
That is something we might be able to fix!
There has been a PR submitted above that allows you to provide a tolerance on this check. I’m hoping someone will graciously review it.
Link to PR posted for this: https://github.com/microsoft/playwright/pull/27513
Wanted to report in that this happens on my team’s MacBooks as well. We run both M1 and Intel chips. Our CI runs the official image. Locally generated,
fullPagescreenshots differ in 1px height from the CI screenshots.We can mitigate this by generating screenshots in a docker image, but a fix for this would be awesome. If I can help anyhow, let me know 😃
Playwright Version: 1.29.2 Playwright config: https://gist.github.com/thraizz/0b2ab81f31925265a467e75fec58935f macOS 13.1
Hello, we are also having this issue. Tests are always running inside the same environement (GitHub Action Linux Chrome headless)
Using the latest playwright 1.38.1
Any update on this?
Unfortunately this issue is limiting our ability to write and run concise tests. Our Bamboo test runs fail with this 1px issue. We have been able to work around the issue by specifying locators that capture a broader area which is not always ideal.
Error: Screenshot comparison failed:
We had this issue as well (in https://github.com/widgetti/solara/pull/67), we had a screenshot that had a different width sometimes. Based on https://github.com/microsoft/playwright/issues/20015#issuecomment-1427285592, we had the idea of printing out the bounding box.
So I think it is a combination of something else on the page pushing an element to a non-integer pixel position (but not stable every time) that can cause an image to change by 1-pixel width or height.
Hopefully, this is useful for others and also the cause of this issue.
We are also experiencing the same problem. When we run the test using ‘run test button’ and save screenshots we don’t get any problems. But when we try to run test using command line we get the error. This only applies to Google Chrome. Mac OS 13.2.1, M1 Pro
To make easier to vote on this issue (because what count are 👍 for first comment) @filipesmedeiros pls edit your comment and add folding for this huge image
Demo:
Hello
World!Fresh PR started to address this. https://github.com/microsoft/playwright/pull/28453
@dgozman @yury-s
I really think the issue is not with playwright: https://github.com/microsoft/playwright/issues/18827#issuecomment-1513345600 but having a bounding box with non-integer values (e.g 3.5) in combination with a non-stable page (e.g. fonts still loading).