site-kit-wp: VRT tests cannot be run on ARM Macs/M1 Macs

Bug Description

Using an ARM/M1 Mac, running npm run test:visualtest will fail with an error because the Docker image for Backstop is not configured to run an ARM64 image. It will segfault and fail. If you are on a machine that you migrated from an Intel Mac, it will then open whatever previous results you ran in your project folder.

Until BackstopJS provides a fully-integrated solution (see: https://github.com/garris/BackstopJS/issues/1300), we will need to update our codebase to allow the VRTs to run on ARM Macs, as several core team members are using these machines and can’t run (and thus can’t update) the VRTs locally.

Do not alter or remove anything below. The following sections will be managed by moderators only.

Acceptance criteria

It should be possible to run the VRT (Visual Regression Tests) using npm run test:visualtest locally on both Intel and ARM machines. It would be preferable if these could be run on ARM without needing to create any custom images/etc. locally.

Implementation Brief

Create a custom Docker image based on the one in this comment: https://github.com/google/site-kit-wp/issues/4619#issuecomment-1201808910
- It should be noted this image let me run the VRTs on macOS after updating Docker to use at least 4 CPU cores. This might need to be added to the docs or a script to detect CPUs available to Docker. 🤔
Publish the image to GitHub Packages so CI and team members can use the image
Ensure the BackstopJS code uses our new googlesitekit/vrt:VERSION image instead of backstopjs/backstopjs:VERSION
Update reference images in case of any minor/font differences
Ensure tests run on ARM and Intel macs do not differ; ensure CI tests still pass after being updated by an ARM-based Mac.

Test Coverage

VRT reference images will likely need updating after this change.

QA Brief

No QA needed.

Changelog entry

Update Visual Regression test code to run on ARM-based Macs.

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 3
Comments: 16

Most upvoted comments

Creating a custom Docker image using this Dockerfile:

FROM node:16.3.0

ARG BACKSTOPJS_VERSION

ENV \
	BACKSTOPJS_VERSION=$BACKSTOPJS_VERSION

# Base packages
RUN apt-get update && \
	apt-get install -y git sudo software-properties-common

RUN sudo PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true npm install -g --unsafe-perm=true --allow-root backstopjs@${BACKSTOPJS_VERSION}

RUN wget https://dl-ssl.google.com/linux/linux_signing_key.pub && sudo apt-key add linux_signing_key.pub
RUN sudo add-apt-repository "deb http://dl.google.com/linux/chrome/deb/ stable main"

# RUN	apt-get -y update && apt-get -y install google-chrome-stable

# RUN apt-get install -y firefox-esr

# gconf-service libxext6.... added for https://github.com/garris/BackstopJS/issues/1225

RUN apt-get -qqy update \
  && apt-get -qqy --no-install-recommends install \
    chromium \
    libfontconfig \
    libfreetype6 \
    xfonts-cyrillic \
    xfonts-scalable \
    fonts-liberation \
    fonts-ipafont-gothic \
    fonts-wqy-zenhei \
    gconf-service libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxss1 libxtst6 libappindicator1 libnss3 libasound2 libatk1.0-0 libc6 ca-certificates fonts-liberation lsb-release xdg-utils wget \
  && rm -rf /var/lib/apt/lists/* \
  && apt-get -qyy clean


WORKDIR /src

ENTRYPOINT ["backstop"]

And then building it and using that image, along with ensuring Docker for Mac could use at least 4 CPUs let me run the VRTs. Without upping the CPU limit to at least 4, it would hang:

tofumatt on Aug 1, 2022

Another update: looks like Puppeteer will soon have an ARM build, there’s a PR here: https://github.com/puppeteer/puppeteer/pull/7546 and the issue is here: https://github.com/puppeteer/puppeteer/issues/6622.

Waiting a bit longer on this one as the workarounds aren’t working for me…

tofumatt on Apr 20, 2022