jest-puppeteer: Take screenshot when a test fails

It is amazing to see that everyone has a solution and nobody wants to make a PR, welcome in Wordpress world 🤗

Thought I’d share my solution to this, using jest-circus and extending the jest-puppeteer environment. It’s a little messy with the sleeps, I want to find a better way to handle that, but it works for now. Maybe jest-puppeteer can integrate with jest-circus out of the box once circus has more adoption and make this better.

const fs = require('fs');
const PuppeteerEnvironment = require('jest-environment-puppeteer');
require('jest-circus');

const retryAttempts = process.env.RETRY_ATTEMPTS || 1;

class CustomEnvironment extends PuppeteerEnvironment {

  async setup() {
    await super.setup();
  }

  async teardown() {
    // Wait a few seconds before tearing down the page so we
    // have time to take screenshots and handle other events
    await this.global.page.waitFor(2000);
    await super.teardown()
  }

  async handleTestEvent(event, state) {
    if (event.name == 'test_fn_failure') {
      if (state.currentlyRunningTest.invocations > retryAttempts) {
        const testName = state.currentlyRunningTest.name;
        // Take a screenshot at the point of failure
        await this.global.page.screenshot({ path: `test-report/screenshots/${testName}.png` });
      }
    }
  }

}

module.exports = CustomEnvironment

@ricwal-richa yes, takeScreenshot was not part of my code sample so it’s normal you got this error. Here is the full code I use (this is typescript):

import path from 'path';
import mkdirp from 'mkdirp';

const screenshotsPath = path.resolve(__dirname, '../testReports/screenshots');

const toFilename = (s: string) => s.replace(/[^a-z0-9.-]+/gi, '_');

export const takeScreenshot = (testName: string, pageInstance = page) => {
  mkdirp.sync(screenshotsPath);
  const filePath = path.join(
    screenshotsPath,
    toFilename(`${new Date().toISOString()}_${testName}.png`),
  );
  return pageInstance.screenshot({
    path: filePath,
  });
};

export const registerScreenshotReporter = () => {
  /**
   * jasmine reporter does not support async.
   * So we store the screenshot promise and wait for it before each test
   */
  let screenshotPromise: Promise<any> = Promise.resolve();
  beforeEach(() => screenshotPromise);
  afterAll(() => screenshotPromise);

  /**
   * Take a screenshot on Failed test.
   * Jest standard reporters run in a separate process so they don't have
   * access to the page instance. Using jasmine reporter allows us to
   * have access to the test result, test name and page instance at the same time.
   */
  (jasmine as any).getEnv().addReporter({
    specDone: async (result: any) => {
      if (result.status === 'failed') {
        screenshotPromise = screenshotPromise
          .catch()
          .then(() => takeScreenshot(result.fullName));
      }
    },
  });
};

+1 for a much needed feature

Here is how I did it:

export const registerScreenshotReporter = () => {
  /**
   * jasmine reporter does not support async.
   * So we store the screenshot promise and wait for it before each test
   */
  let screenshotPromise = Promise.resolve();
  beforeEach(() => screenshotPromise);
  afterAll(() => screenshotPromise);

  /**
   * Take a screenshot on Failed test.
   * Jest standard reporters run in a separate process so they don't have
   * access to the page instance. Using jasmine reporter allows us to
   * have access to the test result, test name and page instance at the same time.
   */
  jasmine.getEnv().addReporter({
    specDone: async result => {
      if (result.status === 'failed') {
        screenshotPromise = screenshotPromise
          .catch()
          .then(() => takeScreenshot(result.fullName));
      }
    },
  });
};

Use this package: https://github.com/jeeyah/jestscreenshot

To come back at this issue, before I found this issue I had several attempts at this myself. My goal had higher ambitions though: I wanted jest to automatically generate a screenshot before and after each individual test. Disclaimer: I haven’t really used the jest API before, so this is probably the best thing I came up with:

const path = require('path');
const fs = require('fs').promises;

const hash = (str) => {
    if (str.length === 0) {
        return 0;
    }
    let hash = 0;
    for (let i = 0; i < str.length; i++) {
        hash  = ((hash << 5) - hash) + str.charCodeAt(i);
        hash |= 0; // Convert to 32bit integer
    }
    return hash;
};

const origIt = global.it;
global.it = (description, test, timeout) => {
    const result = origIt(description, async () => {
        const basePath = path.join(result.result.testPath, '..', '__screenshots__');
        try {
            await fs.mkdir(basePath, { recursive: true });
            await page.screenshot({ path: path.join(basePath, `before-${hash(result.result.fullName)}.png`) });
        } finally {
            try {
                return await test();
            } finally {
                await page.screenshot({ path: path.join(basePath, `after-${hash(result.result.fullName)}.png`) });
            }
        }
    }, timeout);
    return result;
};

Re-defining it is kinda ugly but it works better than all my other versions. Also I ignored test for now, because in my case I don’t need it. Now with Jest 24 the file can be specified in setupFilesAfterEnv as part of an array. I’d really like to see such a feature soon in this project, and I would create a PR as well, it’s just that I’m not confident enough in my code. So if someone could nudge me into the right direction that would be great.

@petrogad check if you’re not calling jestPuppeteer.resetPage() or doing something else with global page object in afterEach/afterAll. This was the reason I was getting blank screenshots.

@neoziro I should be able to submit a PR with adapted solution of @testerez in following days

@AlexMarchini thanks for that! Perfect, I wanted to use jest-circus as well. If anyone else is wondering (or anyone has suggestions) like I was how to actually use this this is what my jest-config.js looks like:

{
    ...
    globalSetup: 'jest-environment-puppeteer/setup',
    globalTeardown: 'jest-environment-puppeteer/teardown',
    testEnvironment: './jest-environment.js',
    testRunner: 'jest-circus/runner',
    setupFilesAfterEnv: ['./jest-setup.js'],
    ...
}

So I removed preset and jest-environment.js is the above snippet. And jest-setup.js has nothing but require('expect-puppeteer'); in it.

And that seems to be working… with jest-circus as the runner. (Note just run yarn add --dev jest-circus)

Hope this helps someone.

// jest.setup.js

global.it = async function(name, func) {
  return await test(name, async () => {
    try {
      await func();
    } catch (e) {
      await fs.ensureDir('e2e/screenshots');
      await page.screenshot({ path: `e2e/screenshots/${name}.png` });
      throw e;
    }
  });
};

@lucassardois PR was done but I had to migrate to TypeScript before. Will be done in the next weeks.

Maybe the PR to WordPress Gutenberg by @kevin940726 :

https://github.com/WordPress/gutenberg/pull/28449

…would provide a basis for a PR to the jest-environment-puppeteer package in this repo. Specifically, it seems to be this file that needs to be updated:

https://github.com/smooth-code/jest-puppeteer/blob/master/packages/jest-environment-puppeteer/src/PuppeteerEnvironment.js

Incase it’s helpful, It seemed for me like the page was closed by the time jasmine reporter was called when using one of the above solutions, and overriding it prevents things like it.only from working. Ended up with this riff on @testerez’s code, which takes a screenshot after every test (on a CI env), but only saves the screenshot if the test fails: https://gist.github.com/ajs139/3ddc10e807ee9b94b581c80a762de587

@testerez Thanks for the code! However, I cannot I get undefined error for the pageInstance variable.

I just put all the code (In javascript) in my setupTestFrameworkScriptFile file + a call to registerScreenshotReporter().

I modified the declaration for Vainilla as such:

export const takeScreenshot = (testName, pageInstance = page) => { ... } And page is indeed the name of the page instance in my tests.

In case someone wants to know how to get the test describe name when using jest-circus (https://github.com/smooth-code/jest-puppeteer/issues/131#issuecomment-493267937 saves a screenshot only with the test method name), you can get it like:

const testDescription = state.currentlyRunningTest.parent.name 

I had to spend some time searching in the source code how State is defined inside jest-circus to find that information, hope it helps someone 😄

registerScreenshotReporter should be called in the file you set as setupFilesAfterEnv

@testerez Thanks! Got it working.