puppeteer: Puppeteer with headless:true is extremely slow

Hey guys, so I’ve found a workaround for this. I found the answer in another thread unrelated to puppeteer but also has the problem with chromium being extremely slow in headless mode. I noticed that when I tried to debug chromium that network requests were the slow bit in terms of headless mode so I searched for that.

const browser = await puppeteer.launch({args: ["--proxy-server='direct://'", '--proxy-bypass-list=*']})

When launching the browser add these 2 arguments to the list and it seems (for me anyway) to resolve all issues with speed.

Link to thread if you’re interested: https://github.com/Codeception/CodeceptJS/issues/561 and props to oligee80 (who found the answer) for saving me about a week of work rolling back a bunch of product releases! 🎉

Try setting a valid UserAgent. This worked for me.

await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36')

Hey guys, so I’ve found a workaround for this. I found the answer in another thread unrelated to puppeteer but also has the problem with chromium being extremely slow in headless mode. I noticed that when I tried to debug chromium that network requests were the slow bit in terms of headless mode so I searched for that.

const browser = await puppeteer.launch({args: ["--proxy-server='direct://'", '--proxy-bypass-list=*']})

When launching the browser add these 2 arguments to the list and it seems (for me anyway) to resolve all issues with speed.

Link to thread if you’re interested: Codeception/CodeceptJS#561 and props to oligee80 (who found the answer) for saving me about a week of work rolling back a bunch of product releases! tada

We have a proxy, this fixed the issue for me.

I’m seeing this same issue when running in a Windows service using node-windows. Runs pretty fast with {headless: false}, but times out after 30s almost every time when run with {headless: true}. The same app runs very fast as a standalone node app from the command line.

Try setting a valid UserAgent. This worked for me.

await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36')

Thank you. Mine is macOS, in headless mode set a userAgent like this is 5x faster! 👍

macOS: 10.13.4 Node: 8.10.0 Puppeteer: 1.4.0

I am also seeing a horrendous amount of time added to my requests when running headless: true. Here is a simple code example;

(async () => {
  const browser = await puppeteer.launch({ headless: true });
  const promises = [];
  console.time('Timer');
  for (let i = 0; i < 10; i++) {
    promises.push(renderPage(browser));
  }
  await Promise.all(promises);
  console.timeEnd('Timer');
})();

const renderPage = async (browser: puppeteer.Browser) => {
  const page = await browser.newPage();
  await page.setViewport({ width: 1920, height: 1080 });
  await page.setContent('<h1>Some really simple HTML</h1>');
  const screenshot = await page.screenshot({ fullPage: true }); // Sets buffer but isn't used in this example.
  await page.close();
};

Running the above code with headless: true results in 7747.209ms Running the above code with headless: false results in 4775.771ms

This is a HUGE difference in performance. Obviously I am working with more complex HTML than a simple <h1> tag and the disparity between the two modes is far greater.

I solved this using the flag: –deterministic-fetch https://peter.sh/experiments/chromium-command-line-switches/#deterministic-fetch

@parthasarathydNU it’s because you use Puppeteer 21.0.1 and you use ‘new’ headless. Read more https://developer.chrome.com/articles/new-headless/

Edit: I was just curious of knowing if these flags https://github.com/puppeteer/puppeteer/issues/1718#issuecomment-397532083 could be taken away in ‘new’ headless. Your results are promising, ty!

very strange… i found this similar case but resolved lately after using a valid user agent.

I been getting conflicting results for different websites that I hit. Sometimes headless takes ages longer than headful, other times about the same. Here is my experiment.

methodology I loaded the same page 100 times in both headful and headless mode and stored the results for time of different stages.

I’ve removed the timekeeping code (and lots of other code) for legibility. You’ll also note that I’ve commented out the ublock extension, which is necessary for blocking out video and other ads that cause the second page to not load otherwise. (The extension works fine in headful mode)

const puppeteer = require('puppeteer');
const fs = require('fs');
var is_ok = false

var headless = true

(async () => {
    for (i = 0; i < 100; i++) {
        await main()
    }

    process.exit(Number(!is_ok));
})();

async function main() {
    const start = new Date()
    // const ext = __dirname + '/ublock';
    // const datadir = __dirname + '/ublock_data';
    const browser = await puppeteer.launch({
        ignoreHTTPSErrors: true,
        dumpio: false,
        // userDataDir: datadir,
        headless: headless,
        args: [
            '--no-sandbox',
            '--disable-setuid-sandbox',
            '--ignore-certificate-errors',
            '--proxy-server="direct://"',
            '--proxy-bypass-list=*',
            // `--disable-extensions-except=${ext}`,
            // `--load-extension=${ext}`
        ]
    });
    // browser_open time

    const page = await browser.newPage();

    await page.setViewport({
        width: 960,
        height: 800
    });
    await page.goto(url, {
            waitUntil: 'networkidle2',
            timeout: 60000,
        }).then((response) => {
            is_ok = response.ok;
        })
        .catch((error) => {
            console.error('died with Error:', error.toString());
        });

    // pageload time

    await page.screenshot({
            path: filepath + ".jpg",
            type: 'jpeg',
            fullPage: true,
        }).catch((error) => {
        console.error('Died on await target.screenshot:', error);
    });

    await browser.close().catch((error) => {
        console.error('Died on await browser.close():', error);
    });

    // screenshot_time
    // total_time
};

Results

Test 1: https://techcrunch.com/2016/09/12/slacks-director-of-engineering-leslie-miley-doesnt-believe-in-diversity-quotas/ (No ublock)

Headful: Headless:

There is no significant difference between headful and headless mode.

Test 2: http://www.pcworld.com/article/3117032/software-productivity/microsoft-may-finally-have-its-slack-killer.html (with ublock)

Headful: Headless:

Where a quarter of screengrabs passed the 60s timeout.

conclusion Other websites I tested had headless execution on par with headful, even when including ublock. Though these other websites were not quite as ad-heavy.

I’m not sure why this one site in particular behaves how it does. It looks like the headful browser helps the ublock extension close off ads faster or something like that.

So yeah, not sure what to do with this information myself, but… here you go.

@ChiggerChug I try that and it works,thank you~~