puppeteer: Page throws timeout exception only in headless mode

Tell us about your environment:

Puppeteer version: puppeteer@next (latest commit)
Platform / OS version: macOS High Sierra 10.13.1
URLs (if applicable): http://www.sanident.it
Node.js version: v8.6.0

What steps will reproduce the problem?

  const browser = await puppeteer.launch({
    headless: true,
  });

  const page = await browser.newPage();
  await page.goto("http://www.sanident.it", {timeout: 10000});
  console.log(await page.content());

What is the expected result?

Promise does not throw “Navigation Timeout Exceeded” no matter if using headless or non-headless mode.

What happens instead?

Promise throws “Navigation Timeout Exceeded” ONLY in headless mode. In non-headless it works fine. This does not work also in v0.13.0 BUT IT WORKS in v0.12.0

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 8
Comments: 15 (2 by maintainers)

Most upvoted comments

Some websites block headless useragent. You might try to change that.

await page.setUserAgent('Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36');
await page.goto('http://www.sanident.it/');

+64

kirosc on Mar 4, 2020

I ended up on this thread because I was doing page.goto('localhost:3000') instead of page.goto('http://localhost:3000'). The protocol seems mandatory in headless mode. Hope this helps

VincentCharpentier on Jan 4, 2019

Can confirm this with puppeteer 2.1.1 running in windows and mac.

const puppeteer = require('puppeteer');
const url = 'https://www.youtube.com/channel/UCxhygwqQ1ZMoBGQM2yEcNug/videos';

async function run() {
  try {
    const browser = await puppeteer.launch({ headless: true });
    const page = await browser.newPage();
    await page.goto(url, { waitUntil: 'networkidle2' });
    await page.waitForSelector('#items');
    const html = await page.evaluate(() => document.body.innerHTML);
    // Throws TimeoutError when headless is set to true
  } catch (e) {
    console.log(e);
  }
}

run();

wobsoriano on Feb 7, 2020

This should be documented somewhere.

In headless mode browser cache is PRESERVED, so you can’t catch all of your precious requests! But in ‘headful’ mode cache somehow gets reset. Just do this:

const goto = async url => {
  const page = await browser.newPage()
  await page.setCacheEnabled(false) // <-- this one is a game changer
  await page.goto(url)
  return page
}

// somewhere
await goto('http://MY_URL:666')

yyynnn on Sep 30, 2019

I can confirm that this timeouts in headless and not headful mode. Perhaps the site does some kind of headless/automation sniffing and something is blocking the page from finishing?

ebidel on Feb 2, 2018

In my case I forced the resolution I used in development and it worked normally

     const browser = await puppeteer. Launch({
                headless: true,
                defaultViewport: { width: 1280, height: 1600 },
                userDataDir: "./tmp",
        });

fclebio on Nov 4, 2022

I had same error, but navigation was caused by click event. Resolved according to the API document, with sample code snippet:

const [response] = await Promise.all([
  page.waitForNavigation(waitOptions),
  page.click(selector, clickOptions),
]);

zhanghuanchong on Feb 9, 2018

I can confirm, setting the defaultViewPort at launch solved this issue for me. Thanks @fclebio 🙏🏽

john-doherty on Jan 17, 2023