puppeteer: page.goto never resolves when page.setRequestInterception is used (for some urls)

Hi again, Thanks as usual for providing this library 👏. Unfortunately I am back with another page.setRequestInterception regression. 😕 It’s pretty much the same as https://github.com/GoogleChrome/puppeteer/issues/662, but since that was fixed back then I thought I’d open a new issue instead. I also saw this new issue which sounds like it’s the same thing, but it’s been closed and info removed, so cannot continue from.

What happens here is just simply that for some URLs, when page.setRequestInterception is used, page.goto never resolves. (And yes, I am calling continue on intercepted requests).

Steps to reproduce

Tell us about your environment:

Puppeteer version: 1.7.0 1.6.2 did not have this issue
Platform / OS version: MacOS 10.13.6 (17G65)
URLs (if applicable): f.e. https://csms.org/tag/telehealth This url I discovered this problem for, but it’s a pattern I see applies for other urls as well (only some though). The url is not mine.
Node.js version: v9.11.2

What steps will reproduce the problem? Run the code below.

const puppeteer = require('puppeteer');

(async() => {
const browser = await puppeteer.launch({
  ignoreHTTPSErrors: true
})
const page = await browser.newPage()

// With page.setRequestInterception(true)
// page.goto never resolves (below)
await page.setRequestInterception(true)
page.on('request', interceptedRequest => {
  interceptedRequest.continue()
})

console.log('goto start...')
await page.goto('https://csms.org/tag/telehealth')
console.log('this never happens')

browser.close();
})();

What is the expected result? The page loads. Using setRequestInterception should not have any impact on whether a page loads or not, at least not if continue is called on every intercepted request.

What happens instead? The page never loads.

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 15
Comments: 17 (6 by maintainers)

Commits related to this issue

downgrade to puppeteer@1.6.2; 1.7 causes new timeouts for some sites Raised an issue for this here: https://github.com/GoogleChrome/puppeteer/issues/3118 — committed to pocketjoso/penthouse by pocketjoso 6 years ago
workarkound for https errors and setRequestInterception problems as suggested here: https://github.com/GoogleChrome/puppeteer/issues/3118#issuecomment-417754246 seems to work better even with puppet... — committed to pocketjoso/penthouse by pocketjoso 6 years ago

Most upvoted comments

Make sure req.continue() is running in the interceptor.

vdkkia on Mar 2, 2021

@pocketjoso First of all, I’m very sorry we regressed your scenario. The reason it doesn’t work any more is because Chrome 70 no longer trusts Symantec certificates. As a result, you’re hitting our long-standing bug #1159.

It’s unfortunate that new releases break behaviors like this. I know the web is wide and wild, and it’s hard to foresee every kind of crazy content that exists out there… but do you have test coverage for these kind of flows?

As you said, web is wide and wild, and we’re running Puppeteer with the tests-first mantra. Every time we break something we add a regression test so that it doesn’t happen in future; over time, the project will become more and more reliable.

Regarding the #1159 - Chromium is slowly completing migration to network service, so once it’s their, I’ll be the first to open champagne. For now, you can workaround this by enabling network service manually.

The following works for me:

const puppeteer = require('puppeteer');

(async() => {
  const browser = await puppeteer.launch({
    args: ['--enable-features=NetworkService'],
    ignoreHTTPSErrors: true
  })
  const page = await browser.newPage()
  await page.setRequestInterception(true)
  page.on('request', interceptedRequest => {
    interceptedRequest.continue()
  })
  console.log('goto start...')
  await page.goto('https://csms.org/tag/telehealth')
  console.log('this never happens')
  browser.close();
})();

Thank you for your reports. Closing as dupe of #1159.

aslushnikov on Aug 31, 2018

@aslushnikov still not working here… anything specific I should do?

    const browser = await puppeteer.launch({
        headless: false,
        ignoreHTTPSErrors: true,
        args:[
            '--window-size=1280,768',
            '--no-sandbox',
        ] 
    });
    const page = await browser.newPage();
    page.setViewport({width:1280, height:768});

    await page.setRequestInterception(true);
    page.on('request', async interceptedRequest => {

    ...

jlcd on Oct 19, 2018

For anyone else tracking requests for a short period of time (not the whole run) – it’s important to turn setRequestInterception back off when you’re done, else all you’re requests will hang for no apparent reason.

This not a bug, but a point of confusion.

Here’s an example of what to do:

const promises = [];
const onRequest = (request) => promises.push(request.continue());
await page.setRequestInterception(true);
page.on('request', onRequest);
// .....
await Promise.all(promises);
page.removeListener('request', onRequest);
await page.setRequestInterception(false);

chaseholland on Jan 20, 2021

From what I saw my issue will be fixed with #3471, so I’ll keep an eye on that one. Thanks.

jlcd on Nov 13, 2018

@pocketjoso yes, the #1159 is now fixed for non-network-service codepath.

aslushnikov on Sep 16, 2018

Do you have any link for more information about the NetworkService work in Chrome?

@pocketjoso network service is a part of a “servicification effort”, you can read more about it here.

aslushnikov on Sep 2, 2018

Thanks for this answer @aslushnikov, it seems to improve things even in 1.6.2 where I also started seeing some similar problems. Will test more, then upgrade to 1.7 and test more again. 😃

Do you have any link for more information about the NetworkService work in Chrome? I have not heard of this before.

pocketjoso on Sep 2, 2018