puppeteer: Request interceptions cause web workers network requests to hang

When setting a request interception to a page, the page’s workers can no longer make network request, because otherwise they will stop responding. both importScripts and fetch.

It is critical to mention that this behavior does work on native CDP

Steps to reproduce

I’ve created a repo with a script that does the following, and also shows that a “native” implementation using CDP alone does work:

Serve a static site with worker.js that makes a network call of some sort

Create a new browser page with interceptions:

await page.setRequestInterception(true);
await page.on('request', request => request.continue);

See how it causes the worker to stop responding!

Tell us about your environment:

Puppeteer version: 1.13.0
Platform / OS version: OSX
URLs (if applicable):
Node.js version: v10.15.3

What steps will reproduce the problem?

Please include code that reproduces the issue.

Clone https://github.com/Schniz/puppeteer-workers-bug
npm i
node puppeteer.js

What is the expected result?

The following works:
--------------------
1
{
  "userId": 1,
  "id": 1,
  "title": "delectus aut autem",
  "completed": false
}
finished

But when intercepted, it doesn't!
--------------------------
1

Although it does work with CDP directly:
----------------------------------------
http://localhost:5000/
http://localhost:5000/worker.js
https://jsonplaceholder.typicode.com/todos/1
1
{
  "userId": 1,
  "id": 2,
  "title": "quis ut nam facilis et officia qui",
  "completed": false
}
finished

What happens instead?

The following works:
--------------------
1
{
  "userId": 1,
  "id": 1,
  "title": "delectus aut autem",
  "completed": false
}
finished

But when intercepted, it doesn't!
--------------------------
1

Although it does work with CDP directly:
----------------------------------------
1
{
  "userId": 1,
  "id": 2,
  "title": "quis ut nam facilis et officia qui",
  "completed": false
}
finished

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 6
Comments: 18 (2 by maintainers)

Commits related to this issue

Introduce `--no-request-interception` This works around a puppeteer issue when running scripts within a web worker. See https://github.com/GoogleChrome/puppeteer/issues/4208. — committed to mantoni/mochify.js by mantoni 5 years ago
Introduce `--no-request-interception` This works around a puppeteer issue when running scripts within a web worker. See https://github.com/GoogleChrome/puppeteer/issues/4208. — committed to mantoni/mochify.js by mantoni 5 years ago
Introduce `--no-request-interception` (#207) This works around a puppeteer issue when running scripts within a web worker. See https://github.com/GoogleChrome/puppeteer/issues/4208. — committed to mantoni/mochify.js by mantoni 5 years ago

Most upvoted comments

Having the same issue, importScript() is just handling and not triggering the request interceptor. The workaround in https://github.com/puppeteer/puppeteer/issues/4208#issuecomment-563178898 doesn’t solve the problem if you need to intercept the requests that happen inside the worker.

felixfbecker on Jul 14, 2020

We’re marking this issue as unconfirmed because it has not had recent activity and we weren’t able to confirm it yet. It will be closed if no further activity occurs within the next 30 days.

stale[bot] on Dec 23, 2022

We’re marking this issue as unconfirmed because it has not had recent activity and we weren’t able to confirm it yet. It will be closed if no further activity occurs within the next 30 days.

stale[bot] on Oct 23, 2022

We’re marking this issue as unconfirmed because it has not had recent activity and we weren’t able to confirm it yet. It will be closed if no further activity occurs within the next 30 days.

stale[bot] on Jun 26, 2022

It appears that either something in Puppeteer fixed it or more likely upstream changes. I am not able to reproduce it with 22.6.3

OrKoN on Apr 5, 2024

@sidvishnoi I don’t think there is an upstream bug. I am not quite sure it’s an upstream bug at this point: perhaps the Network.requestWillBeSent is dispatched on the worker target while the Network.requestIntercepted events are dispatched on the page target. I’d be great if someone could dig into it: I will see if I find time.

OrKoN on Jan 12, 2023

I did it! When the desired effect is achieved, remove the intercept

    await page.setRequestInterception(true)
    page.on('request', async function jsRequest(res) {
        if (/modules\/cube-loader\/cube-loader\.min\.js(?=\?.*)/.test(res.url())) {
            const jsCont = getExtendsJs('src/extendsJS/cube-loader.min.js');
            await res.respond({
                status: 200,
                contentType: 'application/javascript; charset=utf-8',
                body: jsCont
            });
            page.removeListener('request',jsRequest);
            await page.setRequestInterception(false)
            return false
        }
        res.continue();
    });

SilurianYang on Dec 9, 2019

It seems that Network.requestWillBeSent is not fired from CDP for the XHR in the worker…

this hack “fixes” it:

--- a/lib/NetworkManager.js
+++ b/lib/NetworkManager.js
@@ -187,6 +187,7 @@ class NetworkManager extends EventEmitter {
       this._requestIdToRequestWillBeSentEvent.delete(requestId);
     } else {
       this._requestHashToInterceptionIds.set(requestHash, event.interceptionId);
+      this._onRequest(event, null);
     }
   }
 
@@ -306,7 +307,7 @@ class Request {
     this._failureText = null;
 
     this._url = event.request.url;
-    this._resourceType = event.type.toLowerCase();
+    this._resourceType = (event.type || event.resourceType).toLowerCase();
     this._method = event.request.method;
     this._postData = event.request.postData;
     this._headers = {};

Not sure what a proper fix to that issue though.

yurynix on Mar 31, 2019