chrome-remote-interface: Error: No data found for resource with given identifier

This happens because AFAIK you’re only allowed to call Network.getResponseBody when the Network.loadingFinished event has fired. Unfortunately this event doesn’t contain the associated request object so you have to keep track of the requestId for which you want to fetch the response body.

I implemented this in the following using a Set:

const CDP = require('chrome-remote-interface');

setTimeout(() => {
    CDP(async (client) => {
        const {Network, Page, Runtime} = client;

        const requests = new Set(); // <---------- HERE

        Network.requestWillBeSent(({requestId, request}) => {
            if(request.url.indexOf("ct2/results/rpc") != -1){
                console.log(`REQ [${requestId}] ${request.method} ${request.url} \n`);

                requests.add(requestId); // <---------- HERE

            }
        });
        Network.loadingFinished(async ({requestId}) => {

            if (requests.has(requestId)) { // <---------- HERE

                const {body, base64Encoded} = await Network.getResponseBody({requestId});
                console.log(`RES [${requestId}] body: ${body} \n`);
            }
        });
        try {
            await Promise.all([Network.enable(), Page.enable()]);
            await Page.navigate({url: 'https://clinicaltrials.gov/ct2/results?cond=Parents&term=&cntry1=&state1=&Search=Search&recrs=a#wrapper'});
            await Page.loadEventFired();
            await Runtime.evaluate({
                expression: `document.querySelector('.paginate_button.next').click()`
            });
        } catch (err) {
            console.error(err);
        }
    }).on('error', (err) => {
        console.error(err);
    });
}, 1000);

@cyrus-and @ilanc it seems i managed to do what I wanted. However I didn’t use Network.getResponseBody. I used https://chromedevtools.github.io/devtools-protocol/tot/Fetch/ To use that one need to subscribe for Responses matching a pattern. Then you can react on Fetch.requestPaused events. During that you have direct access to request and indirect to response. To get the response call Fetch.getResponseBody with proper requestId. Below I’ve pasted a snippet.

dbg.sendCommand('Fetch.enable', {
            patterns: [
                { urlPattern: interestingURLpattern, requestStage: "Response" }
            ]})

   var getResponseJson = async (requestId) => {
         const res = await dbg.sendCommand("Fetch.getResponseBody", {requestId: requestId})
         return JSON.parse(res.base64Encoded ? Buffer.from(res.body, 'base64').toString() : res.body)
     }
    dbg.on('message', (e, m, p) => {
        if(m === 'Fetch.requestPaused') {
            var reqJson = JSON.parse(p.request.postData)
            var resJson = await getResponseJson(p.requestId)
            ...

        await dbg.sendCommand("Fetch.continueRequest", {requestId: p.requestId})
        }
    });

Also remember to send Fetch.continueRequest as

The request is paused until the client responds with one of continueRequest, failRequest or fulfillRequest

https://chromedevtools.github.io/devtools-protocol/tot/Fetch/#event-requestPaused

Thank you very much for your help, only you are helping people)

@xgj1988 you need to keep track of the actual request URL, e.g., using a Map. In a nutshell:

• in Network.requestWillBeSent associate request.url to requestId;
• in Network.loadingFinished fetch the URL using requestId as key.

This is off topic though, file a new issue if needed.