puppeteer: Page with CSP breaks puppeteer

+1 for ability to disable CSP.

Use case: I’m using puppeteer as a post-build crawler to generate a static dump of my SPA. When crawling, I’d like to inline critical CSS, and add a CSP directive for the CSS text hash. While I’m at it, I want to change the API server from localhost:port to api.example.com. However, if my page already had a CSP meta tag in it (which is good to have during development so I can catch CSP violations early), puppeteer starts getting funky. Would love it if I can disable CSP altogether when crawling.

Here’s my workaround:

  import fetch from 'node-fetch'

  const requestInterceptor = async (request) => {
    const url = request.url()
    const requestHeaders = request.headers()
    const acceptHeader = requestHeaders.accept || ''
    if (url.includes('some-website-with-csp.com') && (acceptHeader.includes('text/html'))) {
      const cookiesList = await page.cookies(url)
      const cookies = cookiesList.map(cookie => `${cookie.name}=${cookie.value}`).join('; ')
      delete requestHeaders['x-devtools-emulate-network-conditions-client-id']
      if (requestHeaders.Cookie) {
        requestHeaders.cookie = requestHeaders.Cookie
        delete requestHeaders.Cookie
      }
      const theseHeaders = Object.assign({'cookie': cookies}, requestHeaders, {'accept-language': 'en-US,en'})

      const init = {
        body: request.postData(),
        headers: theseHeaders,
        method: request.method(),
        follow: 20,
      }
      const result = await fetch(
        url,
        init,
      )
      const resultHeaders = {}
      result.headers.forEach((value, name) => {
        if (name.toLowerCase() !== 'content-security-policy') {
          resultHeaders[name] = value
        } else {
          console.log('CSP', `omitting CSP`, {originalCSP: value})
        }
      })
      const buffer = await result.buffer()
      await request.respond({
        body: buffer,
        resultHeaders,
        status: result.status,
      })
    } else {
      request.continue();
    }
  }

  await page.setRequestInterception(true)
  page.on('request', requestInterceptor)

Related: #599

+1 for ability to disable CSP.

Thank you.

+1 for fixing this.

Upgrade to v1.0.0, fails all my tests too, due to the same reason (all WaitFor… methods failing due to CSP) Rolling back to v0.13.0.

Thanks @niieani — your solution inspired this simplified approach for a backstopjs usecase https://github.com/garris/BackstopJS/commit/67801c259f340c986bd0c25281a7eb5a79c7bc25

@aslushnikov looking forward to #2324! 🎁

Great to hear it was useful @garris! Also looking forward to #2324 😃

Thanks @aslushnikov! I’ve got tests that were passing with an interception workaround (thanks @niieani for the idea)… removed the interception, installed from your fork and the tests still pass. 😃

Just downgrade to version 0.13, it works great. Waiting for update here

Thanks @andrewlinfoot ! I use typescript too, so it’s perfect!

When trying to inject a Javascript file, there’s no need to downgrade here. Simply replicate the 0.13.0 previous code, such as:

  async injectFile(page, filePath) {

    let contents = await new Promise((resolve, reject) => {
      fs.readFile(filePath, 'utf8', (err, data) => {
        if (err) return reject(err);
        resolve(data);
      });
    });
    contents += `//# sourceURL=` + filePath.replace(/\n/g,'');
    return page.mainFrame().evaluate(contents);
  }

Was never CSP in the headers at all… was meta tags with CSP who was the culprit.