playwright: [REGRESSION]: chromium.connect does not work with vanilla CDP servers anymore

Hi there, apologies if this has been asked before but I didn’t find anything.

Up until v1.3.0 it was possible to .connect to a vanilla browser websocket URL (either created through launching Chrome with --remote-debugging) or using Puppeteer).

v1.4.0 introduced the new “New Client / Server Wire protocol”, since then it’s not possible to use .connect with other original CDP websocket servers anymore (it will hang here). I also noticed through logging that the new wire protocol looks much different from the regular CDP chatter.

My question would be if there’s still a way to use chromium.connect with the traditional CDP websocket URLs or if there are pointers on how to best write a translation layer to accomplish that.

My use-case is to have a single websocket url which exposes a fleet of chromium browsers by proxying the websocket connections (similar to browserless.io) and both puppeteer & playwright being able to connect to it and control the browsers.

The following worked prior to v1.4.0:

const puppeteer = require("puppeteer")

puppeteer
  .launch({
    headless: true,
    defaultViewport: null,
    args: [
      "--remote-debugging-port=9222",
      "--remote-debugging-address=0.0.0.0",
    ],
  })
  .then(async (browser) => {
    console.log(browser._connection.url())
  })
// => ws://0.0.0.0:9222/devtools/browser/490a2b32-ce62-4b6b-a530-d4931fdeb046

const pw = require("playwright")

;(async () => {
  const browser = await pw.chromium.connect({
    wsEndpoint: "ws://0.0.0.0:9222/devtools/browser/490a2b32-ce62-4b6b-a530-d4931fdeb046",
  })

  const context = await browser.newContext()
  const page = await context.newPage()

  await page.goto("https://www.example.com/")
  await page.screenshot({ path: "example.png" })

  await browser.close()
})()

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 12
Comments: 16 (7 by maintainers)

Most upvoted comments

@pavelfeldman Not OP, but I want to highlight that there are various reasons to connect to existing Chrome CDP websockets that are not launched by Playwright. For example, if you have a scalable Chrome (e.g. k8s) cluster that already works with Puppeteer/Playwright. Or if you use Playwright with a third party infra solution like Browserless (see comment above of @joelgriffith) or HeadlessTesting.

It would be awesome if there would be a way to call launchServer() with an existing CDP websocket for these usecases.

Niek on Oct 6, 2020

@pavelfeldman thanks for going over this. I think the things you’ve outlined make a lot of sense, in particular the network traffic (an almost 10x drop in network chatter is huge). This is a big one for us.

For us, the problem is maintenance burden as we support playwright, puppeteer, selenium as well as our own REST APIs. Adding the new playwright server was quick, but over the long term the cost does add up, so it’s not a trivial change. As much as the change was a bit of a shock, I think it’s worth the burden given what you’ve said.

A few things that would be nice:

Support for user-data-dir in browser-server. Not sure what the complications are there since it’s not part of the current API.
Allow for a newContext event so we can inject network interceptors for ad-blocking in our stack: https://github.com/browserless/chrome/blob/master/src/chrome-helper.ts#L139. There’s a few other things as well here, but it sounds like playwright takes care of them mostly, so just this ad-blocker would be nice.

joelgriffith on Oct 6, 2020

@dgozman Thanks a lot for the quick response on that matter. 😃

Oh wow, that’s unfortunate to hear. The CDP websocket has always felt like the underlying and unifying standard or common denominator that all CDP based automation frameworks could interface with.

Note: I really appreciate the efforts of the Playwright team and the project contributors and can understand why switching to a custom wire protocol was desired.

Having that said, I wonder what the best course of action would be if someone would be determined enough to develop a custom websocket server compatible with Playwright wire protocol and vanilla CDP clients.

In my scenario I have full control over the websocket server consumed by pptr/pw clients but the server itself has to connect to existing browsers through vanilla CDP (playwright cannot be used to launch them).

Would this be feasible:

Modify launchServer to not launch a browser but use an existing one through CDP (should be possible as internally Playwright still uses the vanilla CDP websocket)
Develop a custom websocket server that can differentiate between Playwright and Puppeteer clients (through e.g. the initial handshake or similar) and forward the websocket frames to either the custom Playwright websocket or the vanilla CDP socket

Thanks a lot for your time and input, much appreciated. 😃

berstend on Oct 6, 2020