puffing-billy: HTTPS proxying does not work in headless Chrome

Chrome 65 is now stable.

HTTPS proxying is working well for us via an approach inspired by @Jack12816.

spec/feature/support/billy_ssl.rb:

module BillySsl
  def add_authority_to_chrome
    puts 'Adding billy CA to chrome'

    cmd = TTY::Command.new(printer: :quiet)
    cmd.run <<~SCRIPT
      cd "#{ENV['HOME']}"
      curl -s -k -o "cacert-root.crt" "http://www.cacert.org/certs/root.crt"
      curl -s -k -o "cacert-class3.crt" "http://www.cacert.org/certs/class3.crt"
      echo > .password
      mkdir -p .pki/nssdb
      CERT_DIR=sql:$HOME/.pki/nssdb
      certutil -N -d .pki/nssdb -f .password
      certutil -d ${CERT_DIR}  -A -t TC \
        -n "CAcert.org" -i cacert-root.crt
      certutil -d ${CERT_DIR} -A -t TC \
        -n "CAcert.org Class 3" -i cacert-class3.crt
      certutil -d sql:$HOME/.pki/nssdb -A \
        -n puffing-billy -t "CT,C,C" -i #{Billy.certificate_authority.cert_file}
    SCRIPT
  end
  module_function :add_authority_to_chrome
end

In spec/feature/feature_helper.rb:

RSpec.configure do |config|
  config.before :suite do
    BillySsl.add_authority_to_chrome
  end
end

In spec/rails_helper.rb:

  Capybara.register_driver :headless_chrome_billy do |app|
    capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(
      # Turn on browser logs
      loggingPrefs: {
        browser: 'ALL'
      }
    )

    options = Selenium::WebDriver::Chrome::Options.new
    options.headless!
    options.add_argument('--disable-gpu')
    options.add_argument("--proxy-server=#{Billy.proxy.host}:#{Billy.proxy.port}")
    options.add_argument('--proxy-bypass-list=127.0.0.1')

    Capybara::Selenium::Driver.new app,
      browser: :chrome,
      options: options,
      desired_capabilities: capabilities,
      driver_opts: {
        # log_path: '/srv/chromedriver.log',
        # verbose: true
      }
  end

  Capybara.javascript_driver = :headless_chrome_billy

  Capybara.server_port = 7787

I’ve now verified that headless chrome works with proxied secure URLs using both browsermob-proxy and mitmproxy. I’ve updated my repro script, which now allows you to run a minimal reproduction against the three proxies:

https://github.com/urbanautomaton/headless_chrome_ssl_proxy/tree/proxy-comparison

By inspecting the traffic flows in wireshark I was able to see that the TLS handshake appears to complete successfully for puffing billy and headless chrome, it’s just that Chrome then immediately drops the connection and retries. I don’t know why this is, or why it would only happen in headless mode.

I’ve run out of time to investigate this, but thought I’d leave the repro and details here in case someone else is interested in following up.

Thanks for the PR @Jack12816! I finally got around to switching to headless Chrome after running into some weird issues with poltergeist today. I was able to skip all of the SSL certificate stuff, because the acceptInsecureCerts option is working for me.

Here’s the versions of everything on my MacBook:

• Mac OS Mojave 10.14
• Ruby 2.5.3p105
• puffing-billy 1.1.2
• capybara 3.10.0
• chromedriver 2.44.609545
• Google Chrome 71.0.3578.80

The specs are also passing on CI (GitLab), running in a Docker container based on Debian jessie. My Dockerfile is based on this one. (Same versions of everything, except chromedriver was slightly updated to 2.44.609551.)

My spec/rails_helper.rb:

Capybara.register_driver :headless_chrome_billy do |app|
    capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(
      acceptInsecureCerts: true,
      loggingPrefs: { browser: 'ALL' }
    )
    options = Selenium::WebDriver::Chrome::Options.new
    options.headless!
    options.add_argument('--disable-gpu')
    options.add_argument('--window-size=1280,1000')
    options.add_argument("--proxy-server=#{Billy.proxy.host}:#{Billy.proxy.port}")
    options.add_argument('--proxy-bypass-list=127.0.0.1;localhost')

    Capybara::Selenium::Driver.new app,
      browser: :chrome,
      options: options,
      desired_capabilities: capabilities,
      driver_opts: {
        log_path: Rails.root.join('log/chromedriver.log').to_s,
        verbose: true,
      }
  end
  Capybara.javascript_driver = :headless_chrome_billy

I didn’t need the --disable-web-security or --no-sandbox options on my Mac. I also read this post:

Note: --no-sandbox is not needed if you properly setup a user in the container.

I set up the the chrome user in my CI Docker image instead of using --no-sandbox, but you might need it: options.add_argument('--no-sandbox').

---

One thing I should mention is that I spent about an hour trying to get this to work, but then it suddenly started working and I don’t really know why.

I was printing the browser logs in one of my specs, which was really helpful: puts page.driver.browser.manage.logs.get(:browser).

When I didn’t have the acceptInsecureCerts: true option, I was getting this error:

SEVERE 2018-12-07 21:15:15 +0700: https://fonts.googleapis.com/css?family=... 
- Failed to load resource: net::ERR_CERT_AUTHORITY_INVALID

After I added acceptInsecureCerts: true, I started getting this error instead:

SEVERE 2018-12-07 21:15:15 +0700: https://fonts.googleapis.com/css?family=... 
- Failed to load resource: net::ERR_TOO_MANY_RETRIES

Unfortunately I can’t remember the exact series of steps that caused the “too many retries” error to disappear, and I can’t reproduce it anymore. I think it might have stopped happening after I deleted spec/req_cache, removed the --headless argument, and ran the tests from scratch in a real browser window. Then I added --headless again, and the tests were still passing and using the cached responses. I ran rm -rf spec/req_cache one more time, and the tests were still passing and correctly caching the responses. So if you run into any problems, try removing spec/req_cache and running without the --headless flag.

@AlanFoster I started a wireshark session on the loopback interface:

To show only the chrome <-> proxy traffic I added a display filter for tcp.port == 8081 (or whatever port the proxy was running on - I found it helpful to fix it as 8081 in the puffing billy config so I could more easily look at old traces).

The most useful view I found was the traffic flow - once you’ve got a recording with a request in go Statistics > Flow Graph, and select “Show: displayed packets” to make it respect the display filter. You should then be looking at the back and forth between chrome and puffing billy.

That’s about the limit of my wireshark expertise - good luck, hope you find out what’s going on. 👍

In the mean time, if you want headless firefox - that works pretty nice:

Capybara.register_driver :selenium_headless_firefox_billy do |app|
  options = ::Selenium::WebDriver::Firefox::Options.new
  options.add_argument('--headless')

  capabilities = Selenium::WebDriver::Remote::Capabilities.firefox(
    acceptInsecureCerts: true,
    proxy: {
      http: "#{Billy.proxy.host}:#{Billy.proxy.port}",
      ssl: "#{Billy.proxy.host}:#{Billy.proxy.port}"
    }
  )

  ::Capybara::Selenium::Driver.new(
    app,
    options: options,
    desired_capabilities: capabilities
  )
end

Note that unlike capybara webkit, firefox makes additional requests to see what version it is on etc, I have disabled this as part of my puffing billy set up.

Headless firefox appears to be pretty slow though, from 5 minutes on poltergeist to 20 minutes on headless firefox, and 18 minutes on normal firefox 🤔

cc @ronwsmith Let me know if we should add headless firefox support to puffing billy out of the box or not 👍

@Aesthetikx Have you had any luck using Chrome 65, chromedriver 2.35, and using both acceptInsecureCerts + Proxy configuration? It seems to hang for me.

I haven’t had any luck with the following configuration:

Capybara.register_driver :selenium_headless_chrome_billy do |app|
  options = Selenium::WebDriver::Chrome::Options.new
  options.add_argument('--headless')
  options.add_argument('--disable-gpu')
  options.add_argument('--no-sandbox')
  options.add_argument("--proxy-server=#{Billy.proxy.host}:#{Billy.proxy.port}")

  capabilities = Selenium::WebDriver::Remote::Capabilities.chrome
  capabilities['acceptInsecureCerts'] = true

  Capybara::Selenium::Driver.new app,
                                 browser: :chrome,
                                 desired_capabilities: capabilities,
                                 options: options
end

I had time to implement a solution for macOS.

https://github.com/Jack12816/puffing-billy/blob/6dc27da3a473c04818f2d3eeeac854de666a94fc/README.md#google-chrome-headless-example

/cc @AlanFoster @adamniedzielski

@Jack12816 haha, yeah - i was aware of that when i was investigating it! The initial goal was to just get it working for the currently paused rspec session, then I’d automate it and post a solution. In the end i had no luck though!

I’ll stay tuned though, let me know if i can do anything to help 👍

Hello! I was having the same problem as you all, but with Cucumber and testing an external application, the BillySsl module works fine, but I had a bit of a problems with password when running the certutil commands, mostly when running more than one time, so, I’ve made some changes, and works like a charm! Here my code in case anyone have the same problem:

module BillySsl
  def add_authority_to_chrome
    puts 'Adding billy CA to chrome'

    cmd = TTY::Command.new(printer: :null)
    cmd.run <<~SCRIPT
      cd "#{ENV['HOME']}"
      curl -s -k -o "cacert-root.crt" "http://www.cacert.org/certs/root.crt"
      curl -s -k -o "cacert-class3.crt" "http://www.cacert.org/certs/class3.crt"
      if ! [ -d .pki/nssdb ]; then
        mkdir -p .pki/nssdb
        certutil -N -d .pki/nssdb --empty-password
      fi
      CERT_DIR=sql:$HOME/.pki/nssdb
      certutil -d ${CERT_DIR}  -A -t TC \
        -n "CAcert.org" -i cacert-root.crt
      certutil -d ${CERT_DIR} -A -t TC \
        -n "CAcert.org Class 3" -i cacert-class3.crt
      certutil -d sql:$HOME/.pki/nssdb -A \
        -n puffing-billy -t "CT,C,C" -i #{Billy.certificate_authority.cert_file}
    SCRIPT
  end
  module_function :add_authority_to_chrome
end

Then in my env.rb file:

BillySsl.add_authority_to_chrome

That’s it, works both with or without headless and works fine in a Docker/Alpine!

Seemingly Chrome 65 with driver 2.35 will support the acceptInsecureCerts flag, https://bugs.chromium.org/p/chromium/issues/detail?id=721739#c95

@AlanFoster Thanks for trying, but each time your test suite run, a new puffing billy root CA is generated. So this won’t work if you import it from your macOS settings dialog. I will look for a programmatic way to do this. Or to use a different certificate store for the new Chrome session. But unfortunately I didn’t found the time for this yet, so stay tuned. 😃