karma: Karma 1.6 breaks Headless support for Chrome

Well I think I found what was causing the issue for me. In my case I noticed that there was a connection that was disconnecting too soon.

I found that in /static/karma.js, when the socket is created, there is a timeout value that is hardcoded to 2 seconds (see below). I just added another 0 to make it 20 seconds and the connection stayed open long enough for the server to respond to the initial request. https://github.com/karma-runner/karma/blob/e79463b94ff6d3ad87526b3c68b38b90e924ea42/client/main.js#L14-L20

The next problem I faced was that Karma thought there was no activity even though there was traffic going back and forth on the socket. To fix that I just added browserNoActivityTimeout: 60000 to the Karma configuration.

Is there anyway the socket timeout can be configurable?

Got into this with karma v1.7.0 Google Chrome 59.0.3071.86

bash -c "node_modules/karma/bin/karma start --browsers ChromeHeadless_without_security ./karma.conf.js --singleRun"
06 06 2017 08:15:59.369:INFO [karma]: Karma v1.7.0 server started at http://0.0.0.0:9876/
06 06 2017 08:15:59.371:INFO [launcher]: Launching browser ChromeHeadless_without_security with unlimited concurrency
06 06 2017 08:15:59.378:INFO [launcher]: Starting browser ChromeHeadless_without_security
06 06 2017 08:15:59.641:INFO [HeadlessChrome 0.0.0 (Linux 0.0.0)]: Connected on socket JoIAImqq8pNor1vAAAAA with id 85283426
HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 0 of 398 SUCCESS (0 secs / 0 secs)
06 06 2017 08:16:11.411:WARN [HeadlessChrome 0.0.0 (Linux 0.0.0)]: Disconnected (1 times), because no message in 10000 ms.
HeadlessChrome 0.0.0 (Linux 0.0.0) ERROR
HeadlessChrome 0.0.0 (Linux 0.0.0) ERROR
  Disconnected, because no message in 10000 ms.
HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 0 of 398 DISCONNECTED (10.004 secs / 0 secs)
HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 0 of 398 DISCONNECTED (10.004 secs / 0 secs)

• Karma version (output of karma --version): 1.7
• Relevant part of karma.config.js file

browsers: ['Chrome, ChromeHeadless, ChromeHeadless_without_security'],
    customLaunchers: {
      ChromeHeadless_without_security: {
        base: 'ChromeHeadless',
        flags: ['--disable-web-security']
      }
    }

I am seeing this on both Windows (7,10) and Linux in Karma v2 and v1.7.1. A google search suggests that this has been an outstanding issue for quite some time, would be great to get a definitive fix.

Still happening with Angular v6 and cli v6.0.

I got exactly same error as @mskutin with same versions of Karma and Chrome. Can’t reproduce it on OS X or Docker container running in Docker for Mac. Although happens every time in Docker container running on build server (Cent OS 7.3).

Investigation showed that it is related to the slower machine and happens, because it took >10 seconds before test bundle was parsed and executed by Chrome and therefore test run was started and communicated back to Karma server. Reasons why it may take long vary.

There are 2 ways to handle timeout:

1. Investigate why your test bundle loads >10 seconds and make sure it loads faster.
2. Increase browserNoActivityTimeout to highter value, so test bundle has enough time to load.

This particular appearance of timeout does not seem to be a Karma issue, but rather problem in the project or misconfiguration.

I’m wondering, is there any maintainer on this project?

This issue filed almost year ago, and now it is second top commented issue in the project.
The reason of the issue was already found by @daryl0313 six months ago.
People cannot use headless chrome through karma because of this.

What’s the problem to fix it? Any response from maintainer/contributors would be good.

@daryl0313 that fixed it for me. But someone should examine, why it takes so long to connect instead of just checking in a work-around

same for me

Chrome v61, karma v1.7.1 (and v1.5.0), windows 7

WARN [launcher]: ChromeHeadless have not captured in 180000 ms, killing.

not sure if this is right place to complain, since i’ve tried karma v1.5.0 which did not work for me either.
confused whether reason is chrome version or karma version or os version or certain combination of them…

This might not add much value to the conversation but I’ll post my workaround anyways.

My use case is testing an Angular app. Karma 2.0.0 + karma-chrome-launcher 2.2.0, and running the tests on headless chrome. I installed karma-spec-reporter to see what test was causing the crash and it turned out that there was a test calling the saveAs method from the package file-saver, thus trying to download file on test execution. (I know, noob mistake). So I added a spy to the saveAs method and just stubbed the call. Problem solved.

Tried all the above suggestions. None worked for me. The test runner times out without executing any tests on chrome headless on Ubuntu Server 14.04.5 LTS. Why does this happen? Any workaround? Note: The tests run perfectly on my OSX machine [Sierra] with karma, karma-chrome-launcher, with headless chrome setup. The problem is just on the linux server.

Error:

25 09 2017 13:56:19.075:INFO [karma]: Karma v1.5.0 server started at http://0.0.0.0:9876/build/
25 09 2017 13:56:19.076:INFO [launcher]: Launching browser ChromeHeadless with unlimited concurrency
25 09 2017 13:56:19.080:INFO [launcher]: Starting browser Chrome
25 09 2017 13:56:19.263:INFO [HeadlessChrome 0.0.0 (Linux 0.0.0)]: Connected on socket zDM7md2A7Je8kqsPAAAA with id 1679961
25 09 2017 13:58:19.552:WARN [HeadlessChrome 0.0.0 (Linux 0.0.0)]: Disconnected (1 times), because no message in 120000 ms.
HeadlessChrome 0.0.0 (Linux 0.0.0) ERROR
  Disconnected, because no message in 120000 ms.

Here is my configuration:

const ChromiumRevision = require('puppeteer/package.json').puppeteer.chromium_revision;
const Downloader = require('puppeteer/utils/ChromiumDownloader');
const revisionInfo = Downloader.revisionInfo(Downloader.currentPlatform(), ChromiumRevision);

process.env.CHROME_BIN = revisionInfo.executablePath;
module.exports = function (config) {
    config.set({
        basePath: 'build/',
        frameworks: ['jasmine'],
        plugins: [
            require('karma-jasmine'),
            require('karma-chrome-launcher'),
        ],
        customLaunchers: {
            'ChromeHeadless': {
                base: 'Chrome',
                flags: [
                    '--headless',
                    '--disable-gpu',
                    // Without a remote debugging port, Google Chrome exits immediately.
                    '--remote-debugging-port=9222'
                ],
                debug: true
            }
        },
        autoWatch: false,
        browsers: ['ChromeHeadless'],
        singleRun: true,
        browserNoActivityTimeout: 120000,
        urlRoot: '/build/'
    })
}

I have logged an issue on karma-chrome-launcher repo here

Using @swetapatil1 answer, I am now able to finally work!

Using the flags from this:

/home/jenkins/workspace/node_modules/puppeteer/.local-chromium/linux-579032/chrome-linux/chrome --user-data-dir=/tmp/karma-62557134 --no-default-browser-check --no-first-run --disable-default-apps --disable-popup-blocking --disable-translate --disable-background-timer-throttling --disable-renderer-backgrounding --disable-device-discovery-notifications --disable-gpu --headless --no-sandbox --remote-debugging-port=9222 http://localhost:9876/?id=62557134

You may also want to try the same command but without --headless to see what happens. (Note that the URL will probably be invalid.)

I was finally able to forge a working karma.conf.json (for headless ubuntu+chrome):

karma.conf.json

module.exports = function (config) {
  config.set({
    basePath: '',
    frameworks: ['jasmine', '@angular-devkit/build-angular'],
    plugins: [
      require('karma-jasmine'),
      require('karma-chrome-launcher'),
      require('karma-jasmine-html-reporter'),
      require('karma-coverage-istanbul-reporter'),
      require('@angular-devkit/build-angular/plugins/karma')
    ],
    client:{
      clearContext: false, // leave Jasmine Spec Runner output visible in browser
      jasmine: {
        random: false
      }
    },
    coverageIstanbulReporter: {
      dir: require('path').join(__dirname, 'coverage'),
      reports: [ 'html', 'lcovonly' ],
      fixWebpackSourcePaths: true,
      thresholds: {
        statements: 85,
        lines: 85,
        branches: 55,
        functions: 80
      }
    },
    captureTimeout: 210000,
    browserDisconnectTolerance: 3,
    browserDisconnectTimeout : 210000,
    browserNoActivityTimeout : 210000,
    reporters: ['progress', 'kjhtml'],
    port: 9876,
    colors: true,
    logLevel: config.LOG_DEBUG,
    autoWatch: true,
    browsers: ['ChromeHeadlessNoSandbox'],
    customLaunchers: {
      ChromeHeadlessNoSandbox: {
        base: 'ChromeHeadless',
        flags: [
                '--no-sandbox',
                '--disable-gpu',
                '--enable-logging',
                '--no-default-browser-check',
                '--no-first-run',
                '--disable-default-apps',
                '--disable-popup-blocking',
                '--disable-translate',
                '--disable-background-timer-throttling',
                '--disable-renderer-backgrounding',
                '--disable-device-discovery-notifications',
                '--remote-debugging-port=9222',
                '--disable-web-security'
        ]
      }
    },
    singleRun: true
  });
};

@Chido42 disabling VirtualBox network adapters fixed it IMMEDIATELY for me. Thanks so much. Two days of digging on this. You’re the best!

I’ve been having this issue with karma 2 with angular-cli, downgrading to 1.7.1 solved this for me

This issue is quite annoying. I’ve wasted a few hours trying to poke around it, but testing in Chrome just will not work. The only solution for us was to use Firefox in our Docker image - worked without any issues at all. We didn’t even have to set browserNoActivityTimeout - it connected right after our angular application was built.

To anybody who might be having this issue - if it’s not of absolute must for you to be using Chrome - safe your time on a wasted cause. Apparently this issue needs more attention from the contributors/owners!

Ping @maksimr, @devoto13 anybody else ?

@artin-phares Well, I feel the solution proposed by @daryl0313 was more a workaround than a permanent fix. Nobody still nailed the issue.

I’m also concern about the lack of response of any maintainer/contributor. As I said, we could have made a PR long time ago with the proposed workaround, but I doubt anybody would have merged it since the issue probably lies somewhere else.

Still, someone more involved in Karma DEV should have a look at this. That being said, I didn’t have a chance of trying it with the Karma 2.x version, wonder if it fixes anything, I know they updated a lot of deps and the issue seems to came from SocketIO somehow…

After a long investigation the error was due to having Virtualbox installed with the VirtualBox Host-Only Network interface enabled.

Once this network interface disabled no more issue. From our investigation it seems this interface slows down the connection a lot and since the timeout to connect it hardcoded inside the lib it fails to connect.

While changing the hardcoded timeout value (by a huge number) it also solved the issue. Not sure if an issue should be also opened to virtualbox.

@bas-l @MikaAK That’s weird seems like 2.0.0 + karma-chrome-launcher 2.2.0 solve it for me.

@applecool

Just managed to get puppeteer to work, inside a docker container.

Just note that, if using docker, you must run npm install in the container.

If not running docker, make sure your host got all the libraries installed.

Here are the relevant bits:

package.json

    "karma": "0.13.19",
    "karma-chrome-launcher": "^2.2.0",
    "karma-mocha": "0.2.1",
    "puppeteer": "^0.10.2",

karma.conf.js:

// Chrome puppeteer support: https://github.com/karma-runner/karma-chrome-launcher/tree/0ef4ee4f0b3d97c778bd02546504c6bdc7caec84#headless-chrome-with-puppeteer
const ChromiumRevision = require('puppeteer/package.json').puppeteer.chromium_revision;
const Downloader = require('puppeteer/utils/ChromiumDownloader');
const revisionInfo = Downloader.revisionInfo(Downloader.currentPlatform(), ChromiumRevision);
process.env.CHROME_BIN = revisionInfo.executablePath;

    browsers: [
      'ChromeHeadless_custom',
    ],

    customLaunchers: {
      'ChromeHeadless_custom': {
        base: 'ChromeHeadless',
        flags: [
          // We must disable the Chrome sandbox when running Chrome inside Docker (Chrome's sandbox needs
          // more permissions than Docker allows by default)
          // Also: https://github.com/GoogleChrome/puppeteer/issues/560
          '--no-sandbox',
          '--disable-setuid-sandbox',
        ],
      },
    },

Dockerfile

FROM node:8

WORKDIR /html5-client/

# https://github.com/GoogleChrome/puppeteer/issues/290#issuecomment-322921352
RUN apt-get update && \
apt-get install -yq gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 \
libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 \
libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 \
libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 \
ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget

Not working for me.

Using latest karma, latest karma chrome runner, latest jasmine and latest Chrome.

Headless mode does not work. Same ‘disconnected’ issues as everyone else is having.

For me it does work again after downgrading to karma 1.5.

I’m having to run it in this docker:

FROM markadams/chromium-xvfb

RUN echo 'deb http://deb.debian.org/debian jessie-backports main' > /etc/apt/sources.list.d/jessie-backports.list && \
    echp 'deb http://packages.linuxmint.com debian import' >> /etc/apt/sources.list

RUN apt-get update && apt-get install -y curl
RUN curl -sL https://deb.nodesource.com/setup_7.x | bash -
RUN apt-get install -t jessie-backports -y openjdk-8-jre-headless ca-certificates-java
RUN apt-get install -y nodejs git-core netcat libxi6 libgconf-2-4 firefox && \
    npm install -g npm@latest gulp protractor serve

RUN webdriver-manager update

Which times out karma 1.6 but works fine with karma 1.5. Fully reproducible.

Setting browserNoActivityTimeout: 60000 helped us solve a whole class of problems. We run our stuff in a Docker container (both Chrome headless and the tests themselves live in the same container).

Is virtualization or containerization contributing to this issue at all?

Installation of karma-cli resolved that issue for me

@daryl0313 It’s not necessary to define your own launcher. karma-firefox-launcher makes the FirefoxHeadless launcher available so you may just use it. If it doesn’t work for you then you may have an older version of the package; just update karma-firefox-launcher and it’ll be fine.

I’ve found also that when we upgraded build machine for tests to 4 cores, 16GB of RAM and SSD disk, problem disappeared. I assume that karma has some timeouts that breaks on slower machines. Maybe this is just a lucky, I don’t know. We just upgraded Azure machine…

I think, that is stopped being funny a while ago. Same issues on Ubuntu, trying now with changes both timeouts.

I’m using Karma version 1.4.1 and I also having this behavior on windows. First run works, but when I start it a second time it would fail. If I close the console and reopen it then it works (but again only for the first time, second call breaks again).

is there anything that change here that can break us?

You can look at CHANGELOG.

Small precision, it crash on Bitbucket Pipeline, which run some CoreOS Linux OS. We are not using macs, essentially Windows and Linux boxes for builds.

Did you try run sample project on Bitbucket? Is it reproduced locally? Does it work if downgrade on karma@1.5.x?