puppeteer: Puppeteer on docker crashes when loading pages in parallel

Here we go: https://github.com/joelgriffith/browserless Docker: https://hub.docker.com/r/browserless/chrome/

@ebidel That’s a bad idea.

I’ve been using chrome-remote-interface to build something very similar to Puppeteer (a public talk about this is here - https://paambaati.github.io/rendering-at-scale/), and use it at scale. I ran into each of these problems (memory leaks, segfaults, runaway Chrome instances, slow downs, etc.) and in the end, decided to run them without containers but in their own dedicated AWS ASG, with Mesos/Marathon running them. Here’s what I’ve learnt —

1. Set /dev/shm to 1 GB or more.
2. Never use long-running Chrome - it slowly eats up memory; always launch Chrome per-request. SIGKILL Chrome at the end, and cleanup the data directory later, outside of your Puppeteer or chrome-remote-interface based app.
3. Use a zombie-reaper like dumb-init to start your main CMD or ENTRYPOINT process.
4. Increase --max-old-space-size.

While these things helped a lot with the stability issues, ultimately, Chrome inside Docker was still unstable and would either slow down (accept socket connections over the remote debugging port but then not respond to a few commands) or segfault after a few hours.

I’ve been debating whether or not to open-source the images I run at browserless, as it does take days/weeks to get right. Browserless goes a step further and treats the browser more like an appliance (think database) then just a binary to execute arbitrarily. There’s also other fixes included like emoji’s and other fonts. Finally, I maintain builds for specific puppeteer versions + a little live debugger that makes it really painless to test scripts against your headless farm. Would that be of interest?

If you want to use the provided version bundled with puppeteer (which I recommend after trying to download it via various package managers), then you’ll need a list like so:

# Dependencies needed for packages downstream
RUN apt-get update && apt-get install -y \
  wget \
  unzip \
  fontconfig \
  locales \
  gconf-service \
  libasound2 \
  libatk1.0-0 \
  libc6 \
  libcairo2 \
  libcups2 \
  libdbus-1-3 \
  libexpat1 \
  libfontconfig1 \
  libgcc1 \
  libgconf-2-4 \
  libgdk-pixbuf2.0-0 \
  libglib2.0-0 \
  libgtk-3-0 \
  libnspr4 \
  libpango-1.0-0 \
  libpangocairo-1.0-0 \
  libstdc++6 \
  libx11-6 \
  libx11-xcb1 \
  libxcb1 \
  libxcomposite1 \
  libxcursor1 \
  libxdamage1 \
  libxext6 \
  libxfixes3 \
  libxi6 \
  libxrandr2 \
  libxrender1 \
  libxss1 \
  libxtst6 \
  ca-certificates \
  fonts-liberation \
  libappindicator1 \
  libnss3 \
  lsb-release \
  xdg-utils \
  wget

Closing this since @joelgriffith provided a docker image that everyone likes.

Love it.

/dev/shm was key.

DockerFile

# ubuntu:xenial is the 16.04 LTS release
FROM ubuntu:xenial

# From: https://github.com/GoogleChrome/puppeteer/issues/1345#issuecomment-343554457
# Install necessary apt packages for puppeteer bundled chromium work
RUN apt-get update && apt-get install --no-install-recommends -y \
  ca-certificates \
  curl \
  fontconfig \
  fonts-liberation \
  gconf-service \
  git \
  libappindicator1 \
  libasound2 \
  libatk1.0-0 \
  libc6 \
  libcairo2 \
  libcups2 \
  libdbus-1-3 \
  libexpat1 \
  libfontconfig1 \
  libgcc1 \
  libgconf-2-4 \
  libgdk-pixbuf2.0-0 \
  libglib2.0-0 \
  libgtk-3-0 \
  libnspr4 \
  libnss3 \
  libpango-1.0-0 \
  libpangocairo-1.0-0 \
  libstdc++6 \
  lib\x11-6 \
  libx11-xcb1 \
  libxcb1 \
  libxcomposite1 \
  libxcursor1 \
  libxdamage1 \
  libxext6 \
  libxfixes3 \
  libxi6 \
  libxrandr2 \
  libxrender1 \
  libxss1 \
  libxtst6 \
  locales \
  lsb-release \
  unzip \
  wget \
  xdg-utils

# Install dumb-init. Node has issues being pid 1
RUN wget https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64.deb\
  && dpkg -i dumb-init_*.deb \
  && rm dumb-init_*.deb

# Install nodejs 8
RUN curl -sL https://deb.nodesource.com/setup_8.x | bash - \
  && apt-get install -y nodejs

# Add pptr user
RUN useradd --create-home --user-group --shell /bin/bash pptr
USER pptr
WORKDIR /home/pptr

# Install npm deps first since we want to keep modules cached for super fast builds
COPY package.json .
RUN npm install

COPY tests.js .

# run the tests by default
CMD [ "dumb-init", "node", "tests.js"]

Running as

docker run -it --name tests --shm-size 1gb tests Works wonders.

I wonder if /dev/shm = 1gb can be baked into the dockerfile rather than being passed as param. Would look cleaner.

Also @joelgriffith / Chrome puppeteer core contributors. Would you mind if I send a Dockerfile as a PR? And would you consider it publishing it to official docker registry as GoogleChrome/puppeteer base image ?

Having a base image that always works without crazy setup would be super helpful to rest of the community.