ArchiveBox: Intermittent network response dropping when building and executing inside docker

So… even more… I changed puppeteer for puppeteer-core (a version of Puppeteer that doesn’t download Chromium by default) in the Dockerfile, because we’re installing chromium anyways separately. This at first failed as well:

Step 7/15 : RUN npm i puppeteer-core
 ---> Running in 4cfe4c562904
npm ERR! code EPROTO
npm ERR! errno EPROTO
npm ERR! request to https://registry.npmjs.org/rimraf failed, reason: write EPROTO 139977009982336:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1407:SSL alert number 40
npm ERR!

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2019-01-28T05_08_25_248Z-debug.log
ERROR: Service 'archivebox' failed to build: The command '/bin/sh -c npm i puppeteer-core' returned a non-zero code: 1

There seems to be something going on either with my network connection or the npm servers. I tried again:

Step 7/15 : RUN npm i puppeteer-core
 ---> Running in e1b3a79eaf9c
npm WARN tarball tarball data for es6-promise@^4.0.3 (sha512-n6wvpdE43VFtJq+lUDYDBFUwV8TZbuGXLV4D6wKafg13ldznKsyEvatubnmUe31zcvelSzOHF+XbaT+Bl9ObDg==) seems to be corrupted. Trying one more time.
npm WARN tarball tarball data for puppeteer-core@latest (sha512-JTsJKCQdrk1RqEGZN3l2TyW7Rhy7GWRRzd3PftYyA3B35l0t0lLU+gdF7czemnpSVVMiAgHpM1Uk/iO6jLreMA==) seems to be corrupted. Trying one more time.

> puppeteer-core@1.11.0 install /node_modules/puppeteer-core
> node install.js

npm WARN saveError ENOENT: no such file or directory, open '/package.json'
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN enoent ENOENT: no such file or directory, open '/package.json'
npm WARN !invalid#1 No description
npm WARN !invalid#1 No repository field.
npm WARN !invalid#1 No README data
npm WARN !invalid#1 No license field.

+ puppeteer-core@1.11.0
added 43 packages from 22 contributors and audited 50 packages in 14.773s
found 0 vulnerabilities

 ---> 7538b1c16fbc

This finally did work. Not sure about the tarball errors.

Back to the original purpose of the ticket, pocket feeds not being properly imported: I tried the same RSS feed and this time my two links were parsed / downloaded correctly; screenshot, html, pdf confirmed and working.

Thanks again for your support and this project. Love it and i think it’s very important. You might want to consider puppeteer-core .

OK, I did more digging. I edited the Dockerfile to include an RUN npm cache clean --force before the puppeteer (now step 8 instead of 7) installation, but no luck there as well:

 ---> 73357b1217dc
Removing intermediate container 84e268fc0b12
Step 6/16 : RUN chmod +x /usr/local/bin/dumb-init
 ---> Running in bd8430cbfbf9
 ---> dcaaf479c297
Removing intermediate container bd8430cbfbf9
Step 7/16 : RUN npm cache clean --force
 ---> Running in 7d6f8353ba49
npm WARN using --force I sure hope you know what you are doing.
 ---> 22fe375cd41a
Removing intermediate container 7d6f8353ba49
Step 8/16 : RUN npm i puppeteer
 ---> Running in 8a5d51af5bac
npm ERR! Unexpected end of JSON input while parsing near '...s/extract-zip":"^1.6.'

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2019-01-28T04_26_16_469Z-debug.log

I then reduced the Dockerfile to the bare minimum to see if that would give me any clue:

FROM node:11-slim
LABEL maintainer="Nick Sweeting <archivebox-git@sweeting.me>"

# RUN apt-get update \
#    && apt-get install -yq --no-install-recommends \
#        git wget curl youtube-dl gnupg2 libgconf-2-4 python3 python3-pip \
#    && rm -rf /var/lib/apt/lists/*

# Install latest chrome package and fonts to support major charsets (Chinese, Japanese, Arabic, Hebrew, Thai and a few others)
RUN apt-get update && apt-get install -y wget --no-install-recommends \
    && wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
    && sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
    && apt-get update \
    && apt-get install -y google-chrome-unstable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst ttf-freefont \
      --no-install-recommends \
    && rm -rf /var/lib/apt/lists/* \
    && rm -rf /src/*.deb

# It's a good idea to use dumb-init to help prevent zombie chrome processes.
#ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64 /usr/local/bin/dumb-init
#RUN chmod +x /usr/local/bin/dumb-init

# Do a npm clean
#RUN npm cache clean --force

# Install puppeteer so it's available in the container.
RUN npm i puppeteer

# Add user so we don't need --no-sandbox.
#RUN groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
#    && mkdir -p /home/pptruser/Downloads \
#    && chown -R pptruser:pptruser /home/pptruser \
#    && chown -R pptruser:pptruser /node_modules

# Install the ArchiveBox repository and pip requirements
#RUN git clone https://github.com/pirate/ArchiveBox /home/pptruser/app \
#    && mkdir -p /data \
#    && chown -R pptruser:pptruser /data \
#    && ln -s /data /home/pptruser/app/archivebox/output \
#    && ln -s /home/pptruser/app/bin/archivebox /bin/archive \
#    && chown -R pptruser:pptruser /home/pptruser/app/archivebox
#    # && pip3 install -r /home/pptruser/app/archivebox/requirements.txt

VOLUME /data

ENV LANG=C.UTF-8 \
    LANGUAGE=en_US:en \
    LC_ALL=C.UTF-8 \
    PYTHONIOENCODING=UTF-8 \
    CHROME_SANDBOX=False \

But still (this time errored out on the same spot):

Step 4/10 : RUN npm i puppeteer
 ---> Running in cdcae8339d94
npm ERR! Unexpected end of JSON input while parsing near '...s/extract-zip":"^1.6.'

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2019-01-28T04_35_30_363Z-debug.log
ERROR: Service 'archivebox' failed to build: The command '/bin/sh -c npm i puppeteer' returned a non-zero code: 1

So the error must be within the npm package of puppeteer?!