gatsby: [gatsby-remark-images-contentful] intermittent build errors with/to base64

@floriangaechter thanks for the repro i’ll try to look at this today or else tomorrow.

We’re running into this issue as well with the Figma marketing site when building on Ubuntu. We had a similar issue on some machines on macOS, but running yarn cache clean resolved the issue on those machines.

To resolve in production we’ve replaced all Gatsby Image gql queries w/ the _noBase64 variants.

Any chance someone on the @gatsbyjs/core team can take a look?

  System:
    OS: Linux 4.4 Ubuntu 16.04.5 LTS (Xenial Xerus)
    CPU: x64 Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
    Shell: 4.3.48 - /bin/bash
  Binaries:
    Node: 10.14.1 - /usr/local/bin/node
    Yarn: 1.15.2 - /usr/local/bin/yarn
    npm: 6.4.1 - /usr/local/bin/npm
  Languages:
    Python: 2.7.12 - /usr/bin/python
  npmPackages:
    gatsby: ^2.1 => 2.1.23 
    gatsby-image: ^2.0.19 => 2.0.19 
    gatsby-plugin-drift: ^1.0.0 => 1.0.0 
    gatsby-plugin-emotion: ^4.0.1 => 4.0.1 
    gatsby-plugin-google-tagmanager: ^2.0.0 => 2.0.5 
    gatsby-plugin-postcss: ^2.0.0 => 2.0.0 
    gatsby-plugin-react-css-modules: ^2.0.0 => 2.0.1 
    gatsby-plugin-react-helmet: ^3.0.0 => 3.0.0 
    gatsby-plugin-resolve-src: ^2.0.0-beta.1 => 2.0.0-beta.1 
    gatsby-plugin-segment-analytics: ^0.1.1 => 0.1.1 
    gatsby-plugin-sharp: ^2.0.20 => 2.0.20 
    gatsby-plugin-svgr: https://github.com/coreyward/gatsby-plugin-svgr#904c71904d10e1c9856cb6ee4346c1767563ccef => 1.0.0 
    gatsby-plugin-web-font-loader: ^1.0.4 => 1.0.4 
    gatsby-source-contentful: ^2.0.1 => 2.0.1 
    gatsby-source-filesystem: ^2.0.1 => 2.0.1 
    gatsby-source-lever: ^2.0.0 => 2.0.0 
    gatsby-transformer-json: ^2.1.1 => 2.1.2 
    gatsby-transformer-sharp: ^2.1.13 => 2.1.13 

@thetrevorharmon, I ended up having to remove the plugin for now. gatsby-remark-images-contentful doesn’t seem to ignore .gif files like gatsby-remark-images does which was causing a whole slew of errors for me. Once I removed the plugin, all of my build errors were resolved.

Hi all,

I’m also seeing this behaviour, but it’s originating from gatsby-remark-images-contentful.

The issue seems intermittent for me also, sometimes building fine, other times hanging on the queries stage.

I ran the build via Charles Proxy and can see the image requests going out. When the build hangs, I can see one or more images that have started transferring but don’t complete, axios then hangs indefinitely. I’ve only seen this effect the original images, not the w=40 variants.

I tried adding a timeout to the axios call, but this doesn’t have any effect. I believe this is due to the fact that the connection has started, rather than failed to connect at all.

With my very limited knowledge in this area, it does look like the code always expects the connection to be successful and timely. I wonder if we could bolster the connection handling to allow it to time out and then gracefully re-try the failed resources.

Interestingly when I requested the same resources via a bash script, concurrently requested, it succeeded every time. The transfers appeared to slow down towards the end, but no dropouts.

System:
    OS: macOS 10.14.5
    CPU: (8) x64 Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz
    Shell: 3.2.57 - /bin/bash
  Binaries:
    Node: 10.15.0 - /usr/local/bin/node
    Yarn: 1.13.0 - /usr/local/bin/yarn
    npm: 6.4.1 - /usr/local/bin/npm
  Languages:
    Python: 2.7.15 - /usr/local/bin/python
  Browsers:
    Chrome: 74.0.3729.169
    Firefox: 66.0.5
    Safari: 12.1.1
  npmPackages:
    gatsby: ^2.8.5 => 2.8.5
    gatsby-image: ^2.1.2 => 2.1.2
    gatsby-link: ^2.1.1 => 2.1.1
    gatsby-plugin-catch-links: ^2.0.15 => 2.0.15
    gatsby-plugin-facebook-pixel: ^1.0.3 => 1.0.3
    gatsby-plugin-feed: ^2.2.2 => 2.2.2
    gatsby-plugin-google-analytics: ^2.0.20 => 2.0.20
    gatsby-plugin-google-tagmanager: ^2.0.15 => 2.0.15
    gatsby-plugin-manifest: ^2.1.1 => 2.1.1
    gatsby-plugin-nprogress: ^2.0.10 => 2.0.10
    gatsby-plugin-offline: ^2.1.1 => 2.1.1
    gatsby-plugin-preact: ^3.0.0 => 3.0.0
    gatsby-plugin-react-helmet: ^3.0.12 => 3.0.12
    gatsby-plugin-s3: ^0.2.5 => 0.2.5
    gatsby-plugin-sass: ^2.0.11 => 2.0.11
    gatsby-plugin-sharp: ^2.1.3 => 2.1.3
    gatsby-plugin-sitemap: ^2.1.0 => 2.1.0
    gatsby-plugin-twitter: ^2.0.13 => 2.0.13
    gatsby-plugin-web-font-loader: ^1.0.4 => 1.0.4
    gatsby-remark-component: ^1.1.3 => 1.1.3
    gatsby-remark-copy-linked-files: 2.0.12 => 2.0.12
    gatsby-remark-external-links: ^0.0.4 => 0.0.4
    gatsby-remark-images: ^2.0.6 => 2.0.6
    gatsby-remark-images-contentful: ^2.0.12 => 2.0.13
    gatsby-remark-responsive-iframe: ^2.1.1 => 2.1.1
    gatsby-remark-smartypants: ^2.0.9 => 2.0.9
    gatsby-source-contentful: ^2.0.67 => 2.0.67
    gatsby-source-filesystem: ^2.0.38 => 2.0.38
    gatsby-source-wordpress: ^3.0.64 => 3.0.64
    gatsby-transformer-json: ^2.1.11 => 2.1.11
    gatsby-transformer-remark: ^2.3.12 => 2.3.12
    gatsby-transformer-sharp: ^2.1.21 => 2.1.21

@ddsr17 yes, as soon as we pull in data from Contentful the build task crashes, stalls, or, if we’re very lucky, succeeds 😄 but to be serious, yes, we still have the issue.

Currently we’re doing without any image preprocessing, just serving the images without traced SVGs or blurred versions of the image.

Some new information – sometimes I’m getting the following error:

error UNHANDLED REJECTION


  Error: read ETIMEDOUT

  - stream_base_commons.js:162 TLSWrap.onStreamRead
    internal/stream_base_commons.js:162:27

Here’s the stack trace:

{ Error: read ETIMEDOUT
      at TLSWrap.onStreamRead (internal/stream_base_commons.js:162:27)
    errno: 'ETIMEDOUT',
    code: 'ETIMEDOUT',
    syscall: 'read',
    config:
     { adapter: [Function: httpAdapter],
       transformRequest: [Object],
       transformResponse: [Object],
       timeout: 0,
       xsrfCookieName: 'XSRF-TOKEN',
       xsrfHeaderName: 'X-XSRF-TOKEN',
       maxContentLength: -1,
       validateStatus: [Function: validateStatus],
       headers: [Object],
       method: 'get',
       responseType: 'arraybuffer',
       url:
        'https://images.ctfassets.net/i8q4h01orwwf/3LL7xtOMlcXjYKi2S5Xpto/573dc94143c45c10636562f753e76df0/beloved_features_fonts.gif?w=40',
       data: undefined },
    request:
     Writable {
       _writableState: [WritableState],
       writable: true,
       _events: [Object],
       _eventsCount: 2,
       _maxListeners: undefined,
       _options: [Object],
       _redirectCount: 0,
       _redirects: [],
       _requestBodyLength: 0,
       _requestBodyBuffers: [],
       _onNativeResponse: [Function],
       _currentRequest: [ClientRequest],
       _currentUrl:
        'https://images.ctfassets.net/i8q4h01orwwf/3LL7xtOMlcXjYKi2S5Xpto/573dc94143c45c10636562f753e76df0/beloved_features_fonts.gif?w=40' },
    response: undefined } }

and:

{ Error: read ETIMEDOUT
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:162:27)
  errno: 'ETIMEDOUT',
  code: 'ETIMEDOUT',
  syscall: 'read',
  config:
   { adapter: [Function: httpAdapter],
     transformRequest: { '0': [Function: transformRequest] },
     transformResponse: { '0': [Function: transformResponse] },
     timeout: 0,
     xsrfCookieName: 'XSRF-TOKEN',
     xsrfHeaderName: 'X-XSRF-TOKEN',
     maxContentLength: -1,
     validateStatus: [Function: validateStatus],
     headers:
      { Accept: 'application/json, text/plain, */*',
        'User-Agent': 'axios/0.18.0' },
     method: 'get',
     responseType: 'arraybuffer',
     url:
      'https://images.ctfassets.net/i8q4h01orwwf/3LL7xtOMlcXjYKi2S5Xpto/573dc94143c45c10636562f753e76df0/beloved_features_fonts.gif?w=40',
     data: undefined },
  request:
   Writable {
     _writableState:
      WritableState {
        objectMode: false,
        highWaterMark: 16384,
        finalCalled: false,
        needDrain: false,
        ending: false,
        ended: false,
        finished: false,
        destroyed: false,
        decodeStrings: true,
        defaultEncoding: 'utf8',
        length: 0,
        writing: false,
        corked: 0,
        sync: true,
        bufferProcessing: false,
        onwrite: [Function: bound onwrite],
        writecb: null,
        writelen: 0,
        bufferedRequest: null,
        lastBufferedRequest: null,
        pendingcb: 0,
        prefinished: false,
        errorEmitted: false,
        emitClose: true,
        autoDestroy: false,
        bufferedRequestCount: 0,
        corkedRequestsFree: [Object] },
     writable: true,
     _events:
      [Object: null prototype] {
        response: [Function: handleResponse],
        error: [Function: handleRequestError] },
     _eventsCount: 2,
     _maxListeners: undefined,
     _options:
      { protocol: 'https:',
        maxRedirects: 21,
        maxBodyLength: 10485760,
        path:
         '/i8q4h01orwwf/3LL7xtOMlcXjYKi2S5Xpto/573dc94143c45c10636562f753e76df0/beloved_features_fonts.gif?w=40',
        method: 'get',
        headers: [Object],
        agent: undefined,
        auth: undefined,
        hostname: 'images.ctfassets.net',
        port: null,
        nativeProtocols: [Object],
        pathname:
         '/i8q4h01orwwf/3LL7xtOMlcXjYKi2S5Xpto/573dc94143c45c10636562f753e76df0/beloved_features_fonts.gif',
        search: '?w=40' },
     _redirectCount: 0,
     _redirects: [],
     _requestBodyLength: 0,
     _requestBodyBuffers: [],
     _onNativeResponse: [Function],
     _currentRequest:
      ClientRequest {
        _events: [Object],
        _eventsCount: 6,
        _maxListeners: undefined,
        outputData: [],
        outputSize: 0,
        writable: true,
        _last: true,
        chunkedEncoding: false,
        shouldKeepAlive: false,
        useChunkedEncodingByDefault: false,
        sendDate: false,
        _removedConnection: false,
        _removedContLen: false,
        _removedTE: false,
        _contentLength: 0,
        _hasBody: true,
        _trailer: '',
        finished: true,
        _headerSent: true,
        socket: [TLSSocket],
        connection: [TLSSocket],
        _header:
         'GET /i8q4h01orwwf/3LL7xtOMlcXjYKi2S5Xpto/573dc94143c45c10636562f753e76df0/beloved_features_fonts.gif?w=40 HTTP/1.1\r\nAccept: application/json, text/plain, */*\r\nUser-Agent: axios/0.18.0\r\nHost: images.ctfassets.net\r\nConnection: close\r\n\r\n',
        _onPendingData: [Function: noopPendingOutput],
        agent: [Agent],
        socketPath: undefined,
        timeout: undefined,
        method: 'GET',
        path:
         '/i8q4h01orwwf/3LL7xtOMlcXjYKi2S5Xpto/573dc94143c45c10636562f753e76df0/beloved_features_fonts.gif?w=40',
        _ended: false,
        res: null,
        aborted: false,
        timeoutCb: null,
        upgradeOrConnect: false,
        parser: null,
        maxHeadersCount: null,
        _redirectable: [Circular],
        [Symbol(isCorked)]: false,
        [Symbol(outHeadersKey)]: [Object] },
     _currentUrl:
      'https://images.ctfassets.net/i8q4h01orwwf/3LL7xtOMlcXjYKi2S5Xpto/573dc94143c45c10636562f753e76df0/beloved_features_fonts.gif?w=40' },
  response: undefined }

I think it has to do with the way the images are pulled from Contentful…