gatsby: Gatsby 4 - EPIPE error on Netlify

Preliminary Checks

Description

Just upgraded to Gatsby 4 and everything is working fine locally. I can npm develop and build without issues. However, as soon as I push to Netlify the below error is returned near the end of building.

    4:16:57 PM: error UNHANDLED EXCEPTION write EPIPE
    4:16:57 PM: 
    4:16:57 PM: 
    4:16:57 PM:   Error: write EPIPE
    4:16:57 PM:   
    4:16:57 PM:   - child_process:846 ChildProcess.target._send
    4:16:57 PM:     node:internal/child_process:846:20
    4:16:57 PM:   
    4:16:57 PM:   - child_process:719 ChildProcess.target.send
    4:16:57 PM:     node:internal/child_process:719:19
    4:16:57 PM:   
    4:16:57 PM:   - index.js:290 WorkerPool.sendMessage
    4:16:57 PM:     [repo]/[gatsby-worker]/dist/index.js:290:19
    4:16:57 PM:   
    4:16:57 PM:   - worker-messaging.ts:22 
    4:16:57 PM:     [repo]/[gatsby]/src/utils/jobs/worker-messaging.ts:22:22
    4:16:57 PM:   
    4:16:57 PM: 
    4:16:57 PM: not finished Merge worker state - 0.168s

The Netlify site is deploying on node v16.13.0 (npm v8.1.0) - which is the same as my local environment(Windows 10).

Netlify support believe it’s an issue with Gatsby, which surely can’t be the case if it’s working locally?

Reproduction Link

Unable to provide

Steps to Reproduce

  1. Install the following packages after cloning https://github.com/gatsbyjs/gatsby-starter-wordpress-blog : gatsby: ^4.0.2 => 4.0.2 gatsby-background-image: ^1.5.3 => 1.5.3 gatsby-plugin-feed: ^4.0.0 => 4.0.0 gatsby-plugin-google-tagmanager: ^4.0.0 => 4.0.0 gatsby-plugin-image: ^2.0.0 => 2.0.0 gatsby-plugin-manifest: ^4.0.0 => 4.0.0 gatsby-plugin-nprogress: ^4.0.0 => 4.0.0 gatsby-plugin-offline: ^5.0.0 => 5.0.0 gatsby-plugin-react-helmet: ^5.0.0 => 5.0.0 gatsby-plugin-react-svg: ^3.0.1 => 3.1.0 gatsby-plugin-remove-fingerprints: ^0.0.2 => 0.0.2 gatsby-plugin-sass: ^5.0.0 => 5.0.0 gatsby-plugin-sharp: ^4.0.1 => 4.0.1 gatsby-plugin-sitemap: ^5.0.0 => 5.0.0 gatsby-plugin-styled-components: ^5.0.0 => 5.0.0 gatsby-plugin-svgr-loader: ^0.1.0 => 0.1.0 gatsby-plugin-web-font-loader: ^1.0.4 => 1.0.4 gatsby-source-filesystem: ^4.0.0 => 4.0.0 gatsby-source-wordpress: ^6.0.0 => 6.0.0 gatsby-transformer-sharp: ^4.0.0 => 4.0.0

  2. Setup WordPress installation and configure with the following gatsby-config:

 {
    resolve: `gatsby-source-wordpress`,
    options: {
      url: process.env.GRAPHQL_API,
      debug: {
        graphql: {
          writeQueriesToDisk: true,
        },
      },
      production: {
        allow404Images: true,
      },
      auth:
        process.env.NODE_ENV === "production"
          ? {
              htaccess: {
                username: process.env.HTACCESS_USERNAME,
                password: process.env.HTACCESS_PASSWORD,
              },
            }
          : {},
      type: {
        Post: {
          limit:
            process.env.NODE_ENV === `development`
              ? 30
              : process.env.POST_LIMIT || 999999,
        },
        Comment: {
          exclude: true,
        },
        PostFormat: {
          exclude: true,
        },
      },
    },
  },
  1. Deploy to Netlify

Expected Result

Site builds successfully

Actual Result

Site build fails each time with UNHANDLED EXCEPTION write EPIPE error.

Environment

My local environment: 

 System:
    OS: Windows 10 10.0.19042
    CPU: (16) x64 Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz
  Binaries:
    Node: 16.13.0 - C:\Program Files\nodejs\node.EXE
    npm: 8.1.0 - C:\Program Files\nodejs\npm.CMD
  Languages:
    Python: 3.10.0
  Browsers:
    Edge: Spartan (44.19041.1266.0), Chromium (95.0.1020.30)
  npmPackages:
    gatsby: ^4.0.2 => 4.0.2
    gatsby-background-image: ^1.5.3 => 1.5.3
    gatsby-plugin-feed: ^4.0.0 => 4.0.0
    gatsby-plugin-google-tagmanager: ^4.0.0 => 4.0.0
    gatsby-plugin-image: ^2.0.0 => 2.0.0
    gatsby-plugin-manifest: ^4.0.0 => 4.0.0
    gatsby-plugin-nprogress: ^4.0.0 => 4.0.0
    gatsby-plugin-offline: ^5.0.0 => 5.0.0
    gatsby-plugin-react-helmet: ^5.0.0 => 5.0.0
    gatsby-plugin-react-svg: ^3.0.1 => 3.1.0
    gatsby-plugin-remove-fingerprints: ^0.0.2 => 0.0.2
    gatsby-plugin-sass: ^5.0.0 => 5.0.0
    gatsby-plugin-sharp: ^4.0.1 => 4.0.1
    gatsby-plugin-sitemap: ^5.0.0 => 5.0.0
    gatsby-plugin-styled-components: ^5.0.0 => 5.0.0
    gatsby-plugin-svgr-loader: ^0.1.0 => 0.1.0
    gatsby-plugin-web-font-loader: ^1.0.4 => 1.0.4
    gatsby-source-filesystem: ^4.0.0 => 4.0.0
    gatsby-source-wordpress: ^6.0.0 => 6.0.0
    gatsby-transformer-sharp: ^4.0.0 => 4.0.0
  npmGlobalPackages:
    gatsby-cli: 4.0.0

Netlify site:
    node v16.13.0 (npm v8.1.0) 
    Using ruby version 2.7.1
    Using PHP version 8.0

Config Flags

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 8
  • Comments: 35 (5 by maintainers)

Commits related to this issue

Most upvoted comments

I have the same problem on Gatsby Version 4.11. Why is this issue closed?

+17
danielmoessner on Apr 7, 2022

@benlavalley sorry for the radio silence. I ended up moving to Gatsby Cloud for the builds and Netlify for the hosting. Didn’t need to change anything, plus Gatsby builds 70% faster.

+15
SamHoltby on Nov 17, 2021

Hello, I also started having the same problem recently, is there any proper solution?

+7
TiagoJeronimo on Mar 1, 2022

@benlavalley sorry for the radio silence. I ended up moving to Gatsby Cloud for the builds and Netlify for the hosting. Didn’t need to change anything, plus Gatsby builds 70% faster.

Whats the benefit of hosting on Netlify instead of just hosting on Gatsby Cloud?

Also this issue shouldn’t be closed, this solution is more of a hack than a solution

+4
samgermain on Apr 16, 2022

I made some changes to my app, removed content and slowly re-added it, and I seem to be able to deploy now, though I don’t think it will work consistently due to the variable nature of memory allocation in Netlify.

For anyone coming across this, changes I made to my app to try and improve efficiency/lower memory usage:

Gatsby-specific:

  • Migrate from gatsby-image to gatsby-plugin-image
  • Removed a chunk of unecessary images from processing
  • Tried adding GATSBY_CONCURRENT_DOWNLOAD environment variable and set to 15 (default 200)
  • Set GATSBY_CPU_COUNT to 1 for production deploys (attempts to reduce parallelization/peak memory usage)

Netlify-specific:

  • Added gatsby-plugin-remove-fingerprints package in my netlify config
  • Migrated from Xenial to Focal build image
  • Added @netlify/plugin-gatsby

I have a feeling that if my site’s cache is cleared I might see deployment failures again. Crossing fingers, and I hope some of this might help Sam and any others that run into this problem.

+4
benlavalley on Nov 12, 2021

I locked my Gatsby version to =v4.1.3 in my packag.json and I can actually deploy my site now, though I still have to reduce the overall amount of static content my site includes.

Just wanted to comment that I was also running into this issue, and @benlavalley’s above advice fixed it for me. 👏👏👏

+3
jessmartin on Mar 18, 2022

@benlavalley Yes I got the same issue after upgrading to the latest 4.9

+3
wahidshafique on Mar 4, 2022

Hi guys, do we have proper solution for this?

+3
MJDCKP on Feb 28, 2022

Upon further testing and following some suggestions from the Netlify support team, I was able to run my own Netlify local build processes and attempt to reproduce the Netlify production build error I was seeing.

Mission success – the issue was reproduced and I seemed to get the same write EPIPE error.

Upon bumping my Docker memory limit up from 3GB to 4GB, my build succeeds.

I’d guess it is only natural that Gatsby v4 uses more memory with it’s SSR enhancements over Gatsby v3.

I put together some really clever routines to parse file-based content with Gatsby and offer up what would otherwise seem to be server-side generated content (I have drop-down filters, searching, and paging for online shop and sports team roster content – all statically generated) but it seems like this creating generating too much data or too many pages for Netlify to support with the build process 3gb memory limit 😦

More detail in the Netlify support forum here: https://answers.netlify.com/t/gatsby-v4-works-locally-but-timed-out-on-netlify/46339/9

Netlify states they can’t budge on their 3GB memory limit, so I have to figure out how I’m going to move forward now – do I rearchitect my site to get rid of a ton of clever work that went into building an interactive site that was backed by the performance of statically generated content, or do I get rid of it all and dive into Gatsby’s SSR support and then hope that my builds stay under Netlify’s 3GB memory limit…

+3
benlavalley on Nov 11, 2021

Hi everyone. There is now fix out for this, so if you upgrade to gatsby@4.13.1 it should solve the problem. This is being tracked at #35055, so please post there about whether or not this fixes it for you.

+2
ascorbic on Apr 29, 2022

in my case, version 4.7.0 helped solve the problem, on version 4.7.2 the error persisted

+2
andrewgolovanov on Apr 1, 2022

This worked for me too ! (from v4.10.2 to v4.7.2) And a small tips: v4.7.2 not support built in gatsby-config.ts, use gatsby-config.js instead.

+2
ChangerHe on Mar 26, 2022

Could you check peak RAM usage? My guess is that the worker crashed due to OOM. EPIPE usually means that the communication channel got destroyed for any reason.

+2
wardpeet on Nov 8, 2021

+1 I’m also experiencing this error and fixed it by rolling back to Gatsby v4.7.0

Looking forward to what others suggest to move forward!

+1
michaellunzer on Apr 21, 2022

Same problem with: "gatsby": "^4.12.1",

Worked only after downgrading to 4.7.0

image

+1
RazvanRauta on Apr 16, 2022

I was seeing this same build error on my Netlify hosted site when I upgraded to Gatsby 4.9. I saw a couple people mention lowering GATSBY_CONCURRENT_DOWNLOAD. I added this to my Netlify environment variables with a value of 16 and my builds complete without errors and with no other changes.

Update: A few days after I posted this above message, my Netlify builds started failing with the same EPIPE error. I guess one build randomly worked. Now Im consistently getting this error. Its very frustrating because builds always work fine locally.

+1
davidbbe on Mar 16, 2022

@benlavalley Thanks for doing all the legwork!

It’s interesting that Netlify won’t budge on the 3GB memory limit. On their pricing page, via the High-Performance Build section, they state Up to 6GB of memory…

I’ve asked Netlify to increase my memory. If it’s possible I’ll update this issue with the results.

+1
SamHoltby on Nov 11, 2021

Could you check peak RAM usage? My guess is that the worker crashed due to OOM. EPIPE usually means that the communication channel got destroyed for any reason.

This seems like a culprit.

I just drastically reduced the static file content footprint of my site to test, and it deployed successfully.

When I added my content back, the deploy fails again.

@fool Can you bump my memory limit up? 😃

+1
benlavalley on Nov 8, 2021

@fool Yes it does happen after a period of processing time.

1:43:47 PM: success run queries in workers - 139.530s - 340/340 2.44/s 1:47:45 PM: error UNHANDLED EXCEPTION write EPIPE 1:47:45 PM: 1:47:45 PM: 1:47:45 PM: Error: write EPIPE 1:47:45 PM: 1:47:45 PM: - child_process.js:841 ChildProcess.target._send 1:47:45 PM: internal/child_process.js:841:20 1:47:45 PM: 1:47:45 PM: - child_process.js:712 ChildProcess.target.send 1:47:45 PM: internal/child_process.js:712:19 1:47:45 PM: 1:47:45 PM: - index.js:290 WorkerPool.sendMessage 1:47:45 PM: [repo]/[gatsby-worker]/dist/index.js:290:19 1:47:45 PM: 1:47:45 PM: - worker-messaging.ts:22 1:47:45 PM: [repo]/[gatsby]/src/utils/jobs/worker-messaging.ts:22:22 1:47:45 PM: 1:47:45 PM: 1:47:45 PM: not finished Merge worker state - 0.152s 1:47:45 PM: ERROR: "build:app" exited with 1.

For anyone from Netlify investigating, feel free to dig into anything with my account – my netlify site name is amazing-wiles-1918a6 and the published URL is https://floridabulldogshockey.com. My first React / Gatsby site.

As I have time I’ll work on trying to get a more simple reproduction put together.

+1
benlavalley on Nov 3, 2021
Read more comments on GitHub
← gatsby: Gatsby 2.27.1 breaks Storybook
gatsby: "gatsby build" breaks when dependencies use es6 code (which Uglify doesn't support) →