gatsby: gatsby-source-filesystem fails with "EMFILE: too many open files" on Windows with lots of files

Description

When I try to gatsby build a site with around 9,500 files in the src folder on Windows 10 I get this error and the build fails:

Error: EMFILE: too many open files

It specifies the error occurred on an image file (usually a different one each time).

Steps to reproduce

My project is an incredibly image heavy website, and it uses gatsby-source-filesystem, gatsby-image and gatsby-transformer-sharp to process the images.

I have lots of folders containing lots of images like this:

content/gigs/[gig name]/[artist_name]/[image1.jpg,image2.jpg,etc]

In each gig folder there’s an index.md which contains metadata for the page. I use a graphql query in gatsby-node.js to turn these markdown files into pages via a template. The template uses a graphql querie and a regex injected via gatsby-node.js to get the images in the folder for that page.

Quite impressively it succeeds with 8,503 files but starts failing if I add in more.

All the code is here: https://github.com/FraserThompson/dunedinsound.com/tree/gatsby

Expected result

I expected it to be able to build my site because the documentation doesn’t mention file limits.

Actual result

The build doesn’t succeed.

Environment

gatsby info --clipboard fails for me, probably because of https://github.com/gatsbyjs/gatsby/issues/11496, but this is the output of gatsby info:

System: OS: Windows 10 CPU: (16) x64 AMD Ryzen 7 2700X Eight-Core Processor Binaries: Yarn: 1.13.0 - C:\Program Files (x86)\Yarn\bin\yarn.CMD Browsers: Edge: 44.17763.1.0 npmPackages: gatsby: ^2.1.4 => 2.1.4 gatsby-image: ^2.0.22 => 2.0.29 gatsby-plugin-feed: ^2.0.8 => 2.0.13 gatsby-plugin-google-analytics: ^2.0.5 => 2.0.14 gatsby-plugin-manifest: ^2.0.5 => 2.0.17 gatsby-plugin-offline: ^2.0.5 => 2.0.23 gatsby-plugin-react-helmet: ^3.0.0 => 3.0.6 gatsby-plugin-sharp: ^2.0.20 => 2.0.20 gatsby-plugin-styled-components: ^3.0.4 => 3.0.5 gatsby-plugin-typography: ^2.2.0 => 2.2.7 gatsby-remark-copy-linked-files: ^2.0.5 => 2.0.9 gatsby-remark-images: ^2.0.4 => 2.0.6 gatsby-remark-prismjs: ^3.0.0 => 3.2.4 gatsby-remark-smartypants: ^2.0.5 => 2.0.8 gatsby-source-filesystem: ^2.0.2 => 2.0.20 gatsby-transformer-remark: ^2.1.15 => 2.2.5 gatsby-transformer-sharp: ^2.1.13 => 2.1.13

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 1
Comments: 19 (9 by maintainers)

Commits related to this issue

added graceful-fs to patch fs large vocabs couldn't be build due to the following error: ``` Error: EMFILE: too many open files, scandir '....' ``` found the solution here: <https://github.com/gats... — committed to openeduhub/skohub-vocabs by sroertgen 4 years ago

Most upvoted comments

I should also add that this issue never went away for me. Every time I change Gatsby versions it seems to come back again.

This should be re-opened imo since it seems a bunch of people with large sites are experiencing this issue.

FraserThompson on Jul 8, 2020

Okay, for anyone else who is having this issue consistently and looking for a permanent fix, I’ve made a version of gatsby-source-filesystem which uses a queue to limit the number of files accessed simultaneously: gatsby-source-filesystem-with-queue

To use it you can just replace gatsby-source-filesystem with gatsby-source-filesystem-with-queue in your package.json and your gatsby-config.js. Run npm install and your builds should start succeeding.

Since it limits the number of files being processed, it might slow things down. You can tweak this value (by default 800) with the GATSBY_CONCURRENT_FILES environment variable to strike a balance between build speed and not producing EMFILE errors.

Another hint for large sites or sites with lots of large files: You’ll probably need to raise the Node heap limit, for example to 8gb with NODE_OPTIONS=--max-old-space-size=8192.

FraserThompson on Nov 29, 2022

Thanks for the response. For me CHOKIDAR_USEPOLLING=1 doesn’t seem to help, and as far as I know there’s no soft file limit on Windows. Maybe I’ll look into building it inside a Linux Docker container or something if this is a Windows problem.

Google seems to suggest https://github.com/isaacs/node-graceful-fs as a drop in replacement for fs, I might also experiment with that to see if it makes a difference.

EDIT: I can confirm that monkeypatching fs with graceful-fs at the top of gatsby-node.js as in the snippet below fixes the issue for me.

const realFs = require('fs')
const gracefulFs = require('graceful-fs')
gracefulFs.gracefulify(realFs)

EDIT2: Actually after upgrading from Node 10 to Node 11 everything seems to be fine without having to patch fs… So all is well!

FraserThompson on Mar 22, 2019

@volkandkaya you may have already somehow solved your issue. For me, using the library fs-extra in version 9.0.0 fixed the issue on ubuntu 18.

dr0i on May 25, 2020

@dr0i THANK YOU! I tried everything mentioned here, and using fs-extra was the only thing that worked! Using graceful-fs did nothing for me.

For those that need it, here’s the documentation: https://github.com/jprichardson/node-fs-extra

note: I’m windows user and this worked. I also had to refactor my code a bit to work with the new api of fs-extra.

philr35 on Mar 1, 2021

Hi all,

the best way i found to solve this issue is using graceful-fs. add in the beginning of gatsby-config.js:

const fs = require('fs');
const gracefulFs = require('graceful-fs');
gracefulFs.gracefulify(fs);

wamir79 on Aug 30, 2020

I am getting this issue with my gatsby-trasformer-asciidoctor project (over 10k asciidoc files, source is private) running on WSL2 ubuntu 20. @volkandkaya for me the problem is happening in the gatsby-source-filesystem plugin so how do I make it use fs-extra? Interestingly this only happened when I upgraded my environment from ubuntu 18 on WSL v1 to ubuntu 20 on WSL 2

adamthrottleup on Jul 6, 2020

Currently we don’t have any solutions to this except increasing open file limit (not sure if it’s possible on Windows). One workaround could be using CHOKIDAR_USEPOLLING=1 env var to tell chokidar (our fs watcher package) to poll instead of using file watchers. This has performance implications, but could help with this problem.

freiksenet on Feb 25, 2019