serverless: "EMFILE - too many open file" error

Previous issue: #3249, #10706

This is a meta-issue to gather all reports about the following error:

EMFILE - too many open files error

This is a very tough issue to solve because it happens randomly. It is caused by having too many files open on the whole machine (not just the serverless process).

A possible root cause could be synchronous usage of the fs API. A solution might be to use graceful-fs and/or the async API to access the filesystem.

We use graceful-fs and reduced the use of sync calls in Serverless Framework to a minimum, but some plugins might not. We’re trying to identify which plugins might cause the issue.

Workarounds

This isn’t ideal, but it is possible to increase the limit of files open on Linux and macOS with ulimit (https://stackoverflow.com/a/21070580/245552), but not on Windows.

If you have information about more workarounds, please post them here and we’ll update the issue.

How to help?

Add a 👍 on this issue (so that we can estimate the impact)
Investigate if some plugins might benefit from using graceful-fs and/or async filesystem calls
Fill the following form and post it as a comment:

OS:
Serverless Framework version ('sls --version'):
Node version ('node --version'):
Plugins used:

Let’s try and see if there’s a pattern (e.g. a plugin that comes up often).

About this issue

Original URL
State: open
Created 2 years ago
Reactions: 127
Comments: 50 (4 by maintainers)

Most upvoted comments

Hello, I got the EMFILE error when launching the serverless “deploy” script. I used graceful-fs like this to solve the issue :

install graceful-fs: npm install graceful-fs
go to file: node_modules/serverless/lib/plugins/package/lib/zip-service.js
replace this line : const fs = BbPromise.promisifyAll(require('fs')); by this:

var realFs = require('fs')
var gracefulFs = require('graceful-fs')
gracefulFs.gracefulify(realFs)
const fs = BbPromise.promisifyAll(realFs);

+41

tobelesa on May 30, 2022

Thank you for continuing to investigate this. After a bunch of digging, I was able to resolve my issue by using the serverless-plugin-common-excludes and serverless-plugin-include-dependencies plugins. This immediately resulted in a necessary not being included, but that issue appears to be specific to the knex ORM library and is probably unrelated to the issue within serverless. Ultimately, that was resolved by manually importing a sub-dependency in my own project.

+31

nunchukgun on Apr 7, 2022

Update: I got to the root of the issue. Here’s the full fix for NodeJS:

Run yarn add -D graceful-fs@4.2.6.
Open up node_modules/serverless/bin/serverless.js and add this right after use strict;:

require('../../graceful-fs/graceful-fs').gracefulify(require('fs'));

serg06 on Oct 30, 2022

Hello, I got the EMFILE error when launching the serverless “deploy” script. I used graceful-fs like this to solve the issue :

install graceful-fs: npm install graceful-fs

go to file: node_modules/serverless/lib/plugins/package/lib/zip-service.js

replace this line : const fs = BbPromise.promisifyAll(require('fs')); by this:
var realFs = require('fs')
var gracefulFs = require('graceful-fs')
gracefulFs.gracefulify(realFs)
const fs = BbPromise.promisifyAll(realFs);

this one worked on a windows 11 machine, thank you!

serverless version: 3.25.0 nodejs: 18.12.1

shierro on Jan 24, 2023

@shierro @tobelesa

I tried all of the solution But still I am sruggling with the issue, Anyone have update to solve this issue? On trying with graceful lib this error occured

indravijay-bot on Feb 27, 2023

I had the same issue. I was able to resolve it and significantly improve packaging speeds by moving plugin packages such as serverless-domain-manager to the devDependencies section of my package.json. Serverless excludes dev dependencies by default unless package.excludeDevDependencies is set to false.

benockert on Mar 9, 2024

Environment: win32, node 20.10.0, framework 3.38.0 (local), plugin 7.2.0, SDK 4.5.1 Credentials: Local, “default” profile Docs: docs.serverless.com Support: forum.serverless.com Bugs: github.com/serverless/serverless/issues

I’m using windows 11 I also encountered this issue will running the serverless deploy command on windows command prompt, but when I run the same command with Git Bash it worked for me.

rc2121 on Jan 10, 2024

Hello, I got the EMFILE error when launching the serverless “deploy” script. I used graceful-fs like this to solve the issue :

install graceful-fs: npm install graceful-fs

go to file: node_modules/serverless/lib/plugins/package/lib/zip-service.js

replace this line : const fs = BbPromise.promisifyAll(require('fs')); by this:
var realFs = require('fs')
var gracefulFs = require('graceful-fs')
gracefulFs.gracefulify(realFs)
const fs = BbPromise.promisifyAll(realFs);

FWIW, I had already done this in the past and the issue came back. I took a fresh look at my zip-service.js file and it was back to original version. Re-implemented this code change and now my deploys work properly again. So, if you have already implemented graceful-fs, double check this!

mojoalan on Dec 14, 2023

Thank you for continuing to investigate this. After a bunch of digging, I was able to resolve my issue by using the serverless-plugin-common-excludes and serverless-plugin-include-dependencies plugins. This immediately resulted in a necessary not being included, but that issue appears to be specific to the knex ORM library and is probably unrelated to the issue within serverless. Ultimately, that was resolved by manually importing a sub-dependency in my own project.

Thanks for sharing this. I was able to fix this issue by using only serverless-plugin-include-dependencies

Environment

Serverless 
Framework Core: 3.19.0 (local) 3.19.0 (global)
Plugin: 6.2.2
SDK: 4.3.2

Node
v16.15.1

erandakarachchi on Jun 11, 2022

@tobelesa this patch shouldn’t have any real effect, as we ensure fs is patched globally before any modules are loaded: https://github.com/serverless/serverless/blob/ed111e021b286eb4a68f9b2fce7fb641e03340bd/scripts/serverless.js#L7-L9

medikoo on May 31, 2022