serverless-python-requirements: Cannot Deploy; EMFILE: too many open files

@medikoo … have been able to work around it with either one or both of the following plugins: serverless-plugin-common-excludes and serverless-plugin-include-dependencies.

Just installing serverless-plugin-common-exludes worked for me. Thanks! 🙇

Hello, I got the EMFILE error when launching the serverless “deploy” script. I used graceful-fs like this to solve the issue :

1. install graceful-fs: npm install graceful-fs
2. go to file: node_modules/serverless/lib/plugins/package/lib/zip-service.js
3. replace this line : const fs = BbPromise.promisifyAll(require('fs')); by this:

var realFs = require('fs')
var gracefulFs = require('graceful-fs')
gracefulFs.gracefulify(realFs)
const fs = BbPromise.promisifyAll(realFs);

@medikoo Looks like this is still an issue – I’ve run into a couple of times before and have been able to work around it with either one or both of the following plugins: serverless-plugin-common-excludes and serverless-plugin-include-dependencies.

However, this issue seems to just randomly come up when trying to run sls deploy. Have you been able to look into why this issue still persists despite saying multiple times that it should be fixed next release or that it’s currently fixed? Because, truth be told it is not fixed yet.

As I was writing this comment, I was JUST able to get past the problem by running npm install serverless-plugin-include-dependencies@4.1.0 specifically. I was using this package at version 5.0.0 earlier with no luck. However, my project is now running properly using version 4.1.0.

I hope this helps someone else who is experiencing this very frustrating issue with serverless framework!

Delete _node_modules _, package.lock.json file and .serverless . after that reinstall all dependencies by npm install. and problem fixed.

A colleague saw the EMFILE: too many files open error recently, and the error was only showing up on their computer. It took a while for us to figure it out what the issue was, but it ended up being a package glob issue.

We had manually listed exclude glob patterns in the serverless.yml like this:

package:
  individually: true
  exclude:
    - .git/**
    - node_modules/**
    - .venv/**

The colleague had created their virtual environment in a directory called venv instead of .venv, so serverless didn’t know to skip their virtual environment when packaging the lambda. It was trying to read everything in their virtual environment which blew up the memory.

We updated the glob pattern to the following:

package:
  individually: true
  include:
    - '!*'
    - '!**'
    - src/*.py

This pattern prevents anything other than python files in the src dir from getting packaged. It turns out we got speed benefits with this new syntax as well, as the glob pattern is much faster when you use ! in an include rather than listing files in the exclude because the glob takes a while to find every matching file that you want to exclude.

@medikoo I don’t believe that’s the case - this issue can be encountered without having serverless-python-requirements installed. It seems more closely related to the combination of serverless with a virtual environment in the directory being packaged.

https://github.com/jvanalst/emfile-serverless-win10-bug

FWIW - I was/am experiencing this with Python. I created a virtual environment for each Lambda handler - and it looks like serverless is including all of the /Lib and /Scripts directories for each virtual environment, thereby inflating the file count needed to package.

Adding exclusions “solved” this for me.

A colleague saw the EMFILE: too many files open error recently, and the error was only showing up on their computer. It took a while for us to figure it out what the issue was, but it ended up being a package glob issue.

We had manually listed exclude glob patterns in the serverless.yml like this:

package:
  individually: true
  exclude:
    - .git/**
    - node_modules/**
    - .venv/**

The colleague had created their virtual environment in a directory called venv instead of .venv, so serverless didn’t know to skip their virtual environment when packaging the lambda. It was trying to read everything in their virtual environment which blew up the memory.

We updated the glob pattern to the following:

package:
  individually: true
  include:
    - '!*'
    - '!**'
    - src/*.py

This pattern prevents anything other than python files in the src dir from getting packaged. It turns out we got speed benefits with this new syntax as well, as the glob pattern is much faster when you use ! in an include rather than listing files in the exclude because the glob takes a while to find every matching file that you want to exclude.

It worked like a charm 🥇

Same here:

Cannot read file node_modules\rxjs\dist\types\internal\operators\sequenceEqual.d.ts due to: EMFILE: too many open files, open 'C:\Users\...\Desktop\...\...\node_modules\rxjs\dist\types\internal\operators\sequenceEqual.d.ts'

can someone look at this issue?

So i closed my project on VS code, and opened a terminal window in the project directory. I Ran sls deploy and everything worked fine. Could it possibly be a compatibility issue with VS code ?

@medikoo Checking now if I can replicate it if I roll back my project a few commits…

Yup… replicable. And I can trigger it without any Serverless Plugins at all:

org: bananaweb
app: bananarego
service: ${file(./config.js):common.service}
frameworkVersion: '2'
provider:
  name: aws
  runtime: nodejs14.x
  region: us-west-2
functions:
  api:
    handler: api/index.apiHandler

That api/index.js file can be empty, BTW. Nothing in there affects what I’m experiencing.

Currently looking through the package.json to see if anything in there affects it. Stripping the serverless-plugins from devDependencies (with clean npm install of course) … no effect. Stripping ALL the other devDependencies… also no effect.

So… starting to strip out dependency dependencies… also no effect. Well this is annoying.

{
  "name": "bananarego",
  "version": "0.0.0-alpha",
  "private": true,
  "dependencies": {
    "serverless": "^2.65.0"
  },
  "devDependencies": {
  }
}

Is it a problem with Serverless 2.65? If it is… 2.72.2 has the same exact bug in it. So that’s not it probably. So far all we’ve managed to change is which node module shows up in the error message.

Serverless Error ---------------------------------------- Cannot read file node_modules\archiver-utils\node_modules\safe-buffer\index.js due to: EMFILE: too many open files, open ‘C:\Users.…\node_modules\archiver-utils\node_modules\safe-buffer\index.js’

I guess I’ll start randomly removing files from the the project’s base folder… Maybe it’s trying to include something weird like include the frontend that’s explicitly blocked in the original code.

Serverless Error ----------------------------------------

Cannot read file node_modules\aws-sdk\lib\services\cloudfront.d.ts due to: EMFILE: too many open files, open ‘C:\Users.…\node_modules\aws-sdk\lib\services\cloudfront.d.ts’

Okay… wiped the project down to package-lock.json, package.json, api/index.js, and node_modules containing only a serverless v2.27.2 install…

Serverless Error ----------------------------------------

Cannot read file node_modules\kafka-node\node_modules\debug.travis.yml due to: EMFILE: too many open files, open ‘C:\Users.…\node_modules\kafka-node\node_modules\debug.travis.yml’

Wow. Seriously? Still broken? Am I doing something crazy? Okay fine… blow away the package-lock and package.json just in case… so bare node_modules with an api/index.js (empty) and the serverless.yml above.

Serverless Error ----------------------------------------

Cannot read file node_modules@sindresorhus\is\dist\index.d.ts due to: EMFILE: too many open files, open ‘C:\Users.…\node_modules@sindresorhus\is\dist\index.d.ts’

Still broken… Last shot. Gonna try it all in a completely different folder on a completely different hard drive. I’m also going to start from scratch in a totally new folder.

Serverless Error ----------------------------------------

Cannot read file node_modules\traverse\examples\stringify.js due to: EMFILE: too many open files, open ‘C:\Users.…\node_modules\traverse\examples\stringify.js’

Still? I’m not nuts am I? Okay… let’s get out of the Users folder entirely. Maybe Windows is doing something weird like trying to sync OneDrive (even though I’m pretty sure that second SSD isn’t sunk). Oh also, I figured out you don’t actually need that api/index.js file. It’ll still break on “Excluding development dependencies…”

Serverless Error ----------------------------------------

Cannot read file node_modules\d\node_modules\type\CHANGELOG.md due to: EMFILE: too many open files, open ‘C:\test\node_modules\d\node_modules\type\CHANGELOG.md’

Okay… I’m at a total loss. All I can think of is that a bug got introduced to Serverless at some point. That or I’m a bizarre edge case.