serverless: excludeDevDependencies causing 'heap out of memory' & 'too many open files' errors

This is a Bug Report

Description

We have a single serverless.yml file with 21 lambdas, half of which are wired into 1 or more API gateway resources. After updating to 1.22.0 and attempting to deploy for a second time, we consistently receive JavaScript heap out of memory or EMFILE: too many open files after running for upwards of 10 minutes. As soon as I disabled this feature our deployment went just fine. The first deployment succeeded, but all subsequent attempts failed until I made the noted change (no other changes have been made to the serverless.yml file). We are using CircleCI as our CI/CD provider, however I also receive these intermittently on my local machine.

  • What went wrong? Deployment processes aborted abruptly due to out of memory errors.
  • What did you expect should have happened? For the deployment to complete.
  • What was the config you used? I can provide offline individually if necessary.
  • What stacktrace or error message from your provider did you see? Here is one of several we received:
 Error --------------------------------------------------
 
  EMFILE: too many open files, scandir '/home/circleci/zctv/api/services/lib/db/assignments.js'
 
     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless
 
  Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           7.10.0
     Serverless Version:     1.22.0

And here is an out of memory example

<--- Last few GCs --->

[1668:0x2fe5aa0]   898400 ms: Mark-sweep 2363.3 (2690.7) -> 2363.3 (2629.7) MB, 2164.1 / 1.5 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 2415 ms) last resort 
[1668:0x2fe5aa0]   900636 ms: Mark-sweep 2363.3 (2629.7) -> 2363.3 (2604.7) MB, 2235.6 / 1.0 ms  last resort 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2be985ea66a1 <JS Object>
    2: _appendBuffer [/home/circleci/zctv/api/node_modules/compress-commons/lib/archivers/zip/zip-archive-output-stream.js:74] [pc=0x338f7ddf4cc6](this=0x39099bf253d9 <JS Object>,ae=0x7e934870be1 <JS Object>,source=0x2391564e8549 <an Uint8Array with map 0x1efc20fac579>,callback=0x7e934870b01 <JS BoundFunction (BoundTargetFunction 0x7e934870ab9)>)
    3: entry [/home/circleci/zctv/api/node_modul...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [node]
 2: 0x12b82ac [node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Factory::NewCode(v8::internal::CodeDesc const&, unsigned int, v8::internal::Handle<v8::internal::Object>, bool, bool, int, bool) [node]
 6: v8::internal::CodeGenerator::MakeCodeEpilogue(v8::internal::MacroAssembler*, v8::internal::EhFrameWriter*, v8::internal::CompilationInfo*, v8::internal::Handle<v8::internal::Object>) [node]
 7: v8::internal::FullCodeGenerator::MakeCode(v8::internal::CompilationInfo*, unsigned long) [node]
 8: v8::internal::FullCodegenCompilationJob::ExecuteJobImpl() [node]
 9: v8::internal::CompilationJob::ExecuteJob() [node]
10: 0xd588dc [node]
11: 0xd5b0aa [node]
12: 0xd5c8a9 [node]
13: v8::internal::Compiler::Compile(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Compiler::ClearExceptionFlag) [node]
14: v8::internal::Runtime_CompileLazy(int, v8::internal::Object**, v8::internal::Isolate*) [node]
15: 0x338f7d4843a7
/bin/bash: line 6:  1668 Aborted                 node node_modules/.bin/sls deploy -s dev
Exited with code 134

Additional Data

  Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           7.10.0
     Serverless Version:     1.22.0

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 4
  • Comments: 15 (8 by maintainers)

Most upvoted comments

Hi @pmuens, we upgraded from 1.2.1 to 1.22. All was working fine prior to the upgrade. We didn’t try any versions in between. (credit to the serverless team / contributors as we didn’t have to make any changes to serverless.yml to make 1.22 deploy).

Yes, the only change I made to get past the OOM errors was to opt out of the dev exclusion.

package:
  individually: true
  excludeDevDependencies: false

We run into a similar issue with version 1.32 (Yet to test against the latest release). The following configuration did resolve it:

package:
  individually: false
  excludeDevDependencies: true

Additionally, we traced it back to the serverless-webpack plugin we were using but were unable to determine the exact root cause of it.