serverless: excludeDevDependencies causing 'heap out of memory' & 'too many open files' errors
This is a Bug Report
Description
We have a single serverless.yml file with 21 lambdas, half of which are wired into 1 or more API gateway resources. After updating to 1.22.0 and attempting to deploy for a second time, we consistently receive JavaScript heap out of memory or EMFILE: too many open files after running for upwards of 10 minutes. As soon as I disabled this feature our deployment went just fine. The first deployment succeeded, but all subsequent attempts failed until I made the noted change (no other changes have been made to the serverless.yml file). We are using CircleCI as our CI/CD provider, however I also receive these intermittently on my local machine.
- What went wrong? Deployment processes aborted abruptly due to out of memory errors.
- What did you expect should have happened? For the deployment to complete.
- What was the config you used? I can provide offline individually if necessary.
- What stacktrace or error message from your provider did you see? Here is one of several we received:
Error --------------------------------------------------
EMFILE: too many open files, scandir '/home/circleci/zctv/api/services/lib/db/assignments.js'
For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Forums: forum.serverless.com
Chat: gitter.im/serverless/serverless
Your Environment Information -----------------------------
OS: linux
Node Version: 7.10.0
Serverless Version: 1.22.0
And here is an out of memory example
<--- Last few GCs --->
[1668:0x2fe5aa0] 898400 ms: Mark-sweep 2363.3 (2690.7) -> 2363.3 (2629.7) MB, 2164.1 / 1.5 ms (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 2415 ms) last resort
[1668:0x2fe5aa0] 900636 ms: Mark-sweep 2363.3 (2629.7) -> 2363.3 (2604.7) MB, 2235.6 / 1.0 ms last resort
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x2be985ea66a1 <JS Object>
2: _appendBuffer [/home/circleci/zctv/api/node_modules/compress-commons/lib/archivers/zip/zip-archive-output-stream.js:74] [pc=0x338f7ddf4cc6](this=0x39099bf253d9 <JS Object>,ae=0x7e934870be1 <JS Object>,source=0x2391564e8549 <an Uint8Array with map 0x1efc20fac579>,callback=0x7e934870b01 <JS BoundFunction (BoundTargetFunction 0x7e934870ab9)>)
3: entry [/home/circleci/zctv/api/node_modul...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [node]
2: 0x12b82ac [node]
3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
5: v8::internal::Factory::NewCode(v8::internal::CodeDesc const&, unsigned int, v8::internal::Handle<v8::internal::Object>, bool, bool, int, bool) [node]
6: v8::internal::CodeGenerator::MakeCodeEpilogue(v8::internal::MacroAssembler*, v8::internal::EhFrameWriter*, v8::internal::CompilationInfo*, v8::internal::Handle<v8::internal::Object>) [node]
7: v8::internal::FullCodeGenerator::MakeCode(v8::internal::CompilationInfo*, unsigned long) [node]
8: v8::internal::FullCodegenCompilationJob::ExecuteJobImpl() [node]
9: v8::internal::CompilationJob::ExecuteJob() [node]
10: 0xd588dc [node]
11: 0xd5b0aa [node]
12: 0xd5c8a9 [node]
13: v8::internal::Compiler::Compile(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Compiler::ClearExceptionFlag) [node]
14: v8::internal::Runtime_CompileLazy(int, v8::internal::Object**, v8::internal::Isolate*) [node]
15: 0x338f7d4843a7
/bin/bash: line 6: 1668 Aborted node node_modules/.bin/sls deploy -s dev
Exited with code 134
Additional Data
Your Environment Information -----------------------------
OS: linux
Node Version: 7.10.0
Serverless Version: 1.22.0
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 4
- Comments: 15 (8 by maintainers)
Hi @pmuens, we upgraded from 1.2.1 to 1.22. All was working fine prior to the upgrade. We didn’t try any versions in between. (credit to the serverless team / contributors as we didn’t have to make any changes to serverless.yml to make 1.22 deploy).
Yes, the only change I made to get past the OOM errors was to opt out of the dev exclusion.
We run into a similar issue with version 1.32 (Yet to test against the latest release). The following configuration did resolve it:
Additionally, we traced it back to the serverless-webpack plugin we were using but were unable to determine the exact root cause of it.