Ghost: Worker initialization failure: EMFILE
Context
Every 2 weeks or so my Ghost website (hosted on a Digital Ocean droplet) crashes with this error message:
[2020-12-27 05:30:19] ERROR
CODE: ERR_WORKER_INIT_FAILED
MESSAGE: Worker initialization failure: EMFILE
Error [ERR_WORKER_INIT_FAILED]: Worker initialization failure: EMFILE
at Worker.[kOnExit] (internal/worker.js:229:26)
at Worker.<computed>.onexit (internal/worker.js:165:20)
[2020-12-27 05:30:19] INFO Worker for job "email-analytics-fetch-latest" exited with code 0
Reproduction scenario
A possible scenario confirming the issue described in this comment below.
Environment
Most cases reported come from: Ghost Version: 3.41.2 Node Version: 12.18.0 Server OS: Ubuntu 18.04 Database: mysql Ver 14.14 Distrib 5.7.32
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 43 (17 by maintainers)
Commits related to this issue
- ✨ Added workerMessageHandler option to ctr options refs https://github.com/TryGhost/Ghost/issues/12496 - `workerMessageHandler` option allows for custom worker message handling and allows to elimina... — committed to TryGhost/Utils by naz 3 years ago
- Migrated jobs to use parentPort.postMessage refs https://github.com/TryGhost/Ghost/issues/12496 - Using ghost-ignition logging caused file handle leaks. As there is no straight-forward way to handle... — committed to naz/Ghost by naz 3 years ago
- Added custom worker message handler refs https://github.com/TryGhost/Ghost/issues/12496 - Handling logging in the main thread avoids file handle leaks which happen due to leaky implementation of bun... — committed to naz/Ghost by naz 3 years ago
- Migrated jobs to use parentPort.postMessage refs https://github.com/TryGhost/Ghost/issues/12496 - Using ghost-ignition logging caused file handle leaks. As there is no straight-forward way to handle... — committed to TryGhost/Ghost by naz 3 years ago
- Added custom worker message handler refs https://github.com/TryGhost/Ghost/issues/12496 - Handling logging in the main thread avoids file handle leaks which happen due to leaky implementation of bun... — committed to TryGhost/Ghost by naz 3 years ago
- Fixed analytics job creating logging instances refs https://github.com/TryGhost/Ghost/issues/12496 By requiring the models layer the shared logging util was being required as a side-effect causing t... — committed to kevinansfield/Ghost by kevinansfield 3 years ago
- Removed models require from analytics job (#12689) refs https://github.com/TryGhost/Ghost/issues/12496 By requiring the models layer the shared logging util was being required as a side-effect cau... — committed to TryGhost/Ghost by kevinansfield 3 years ago
- Removed logging require in `db/connection.js` refs https://github.com/TryGhost/Ghost/issues/12496 - having the logging require here means that workers wanting to use the db are unable to do so witho... — committed to kevinansfield/Ghost by kevinansfield 3 years ago
- Removed logging require in `db/connection.js` (#12690) refs https://github.com/TryGhost/Ghost/issues/12496 - having the logging require here means that workers wanting to use the db are unable to ... — committed to TryGhost/Ghost by kevinansfield 3 years ago
- Migrated jobs to use parentPort.postMessage refs https://github.com/TryGhost/Ghost/issues/12496 - Using ghost-ignition logging caused file handle leaks. As there is no straight-forward way to handle... — committed to TryGhost/Ghost by naz 3 years ago
- Added custom worker message handler refs https://github.com/TryGhost/Ghost/issues/12496 - Handling logging in the main thread avoids file handle leaks which happen due to leaky implementation of bun... — committed to TryGhost/Ghost by naz 3 years ago
- Removed models require from analytics job (#12689) refs https://github.com/TryGhost/Ghost/issues/12496 By requiring the models layer the shared logging util was being required as a side-effect cau... — committed to TryGhost/Ghost by kevinansfield 3 years ago
- Removed logging require in `db/connection.js` (#12690) refs https://github.com/TryGhost/Ghost/issues/12496 - having the logging require here means that workers wanting to use the db are unable to ... — committed to TryGhost/Ghost by kevinansfield 3 years ago
- Migrated jobs to use parentPort.postMessage refs https://github.com/TryGhost/Ghost/issues/12496 - Using ghost-ignition logging caused file handle leaks. As there is no straight-forward way to handle... — committed to dmitrymarokhonov/Ghost by naz 3 years ago
- Added custom worker message handler refs https://github.com/TryGhost/Ghost/issues/12496 - Handling logging in the main thread avoids file handle leaks which happen due to leaky implementation of bun... — committed to dmitrymarokhonov/Ghost by naz 3 years ago
- Removed models require from analytics job (#12689) refs https://github.com/TryGhost/Ghost/issues/12496 By requiring the models layer the shared logging util was being required as a side-effect cau... — committed to dmitrymarokhonov/Ghost by kevinansfield 3 years ago
- Removed logging require in `db/connection.js` (#12690) refs https://github.com/TryGhost/Ghost/issues/12496 - having the logging require here means that workers wanting to use the db are unable to ... — committed to dmitrymarokhonov/Ghost by kevinansfield 3 years ago
- ✨ Added workerMessageHandler option to ctr options refs https://github.com/TryGhost/Ghost/issues/12496 - `workerMessageHandler` option allows for custom worker message handling and allows to elimina... — committed to TryGhost/Ghost by naz 3 years ago
Hi all! 👋🏻 We’ve released 3.41.7 with a fix for this so I’d recommend upgrading to this version with Ghost-CLI. If you continue to spot anything off - please do let us know! 🙂
Thank you for addressing this issue. I have been experiencing this for some time and finally got around to working on the issue when I stumbled on this. I have upgraded to latest release and will keep an eye on it!
Thanks for all the additional debug info. If anyone else encounters this issue and wants to add theirs its always helpful.
I realise this is frustrating, but all of the members and email features are currently beta features, so this kind of thing is expected.
A potential workaround short term is to disable email analytics in config.production.com There are two options:
emailAnalytics: falseturns off the background jobs and hides any analytics related things in the admin.backgroundJobs: {emailAnalytics: false}turns off the background jobs but will still show historic stats and set the open tracking flag on emails if that’s turned on.As @kevinansfield mentioned, turning off the open tracking does not turn off the job.
As for what’s causing the bug, I’m pretty sure it’s something to do with how log files are handled inside the workers as hinted at here.
I’m curious if there’s any common pattern with the errors, has anyone noticed if they happen at a specific time of day, at a common interval, or a certain time after an email is sent? Also curious if you guys have a lot of members/emails.
Also, curious if there’s anything weird in your log directory.
@steakscience Thanks for that information!
@zackify & @reloaddoc Please take the time to read the thread before commenting. I have already provided an explanation, simple way to prevent it and a request for useful debugging information:
https://github.com/TryGhost/Ghost/issues/12496#issuecomment-773585550
Can everyone who is reporting this issue please confirm their debug info:
The original reporter has deleted that part of the issue template, making this incredibly hard to debug. Thanks to @jliikala for providing this, but if we could get a few more that would really help with reproduction.
@reloaddoc Follow this format, but make it
falsehttps://github.com/TryGhost/Ghost/blob/main/core/shared/config/defaults.json#L119Very frustrating to find my site throwing a 502 every few days lately… Is the increasing the max files allowed fixed it for anyone here?
Hey @onurozer , this one could be worth investigating and filing as a separate bug. Would appreciate if you could do it!
How do I track file handles for an extended period of time? As well, I should apparently reiterate that “it seems like the number of active handles with Ghost is really low (10% of the limit)”.