nest: Disgraceful shutdown, request before signal interrupted with ECONNREFUSED

Is there an existing issue for this?

  • I have searched the existing issues

Current behavior

When the application is shutdown (e.g. with SIGTERM shutdown hook), requests that started before shutdown fail with ECONNREFUSED.

Minimum reproduction code

https://github.com/soryy708/nestjs-disgraceful-shutdown-mcve

Steps to reproduce

  1. Install dependencies by running yarn install
  2. Run the tests by running yarn test

Expected behavior

Requests that started before shutdown should succeed, only new requests after shutdown should fail.

Package

Other package

No response

NestJS version

9.3.12

Packages versions

{
"@nestjs/axios": "^2.0.0",
"@nestjs/common": "^9.3.12",
"@nestjs/core": "^9.3.12",
"@nestjs/platform-express": "^9.3.12",
"@nestjs/terminus": "^9.2.2",
"@nestjs/testing": "^9.3.12",
}

Node.js version

16.19.0

In which operating systems have you tested?

  • macOS
  • Windows
  • Linux

Other

No response

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Reactions: 3
  • Comments: 25 (9 by maintainers)

Most upvoted comments

I can see how this issue pops up for regular health checks. But this can also be useful for SSE! I think we should consider keeping this logic in @nestjs/platform-express, but gate it behind an option. Just like Fastify does. We can then introduce this option on the application level. But if you’re of the opinion that we should handle this in @nestjs/terminus, we can keep it that way!

Either way works for me. Since it can be useful for SSE too (and we expose abstractions for SSE from the core) then having it in the platform-specific packages would make sense too!

I’m so sorry, it works exactly as expected when trying again on a fresh project. I was trying something with a special configuration. Have a nice day! This is not an issue for me anymore.

Honestly I’m not sure, I just tried on a new project and it closes open connections with the default. Here is the current implementation of the closeOpenConnections method

this.openConnections will always be empty unless trackOpenConnections (https://github.dev/nestjs/nest/blob/da27730cd4ef37fdf7f29aafee53ed8ac2a8b4b8/packages/platform-express/adapters/express-adapter.ts#L421) is called and it’s executed only if forceCloseConnections is true https://github.dev/nestjs/nest/blob/da27730cd4ef37fdf7f29aafee53ed8ac2a8b4b8/packages/platform-express/adapters/express-adapter.ts#L234

I believe the error lies somewhere here or in one these methods. Somewhere in one of these, the request is being stopped

Screenshot from 2023-04-05 19-49-50

Screenshot from 2023-04-06 15-18-11

So, I could be wrong, but what I have found is that the callDestroyHook method is iteratively calling callModuleDestroyHook and that is what is removing the modules via the shift method on the list of providers. So, what my intuition is telling me, is that the provider is being removed while the controller is utilizing it and thus the request is just being stopped. Again, I could be wrong, but I think this is a potential root cause.