nodemon: Nodemon Fails to Wait for Process Exit when using Docker + Inspector

Created per @remy’s request in this issue.

It seems that when restarting a process, Nodemon isn’t waiting for the old process to completely exit before starting a new one, but:

  • only when in a Docker container and
  • only when the inspector/debug mode is active.

This issue can be intermittent in applications without a shutdown handler. But, having an application that implements a listener for Nodemon’s SIGUSR2 signal that takes even a second or so to complete can be enough to cause this issue to happen reliably.

The presence of --inspect does cause the new process to immediately crash because its trying to open a debug server on the same port that is still in use by the previous process’ debug server, but this merely a side-effect of this issue.

If running Nodemon in a Docker container without --inspect, Nodemon correctly waits for the old process to exit, but when --inspect is used inside the container, Nodemon does not wait for the process to exit, leading to an immediate crash.

This can be reproduced with Nodemon 1.18.7 and the following:

$ node -v
v10.13.0

$ docker -v
Docker version 18.09.0, build 4d60db4

$ docker-compose -v
docker-compose version 1.23.1, build b02f1306

Here is a repo that replicates this issue: https://github.com/darkobits/nodemon-restart-issue.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 22
  • Comments: 44 (6 by maintainers)

Commits related to this issue

Most upvoted comments

I’m seeing intermittent results. Sometimes when I save a file change it restarts correctly. Other times it errors out the following message:

crumb_1     | Starting inspector on 0.0.0.0:9229 failed: address already in use
crumb_1     | [nodemon] app crashed - waiting for file changes before starting...
crumb_1     | [nodemon] restarting due to changes...
crumb_1     | [nodemon] starting `node --inspect=0.0.0.0:9229 ./index.js`

What I’m forced to do, as shown above, is save again which triggers nodemon to restart. The second time around works correctly.

  • version: 1.18.9
  • command ./node_modules/.bin/nodemon --inspect=0.0.0.0:9229 ./index.js
+4
justinpage on Jan 6, 2019

Just started to see this issue when I upgraded to latest versions of nodemon too.

Ubuntu 18 nodemon 1.18.6 global node 10.7.0 docker 18.06.1-ce build e68fc7a

Running this with vscode debugging. Worked perfectly fine previously until I did some package upgrades after the event-stream vulnerability.

NODE_ENV=development nodemon --legacy-watch --delay 2000ms --inspect-brk=0.0.0.0:9222 --nolazy index.js",
+4
ImTheDeveloper on Dec 6, 2018

Hello, today I worked on this and I think I have understood what is happening here.

First of all it seems, that this is only a Linux related issue (from the comments I see docker and Ubuntu). I had this problem since several weeks with Ubuntu 18.

The problem is, that even if you think nodemon starts only one child process, nodemon actually starts two processes: one sh-process AND one node-process (run.js:97). The child.on(‘exit’)-callback in run.js only gets events for the sh-Process (because it’s the parent of the node process). The node documentation explicitly says, that on Linux platform terminating the shell command does not necessarily kill all the shell child commands:

On Linux, child processes of child processes will not be terminated when attempting to kill their parent. This is likely to happen when running a new process in a shell or with the use of the shell option of ChildProcess.

(see https://nodejs.org/api/child_process.html#child_process_subprocess_kill_signal) Although here the native kill command is executed in line run.js:368, it seems it behaves the same way.

Explicitly waiting for all other kids to be terminated before restarting the application solves the problem.

See my PR #1579

+3
marcelkottmann on Jun 7, 2019

I’m still seeing the issue with 1.18.8

+3
bighuggies on Dec 13, 2018

Hello @remy , sorry to say that, but this issue is back again since version 2.0.2. It was fixed in 2.0.0 and 2.0.1. I just had the chance to upgrade to version 2.0.2 and this issue is back again. I looked at 47dfb8be3ff06184793a55d32db4b6171fa2993f and I guess the new line child.kill(signal); should only be called after all sub-processes have already been terminated. Moving this line behind the waitForSubProcesses-function call, should fix this. Otherwise the exit signal of child will restart the process tooo early (again).

+2
marcelkottmann on Jan 13, 2020

We are also having this exact issue with nodemon@1.18.10

+2
aaronup on Mar 7, 2019

I also have this problem with nodemon@1.18.9 and 1.8.10

+2
d-wagner on Feb 28, 2019

Repro repo that fails outside of docker on OS X @remy https://github.com/nomcopter/nodemon-no-wait-repro

+1
nomcopter on Apr 23, 2019

It’s not working here either

+1
alemarcha on Apr 23, 2019

@remy, thoughts regarding leaving this closed vs. re-opening?

+1
darkobits on Jan 4, 2019

I think this might be solved with the next release going up at the moment - assuming it passes, it’ll land in nodemon@1.18.8.

@darkobits can you test this once it’s up?

+1
remy on Dec 10, 2018
Read more comments on GitHub
← nodemon: nodemon fails to see changes in unix -> windows mounts (samba for example)
nodemon: Nodemon hangs if started without internet connection →