vscode: VS Code often does not close after test runs complete

For the last few weeks, my automated test runs have been intermittently (but very frequently) hanging after the tests complete. Mocha writes that the tests completed, with the number of passes, but the VS Code process never terminates, so on Travis we end up with something like:

 ✓ Dart: New Project can be invoked and creates trigger file
  3 passing (5s)

No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
The build has been terminated

When it works, the next line printed after x passing would be Exit code: x which is printed by vscode-test.

I added some code to log before the callback passed into the test runner is called, as well as logging if the process is still alive after 10 seconds:

https://github.com/Dart-Code/Dart-Code/blob/291489432857a433308ab43910d0bbb225cdbfd4/src/test/test_runner.ts#L30-L37

And the output now looks like:

 3 passing (3s)

Test run is complete! Calling VS Code callback with (null, 0)

No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received

This suggests that Mocha is correctly finishing the run, and that the test runner code (which I adapted from https://github.com/microsoft/vscode-test/blob/master/sample/test/suite/index.ts) is correctly calling the callback, and the extension host is disappearing within 10 seconds (since it didn’t print the second line), yet VS Code never quit so the CI build just hung.

I tried passing --verbose to Code, and then the output ended like this:

Test run is complete! Calling VS Code callback with (null, 0)

[main 2019-10-16T07:52:02.179Z] IPC#vscode:exit 0
[main 2019-10-16T07:52:02.180Z] Lifecycle#kill()
[main 2019-10-16T07:52:02.181Z] Lifecycle#window.on('closed') - window ID 1

No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received

So my guess is that either

• VS Code is not terminating; or
• The close event is not firing in vscode-test so it never completes its promise.

I spent a lot of time last week and this week trying to diagnose this, but I feel like I’m running out of ideas. I am completely unable to reproduce it locally (and as far as I can tell, it only happens on macOS on Travis - I don’t think I’ve seen it on Linux).

I’m trying to make a smaller repo that has this issue (my test runs are quit big and involve downloading SDKs), but I figured it was worth opening an issue in case others are seeing it, or you have any suggestions on how to track down.

Edit: Repro is further down in the comments.

I don’t know if it’s related (wouldn’t explain why it’s intermittent), but the example test runner script on the website is very different to the one in the sample repo (one returns a promise, the other takes a callback). I couldn’t find any explanation for the difference.