vapor: Server doesn't stop succesfully in CLI after opening WebSocket connections

Steps to reproduce

Create a simple Vapor app that opens WebSocket connections:

import Vapor

extension Application {
  func configure(
    onWebSocketOpen: @escaping (WebSocket) -> (),
    onWebSocketClose: @escaping (WebSocket) -> ()
  ) {
    webSocket("watcher") { _, ws in
      onWebSocketOpen(ws)
      ws.onClose.whenComplete { _ in onWebSocketClose(ws) }
    }
  }
}

final class Server {
  private struct Lifecycle: LifecycleHandler {
    weak var server: Server?

    /// Closes all active WebSocket connections
    func shutdown(_ app: Application) {
      try! EventLoopFuture<()>.andAllSucceed(
        server?.connections.map { $0.close() } ?? [],
        on: app.eventLoopGroup.next()
      ).wait()
      print("shutdown finished")
    }
  }

  private var connections = Set<WebSocket>()
  private let app: Application

  init() throws {
    var env = Environment(name: "development", arguments: ["vapor"])
    try LoggingSystem.bootstrap(from: &env)
    app = Application(env)
    app.configure(
      onWebSocketOpen: { [weak self] in
        self?.connections.insert($0)
      },
      onWebSocketClose: { [weak self] in
        self?.connections.remove($0)
      }
    )
  }

  /// Blocking function that starts the HTTP server
  func run() throws {
    defer { app.shutdown() }
    app.lifecycle.use(Lifecycle(server: self))
    try app.run()
  }
}

try Server().run()

Expected behavior

When this server app is started from command-line and any WebSocket connections are established, the server process can stopped quickly with Ctrl-C.

Actual behavior

The server process can’t be stopped quickly with Ctrl-C. It hangs for about 5 seconds or so and then raises the following error:

shutdown finished
[ ERROR ] Could not stop HTTP server: Abort.500: Server stop took too long.
ERROR: Cannot schedule tasks on an EventLoop that has already shut down. This will be upgraded to a forced crash in future SwiftNIO versions.

Environment

These are the details of revelant package dependencies from Package.resolved:

      {
        "package": "vapor",
        "repositoryURL": "https://github.com/vapor/vapor.git",
        "state": {
          "branch": null,
          "revision": "88293674e2ea017691c56af20d0938dfff7ece04",
          "version": "4.27.0"
        }
      },
      {
        "package": "websocket-kit",
        "repositoryURL": "https://github.com/vapor/websocket-kit.git",
        "state": {
          "branch": null,
          "revision": "b0736014be634475dac4c23843811257d86dcdc1",
          "version": "2.1.1"
        }
      }

Vapor Framework version: 4.27.0, but this was reproducible in versions as old as 4.5.0
Vapor Toolbox version: not installed
OS version: macOS 10.15.6

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 30 (24 by maintainers)

Commits related to this issue

Close WebSocket connections after HTTP server exits (#85) Resolve #7 as described in https://github.com/vapor/vapor/issues/2451. Additionaly, new formatting is applied with the new version of Swif... — committed to swiftwasm/carton by MaxDesiatov 4 years ago

Most upvoted comments

I’ve confirmed with the NIO team that ChannelShouldQuiesceEvent is indeed meant to be used in this way. Patch incoming.

tanner0101 on Aug 14, 2020

Perfect, I’m able to repro now. Digging in. Thanks!

tanner0101 on Aug 14, 2020

Right, and as for patch 1, would it make more sense for Vapor to close WebSocket connections automatically in (some, I’m not sure which) deinit?

@MaxDesiatov I agree this is not as easy / obvious as it could be. I think I need to do a bit more research here to see how other libraries handle this. The problem is that Vapor is not the one holding onto these WebSockets. The app.webSocket closure simply gives you the WebSocket as it comes in. In carton’s case, it’s the Server type that is holding onto these WebSockets and keeping them alive. So, it makes sense that Server should also be responsible for closing them down. If Vapor were to also be keeping track of these connections and attempting to close them, things could get weird. Maybe this is the right move though. Another option would be Vapor offering a higher level WebSocket API (this has been much requested) or providing better documentation for how to properly manage active WebSocket connections.

Makes sense. Would you like me to create a new issue with this and label it as an enhancement?

@fouadhatem yes I would appreciate that!

tanner0101 on Aug 14, 2020

Hi @tanner0101

I was able to recreate this on both Catalina 10.15.6 and Ubuntu. I’ll lay out the procedure for Ubuntu as tried out on a VM, because I wanted to try and narrow down if if this was a package dependency issue, since you reported that you were not able to reproduce it. Here are the steps I followed:

Start with fresh minimal install of Ubuntu Server 16.04 (also reproduced on 18.04 and 20.04).
Download Swift dependencies as outlined in swift.org.
Download and install latest Swift toolchain (5.2.5 for linux) from swift.org and verify that it is running (though earlier toolchains also showed the same error).
Clone Vapor toolbox from GitHub: $git clone https://github.com/vapor/toolbox and build after switching into toolbox directory: $swift build -c release --disable-sandbox --enable-test-discovery.
Copy build product to /usr/local/bin $sudo mv .build/release/vapor /usr/local/bin
Verify vapor runs by executing vapor --version or vapor --help (by the way, while you’re at it, check out the response you get from the --version option at this stage, particularly the toolbox version).
Switch to a desired working directory and create a new vapor project: vapor new hello-vapor without fluent.
Switch to your new project directory and build your new project using vapor build
Run the project with the bind option -b to attempt to access it from a different machine: vapor run serve -b <IP address>:<port number>
Open a browser in a different (or host) machine and visit the IP address and port combination you used to verify that it works and is accessible.
Terminate the server with CTRL-C and wait for a few seconds.

The error should appear at the CL prompt on your Vapor machine.

Note: No other packages were installed on the VM other than the ones specified as Swift dependencies. While building the VM, I made sure that all additional package options were UNCHECKED, not even standard utilities were included in the VM.

fouadhatem on Aug 11, 2020