node-slack-sdk: Socket mode: Bolt doesn't handle WebSocket disconnections gracefully

👋 Hi, all the developers watching this thread!

We’ve improved the SocketModeClient internals a lot to be more robust. You can try the updated one by installing the RC version by running the following command:

npm i @slack/socket-mode@next
# or npm i @slack/socket-mode@1.3.0-rc.0

If you are interested in all the changes included in the release, refer to https://github.com/slackapi/node-slack-sdk/releases/tag/%40slack%2Fsocket-mode%401.3.0-rc.0 for the details.

If everything goes well, we will release v1.3.0 next week!

I also notice connection failures lately. Twice today actually. Here are the debug logs. App is stuck at this state until i manually intervene and do an app restart. Though stuck at this state, it is still able to receive events but eventually it errors before it reaches the code app.event('app_mention', handler)

[DEBUG]  socket-mode:SocketModeClient:0 calling ack events_api
[DEBUG]  socket-mode:SocketModeClient:0 send() in state: connected,ready
[DEBUG]  socket-mode:SocketModeClient:0 sending message on websocket: {"envelope_id":"xxx","payload":{}}
[ERROR]  socket-mode:SocketModeClient:0 failed to send message on websocket: WebSocket is not open: readyState 3 (CLOSED)
(node:18) UnhandledPromiseRejectionWarning: Error: Failed to send message on websocket: WebSocket is not open: readyState 3 (CLOSED)
    at Object.websocketErrorWithOriginal (/opt/app/node_modules/@slack/socket-mode/dist/errors.js:29:33)
    at /opt/app/node_modules/@slack/socket-mode/dist/SocketModeClient.js:347:48
    at sendAfterClose (/opt/app/node_modules/@slack/socket-mode/node_modules/ws/lib/websocket.js:768:5)
    at WebSocket.send (/opt/app/node_modules/@slack/socket-mode/node_modules/ws/lib/websocket.js:345:7)
    at /opt/app/node_modules/@slack/socket-mode/dist/SocketModeClient.js:344:32
    at new Promise (<anonymous>)
    at SocketModeClient.send (/opt/app/node_modules/@slack/socket-mode/dist/SocketModeClient.js:329:16)
    at ack (/opt/app/node_modules/@slack/socket-mode/dist/SocketModeClient.js:469:24)
    at App.processEvent (/opt/app/node_modules/@slack/bolt/dist/App.js:428:19)
    at runMicrotasks (<anonymous>)
(node:18) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 122)

version 1.3.0 is now available. Please upgrade to the latest version 👋

Anyone trying this, note my mistake in the first commit ⬆️ – if you’re using bolt, then socket-mode is a transitive dependency.

We encountered a very similar problem. But WebSocket disconnections occur constantly.

[ERROR]  socket-mode:SocketModeClient:0 A websocket error occurred: Client network socket disconnected before secure TLS connection was established
[INFO]  socket-mode:SocketModeClient:0 A ping wasn't received from the server before the timeout of 30000ms!
[ERROR]  socket-mode:SocketModeClient:0 cannot send message when client is not ready
/usr/src/app/node_modules/@slack/socket-mode/dist/errors.js:63
    return errorWithCode(new Error('Cannot send message when client is not ready'), ErrorCode.NoReplyReceivedError);
                         ^

Error: Cannot send message when client is not ready
    at sendWhileNotReadyError (/usr/src/app/node_modules/@slack/socket-mode/dist/errors.js:63:26)
    at /usr/src/app/node_modules/@slack/socket-mode/dist/SocketModeClient.js:343:60
    at new Promise (<anonymous>)
    at SocketModeClient.send (/usr/src/app/node_modules/@slack/socket-mode/dist/SocketModeClient.js:334:16)
    at ack (/usr/src/app/node_modules/@slack/socket-mode/dist/SocketModeClient.js:473:24)
    at App.processEvent (/usr/src/app/node_modules/@slack/bolt/dist/App.js:531:19)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async SocketModeClient.<anonymous> (/usr/src/app/node_modules/@slack/bolt/dist/receivers/SocketModeReceiver.js:142:13) {
  code: 'slack_socket_mode_no_reply_received_error'
}

Meanwhile, i did this as a workaround. Not sure if it is the right approach to depend on app.receiver.client.badConnection, but based on my testing seems to works well. Alternatively, would be great for the library to support a health check ability for the bot. My deploy setup would restart the bot if /health fails with 500

  httpApp.get('/health', (req, res) => {
    if (app.receiver.client.badConnection) {
      res.status(500).send('Internal Server Error');
      return;
    }

    res.status(200).send(app.receiver.client.badConnection);
  });

We encountered this exact issue as well. WebSocket disconnections seem to happen pretty rarely but still, I think that this is a pretty major issue for socket mode bots.

Something I thought I would point out is that the default handling of unhandled promise rejections has changed between Node 14.x and 16.x. In 14.x (latest LTS) you just get UnhandledPromiseRejectionWarning but in 16.x the application will be terminated. This probably explains the different behavior reported by @Zenexer and @usubram .

We’re running in 14.x and decided that the easiest fix for now was to add a handler for unhandledRejection that forces the application to crash (so our process manager could then restart it):

process.on("unhandledRejection", error => {
	throw error;
});

But anyhow, restarting the whole app just to reconnect lost WebSocket connection doesn’t seem like a good long term solution…