node-slack-sdk: WebSocket error occurred: Unexpected server response: 409
Socket mode receiver fails to make a successful connection due to WebSocket error occurred: Unexpected server response: 409 error. Retries after this error occurs may eventually lead to a successful connection or worse, rate limiting from Slack servers on the apps.connections.open call.
@slack/bolt version
4.0.1
Your App and Receiver Configuration
Socket mode 2.0.2
Node.js runtime version
v20.17.0
Steps to reproduce:
This error seems to reproduce only when launching in a distributed environment (i.e. launching 2 or more instances of the app). Nothing special is being done on start up. It is minimally reproducible using the example from BoltJS Getting Started docs as a basis.
const app = new App({
socketMode: true,
token: process.env.BOT_TOKEN,
signingSecret: process.env.SIGNING_SECRET,
appToken: process.env.APP_TOKEN,
logger
});
void (async () => {
// Start your app
await app.start(Number(process.env.PORT) || 3000);
console.log('⚡️ Bolt app is running!');
})();
- Open multiple shell terminals (suggested 5) and change
process.env.PORTto a different number in each shell - Quickly start the bolt app in each shell (roughly simultaneously to mimic deployment as a server group)
Expected result:
Each instance of the app should start up successfully without the WebSocket error occurred: Unexpected server response: 409 error
Actual result:
First we see the Socket Mode client successfully initialized.
[14:05:55.636] DEBUG (36123): Initializing SocketModeReceiver
[14:05:55.636] DEBUG (36123): The logLevel given to Socket Mode was ignored as you also gave logger
[14:05:55.637] DEBUG (36123): The logLevel given to WebClient was ignored as you also gave logger
[14:05:55.637] DEBUG (36123): initialized
[14:05:55.637] DEBUG (36123): The Socket Mode client has successfully initialized
[14:05:55.637] DEBUG (36123): apiCall('auth.test') start
[14:05:55.638] DEBUG (36123): http request url: https://slack.com/api/auth.test
[14:05:55.638] DEBUG (36123): http request body: {"token":"[[REDACTED]]"}
[14:05:55.638] DEBUG (36123): http request headers: {"Accept":"application/json, text/plain, */*","User-Agent":"@slack:socket-mode/2.0.2 @slack:bolt/4.0.1 @slack:web-api/7.7.0 node/20.17.0 linux/6.8.0-1017-aws","Authorization":"[[REDACTED]]"}
Then we see the 409 error and the following logs repeat over and over again until:
- the connection is established OR
- the app is rate limited
[14:05:55.639] DEBUG (36123): Starting Socket Mode session ...
[14:05:55.639] DEBUG (36123): Going to retrieve a new WSS URL ...
[14:05:55.639] DEBUG (36123): apiCall('apps.connections.open') start
[14:05:55.639] DEBUG (36123): http request url: https://slack.com/api/apps.connections.open
[14:06:46.734] DEBUG (36239): http request body: {}
[14:06:46.734] DEBUG (36239): http request headers: {"Accept":"application/json, text/plain, */*","User-Agent":"@slack:socket-mode/2.0.2 @slack:bolt/4.0.1 @slack:web-api/7.7.0 node/20.17.0 linux/6.8.0-1017-aws","Authorization":"[[REDACTED]]"}
[14:06:46.755] DEBUG (36239): http response received
[14:06:46.755] DEBUG (36239): http request result: {"ok":true,"url":"wss://wss-primary.slack.com/link/?ticket=084fad38-f4fe-4fba-9057-fedc6792738f&app_id=4cd5f1068f00f78fe83741337c42c958d96b3b3b13a7e17b4cda776e289e362c","response_metadata":{"scopes":["connections:write"],"acceptedScopes":["connections:write"]}}
[14:06:46.755] DEBUG (36239): apiCall('apps.connections.open') end
[14:06:46.755] DEBUG (36239): Initiating new WebSocket connection.
[14:06:46.768] ERROR (36239): WebSocket error occurred: Unexpected server response: 409
[14:06:46.768] DEBUG (36239): WebSocket already disconnected, flushing remainder.
[14:06:46.768] DEBUG (36239): Before trying to reconnect, this client will wait for 5000 milliseconds
[14:06:46.768] ERROR (36239): WebSocket error! Error: Failed to send message on websocket: Unexpected server response: 409
[14:06:46.768] DEBUG (36239): WebSocket close frame received (code: 1006, reason: )
[14:06:46.768] DEBUG (36239): WebSocket already disconnected, flushing remainder.
[14:06:46.768] DEBUG (36239): Before trying to reconnect, this client will wait for 10000 milliseconds
[14:06:46.779] DEBUG (36239): Continuing with reconnect...
Additional Notes
It seems that a successfully established connection does not mean the app is in a “safe” state. While running, I’ve observed both Primary WebSocket error occurred: Unexpected server response: 409 and Secondary WebSocket error occurred: Unexpected server response: 409 errors as well. The behavior is basically identical to as described above.
I’ve also observed this on @slack/bolt-js versions as early as 3.20.0.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 3
- Comments: 24 (11 by maintainers)
Another update: on Friday afternoon EST, a backend configuration problem was identified and we started partially rolling out a fix. Since then, we have seen metrics indicating that 409 errors are properly being consumed internally which should prevent them from being raised to applications. This fix will be rolled out fully today (Monday) and tomorrow, as so far it is only being partially rolled out to monitor impact on applications.
We expect the occurrence of 409s should have reduced since Friday and for them to be gone by Tuesday.
Not exactly. I believe there are two issues that lead to multiple retries being executed simultaneously.
Issue 1
When the socket connection fails with the
409error, both theerrorandcloseevents are emitted from the WebSocketThis leads to
SocketModeClientcalling delayReconnectAttempt twice for a single failurethis.clientPingTimeoutMS * this.numOfConsecutiveReconnectionFailuresandthis.clientPingTimeoutMS * this.numOfConsecutiveReconnectionFailures + 1.This happens because both the
errorandcloseevent handlers callSocketModeClient.disconnectTo avoid this,
SocketModeClientshould check if a retry is pending before callingdelayReconnectAttempt.Issue 2
numOfConsecutiveReconnectionFailures is reset when
apps.connections.opensucceeds.This can lead to retries being scheduled earlier than they should.
Consider the scenario where:
apps.connections.opensucceeds andnumOfConsecutiveReconnectionFailuresset to 0`The retry timing will be:
apps.connections.opensucceeds; socket connect failsapps.connections.opensucceeds; socket connection failsYou can replicate this by running the experiment with
MAX_CALLS=4.I suspect that either of following would be preferable:
numOfConsecutiveReconnectionFailuresonly when the socket connection is actually established ORdelayReconnectAttemptretry mechanisms for theapps.connections.opencall and WebSocket connection failure modesHowever, that might not be necessary as long as Issue 1 is addressed.
Answers to other follow up questions.
I wasn’t able to capture debug level logs when I encountered this in production. But the reproduction experiment is pretty close if not identical I think.
Actually, after trying, I don’t think so. I thought setting
autoReconnectEnabledtofalsein conjunction withrejectRateLimitedCallson theWebClientmight help me catch and implement my own retry logic but that seemed like more effort than just fixing the root problemUpdate here: I have escalated this issue to the real-time server backend team to raise the issue related to unexpected 409 responses from that server.
I am still working on addressing the issue of what this socket-mode library should do in this situation and how to prevent it from getting into a rate-limit situation with the
apps.connections.openAPI.This does seem to be an infrastructural issue; from what I understand, the 409 response should be an internal-only signal for the real-time backend system to find another container to serve the connection request. For whatever reason (a recent change or bug), it is bubbling up to the public / to the apps. The infrastructure team is still investigating the cause.
No problem! I appreciate everyone’s diligence fixing this issue and I’m glad this landed.
Update: the backend fix has been rolled out to 75% and we expect to crank it up to 100% if not today then this week.
Also: as I mentioned, a PR is up with a proposed fix here: https://github.com/slackapi/node-slack-sdk/pull/2099
This fix is available as a pre-release as
@slack/socket-mode@2.0.3-rc.1under the npm dist tag2.0.3rc(seesocket-modeversions here: https://www.npmjs.com/package/@slack/socket-mode?activeTab=versions). Additionally, I have pre-released this pre-release in a bolt-js pre-release (yo dawg) as@slack/bolt@4.1.2-rc.1under the npm dist tag4.1.2rc(seeboltversions here: https://www.npmjs.com/package/@slack/bolt?activeTab=versions). For bolt-js devs affected by this bug, we would love if you could take either of these pre-releases for a spin and let us know how it works for you.We just experienced another outage today. Luckily, we had alerting set up so that we could mitigate within roughly 10-15 minutes. But honestly, even that amount of downtime is tremendously painful for our users.
In case it’s helpful with isolating a change that introduced the bug, we started observing the 409 errors on Oct 28th or 29th. Before that time, these errors virtually never occurred.
What makes this especially painful is that recovery pretty much requires human intervention and downtime is guaranteed:
@bitofsky upgrading to @slack/bolt
4.1.0should resolve the Out of Memory error you’re experiencing. I experienced the same Out of Memory error running3.21.4Hello, I recently encountered the same issue.
When running two instances of a Bolt app, one of the apps would unexpectedly disconnect from the socket connection. During the reconnection process, it would fail repeatedly, doubling the reconnection attempts each time. This cycle continued until it led to an Out of Memory error, eventually causing the app to crash.
Thanks for the analysis. I’ve moved this issue over to the node-slack-sdk (the home of the
socket-modeclient); will sift through everything you’ve posted and see if I can capture the issue as a test to help frame the discussion.Separate issue: I think I will riff on your repro a bit to add
Retry-Afterheader responses to theapps.connections.openendpoint to see if retry handling works as expected. Will see what kind of responses that API raises in a rate limit scenario.I found some time to make a reproduction experiment for what I observed.
The issue seems to lie within in
@slack/socket-mode.SlackWebSocketencounters this409error, it calls to disconnect then it calls to terminatecloseevent to theSocketModeClientand the client is started againI see that the
SocketModeClientis internal toSocketModeReceiver,autoReconnectEnabledistrueby default, and this option is not configureable via AppOptions or SocketModeReceiverOptions. Additionally,isRecoverableistruefor rate limited errors.Given this, it seems that once a rate limit error is encountered, there isn’t a way to recover and the call to
apps.connections.openwill be retried indefinitely.Thanks in advance for investigating. I’m happy to provide any additional information. I’d appreciate any temporary workaround ideas in the short term. Being unexpectedly rate limited during business hours would be a huge disruption as the app where I experienced this is in the critical path for users seeking engineering support from my organization.