selenium: [🐛 Bug]: Draining of nodes not working in a distributed selenium grid

What happened?

On a distributed grid with separate nodes using the standard default AsyncHttpClient and ongoing test on a node, execute

cURL --request POST ‘http://localhost:5553/se/grid/distributor/node/<node-id>/drain’ --header ‘X-REGISTRATION-SECRET: <secret>’

The node is not draining, but immediately crashes the test , removes the node from the grid and stops the node.

How can we reproduce the issue?

On a distributed grid and ongoing test, execute

cURL --request POST 'http://localhost:5553/se/grid/distributor/node/<node-id>/drain' --header 'X-REGISTRATION-SECRET: <secret>'

Relevant log output

Response from cURL command:
'{
  "value": true,
  "message": "Node status was successfully set to draining."
}'

Distributer log:
2023-08-23 19-12:15.827 INFO [GridModel.setAvailability] - Switching Node 07fe9a5d-e932-48c0-a174-57216a860e0f (uri: http://10.199.1.254:5555) from UP to DRAINING
2023-08-23 19-12:17.166 WARN [GridModel.setSession] - Grid model and reality have diverged. Unable to find node 07fe9a5d-e932-48c0-a174-57216a860e0f

Node Log (timestamp from a different test):
19:39:02.875 INFO [LocalNode.drain] - Firing node drain complete message
19:39:03.884 INFO [NodeServer.lambda$createHandlers$3] - Shutting down

Selenium Log:
Could not start a new session. Response code 500. Message: Could not start a new session. Could not start a new session. org.asynchttpclient.exception.RemotelyClosedException: Remotely closed

Operating System

Linux

Selenium version

4.11.0

What are the browser(s) and version(s) where you see this issue?

Chrome 115

What are the browser driver(s) and version(s) where you see this issue?

115.0.5790.110

Are you using Selenium Grid?

4.11.0