aws-iot-device-sdk-java-v2: disconnection with error code [5154] and reason code [SESSION_TAKEN_OVER]
Describe the bug
I have a subsribing client written in Java and Spring Boot.
When I start the client I keep seeing SESSION_TAKEN_OVER.
Already opened a case with AWS support but I was referred to this repo to open an issue here.
I know that SESSION_TAKEN_OVER will happen when I use the same client ID for different clients but I only start one client so this cannot be the issue.
My suspicion is that resources may be leaking from a previous run?
Do I need some cleanup when shutting down my client? I came across CrtResource.waitForNoResources() so is this required?
How can I detect if resources are leaking?
I use my IoT thing name as client ID.
Building my client looks like this:
@Autowired
public IoTCoreMqtt5ConnectionHandler(
@Value("${iot.enabled:false}") Boolean enabled,
@Value("${iot.thing.name}") @NotEmpty String thingName,
@Value("${iot.endpoint.host}") String host,
@Value("${iot.endpoint.port:8883}") @NotNull Long port,
@Value("${iot.auth.certPath}") @NotEmpty String certPath,
@Value("${iot.auth.privateKeyPath}") @NotEmpty String privateKeyPath,
@Value("${iot.auth.caFilePath}") @NotEmpty String caFilePath,
@Value("${iot.subscribeTimeoutSeconds:10}") Long subscribeTimeoutSeconds,
@Value("${iot.keepAliveIntervalSeconds:60}") Long keepAliveIntervalSeconds,
ApplicationEventPublisher applicationEventPublisher, ObjectMapper objectMapper) {
this.enabled = enabled;
this.thingName = thingName;
this.eventTopic = "connector/" + thingName + "/event";
this.host = host;
this.port = port;
this.certPath = certPath;
this.privateKeyPath = privateKeyPath;
this.caFilePath = caFilePath;
this.subscribeTimeoutSeconds = subscribeTimeoutSeconds;
this.keepAliveIntervalSeconds = keepAliveIntervalSeconds;
this.applicationEventPublisher = applicationEventPublisher;
this.objectMapper = objectMapper;
}
for stop:
/**
* Disconnect from the MQTT broker and wait for all CRT resources to be cleaned up.
* A stopped client can be restarted.
*
* @throws ExecutionException if this future completed exceptionally
* @throws InterruptedException if the current thread was interrupted while waiting
*/
private void stop() throws ExecutionException, InterruptedException {
log.debug("stop");
client.stop(new DisconnectPacket.DisconnectPacketBuilder()
.withReasonCode(DisconnectPacket.DisconnectReasonCode.NORMAL_DISCONNECTION)
.withReasonString("client is shutting down")
.build());
}
creating the client:
/**
* Creates a client w/o starting it.
*/
private void createClient() {
if (client != null) {
throw new CreateMqtt5ClientFailedException("Client is already created");
}
ConnectPacket.ConnectPacketBuilder connectPacketBuilder = new ConnectPacket.ConnectPacketBuilder()
.withClientId(thingName)
.withKeepAliveIntervalSeconds(keepAliveIntervalSeconds);
SocketOptions socketOptions = new SocketOptions();
socketOptions.keepAlive = true;
// NEEDS REVIEW maximum of 7 days and even that may require a change to the account limits using the limits process
// currently, although setting 7 days, the negotiated session expiry interval is 3600 seconds (1 hour) which
// would be too short for our use case
connectPacketBuilder.withSessionExpiryIntervalSeconds(60 * 60 * 24 * 7L); // 7 days in seconds
try (AwsIotMqtt5ClientBuilder builder =
AwsIotMqtt5ClientBuilder.newDirectMqttBuilderWithMtlsFromPath(host, certPath, privateKeyPath)
.withConnectProperties(connectPacketBuilder)
.withCertificateAuthorityFromPath(null, caFilePath)
.withLifeCycleEvents(lifecycleEvents)
.withPublishEvents(publishEvents)
// MQTT broker uses keep alive times 1.5 before it drops an idle connection
.withMinReconnectDelayMs((long) (keepAliveIntervalSeconds * 1000 * 1.5))
//.withSocketOptions(socketOptions)
// persistent session (must have same client ID, and a session expiry > 0)
// REVIEW REJOIN_POST_SUCCESS should be enough, but then an existing session was not re-joined
.withSessionBehavior(Mqtt5ClientOptions.ClientSessionBehavior.REJOIN_ALWAYS)
.withPort(port);
) {
client = builder.build();
}
if (client == null) {
throw new CreateMqtt5ClientFailedException("Failed to create the MQTT-5 client");
}
}
starting it:
/**
* Connect to the MQTT broker using MQTT-5.
* A client can be stopped and restarted.
* If the client has not been created, it will be created first.
*
* @throws ExecutionException if this future completed exceptionally
* @throws InterruptedException if the current thread was interrupted while waiting
*/
private void startClient() throws ExecutionException, InterruptedException {
if (client == null) {
createClient();
} else {
if (client.getIsConnected()) {
throw new StartMqtt5ClientFailedException("Client is already connected");
}
}
log.info("Start");
client.start();
}
subscribing:
private void subscribeToConnectorEvents() throws ExecutionException, InterruptedException, TimeoutException {
QOS qos = QOS.AT_LEAST_ONCE;
log.debug("subscribing to topic [{}] with QOS [{}]", eventTopic, qos);
SubAckPacket subAckPacket = client.subscribe(new SubscribePacket.SubscribePacketBuilder()
.withSubscription(eventTopic,
qos,
true,
false,
SubscribePacket.RetainHandlingType.SEND_ON_SUBSCRIBE_IF_NEW) // only for retained messages
.build()).get(subscribeTimeoutSeconds, TimeUnit.SECONDS);
log.info("subscribed to topic [{}] with reason codes {}: {}",
eventTopic, subAckPacket.getReasonCodes(), subAckPacket.getReasonString());
if (!subAckPacket.getReasonCodes().contains(SubAckPacket.SubAckReasonCode.GRANTED_QOS_1)) {
throw new SubscriptionDoesNotMatchRequirementsException(String.format(
"QOS [%s] was not granted although QOS [%s] was requested",
SubAckPacket.SubAckReasonCode.GRANTED_QOS_1, qos));
}
if (hasSubscriptionError(subAckPacket)) {
throw new SubscriptionFailedException(String.format(
"Subscription to topic [%s] failed with reason codes [%s]: %s",
eventTopic, subAckPacket.getReasonCodes(), subAckPacket.getReasonString()));
}
}
shutdown:
public void onShutdown() {
if (!enabled) {
return;
}
stopAndCloseClient();
}
private void stopAndCloseClient() {
try {
stopClient();
} finally {
closeClient();
}
}
private void closeClient() {
log.debug("close");
// Important: When finished with an MQTT5 client, you must call close() on it or any associated
// native resource may leak
client.close();
//CrtResource.waitForNoResources(); // NEEDS REVIEW
//log.info("resources cleaned up");
client = null;
}
private void stopClient() {
try {
stop();
} catch (ExecutionException e) {
throw new DisconnectOnShutdownFailedException(e);
} catch (InterruptedException e) {
Thread.currentThread().interrupt(); // re-interrupt the thread, no need to fail the shutdown
}
}
public void onContextClosedEvent(ContextClosedEvent event) {
log.debug("onApplicationEvent: {}", event);
// Important: When finished with an MQTT5 client, you must call close() on it or any associated
// native resource may leak
onShutdown();
}
startup:
public void onContextRefreshedEvent(ContextRefreshedEvent event) {
log.debug("onApplicationEvent: {}", event);
onStartup();
}
private void onStartup() {
if (!enabled) {
log.warn("IoT Core is disabled");
return;
}
try {
startClient();
} catch (InterruptedException ex) {
Thread.currentThread().interrupt(); // re-interrupt the thread
throw new StartMqtt5ClientFailedException(ex); // and fail the startup
} catch (CrtRuntimeException | ExecutionException ex) {
throw new StartMqtt5ClientFailedException(ex);
}
// REVIEW wait for the initial connection to be established before subscribing
// Once connected, the MQTT5 client can subscribe to one or more topics
// make sure not to do this in the connection success callback because it will
// block the MQTT event loop, and you will run into a timeout
try {
subscribeToConnectorEvents();
} catch (ExecutionException | InterruptedException | TimeoutException e) {
if (e instanceof InterruptedException) {
Thread.currentThread().interrupt(); // re-interrupt the thread
}
throw new ConnectAndSubscribeTopicsFailedException("Failed to subscribe", e);
}
}
So I think these are all relevant code snippets.
Expected Behavior
to not see SESSION_TAKEN_OVER at all
Current Behavior
15:45:33.113 [main] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - onApplicationEvent: org.springframework.context.event.ContextRefreshedEvent[source=org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@305f031, started on Tue May 07 15:45:31 CEST 2024] 15:45:33.500 [main] INFO c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - Start 15:45:33.500 [main] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - subscribing to topic [connector/connector-608/event] with QOS [AT_LEAST_ONCE] 15:45:33.500 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - attempting connect 15:45:33.889 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - connection success with clientID [connector-608] and rejoinedSession [true] and sessionExpiryIntervalSeconds [3600] and serverKeepAliveSeconds [60] : assignedClientIdentifier [null], reasonCode [SUCCESS], reasonString [null], responseInformation [null], sessionExpiryIntervalSeconds [3600], sessionPresent [true], serverKeepAliveSeconds [60], userProperties [] 15:45:33.960 [main] INFO c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - subscribed to topic [connector/connector-608/event] with reason codes [GRANTED_QOS_1]: null 15:45:33.965 [main] INFO c.m.c.s.ConnectorClientApplication - Started ConnectorClientApplication in 2.44 seconds (process running for 2.796) 15:45:34.167 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - disconnection with error code [5154] 15:45:34.167 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - disconnection with reason code [SESSION_TAKEN_OVER] and sessionExpiryIntervalSeconds [null]: null 15:47:01.925 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - attempting connect 15:47:02.426 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - connection success with clientID [connector-608] and rejoinedSession [true] and sessionExpiryIntervalSeconds [3600] and serverKeepAliveSeconds [60] : assignedClientIdentifier [null], reasonCode [SUCCESS], reasonString [null], responseInformation [null], sessionExpiryIntervalSeconds [3600], sessionPresent [true], serverKeepAliveSeconds [60], userProperties [] 15:47:03.127 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - disconnection with error code [5154] 15:47:03.127 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - disconnection with reason code [SESSION_TAKEN_OVER] and sessionExpiryIntervalSeconds [null]: null 15:47:29.742 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - attempting connect 15:47:30.053 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - connection success with clientID [connector-608] and rejoinedSession [true] and sessionExpiryIntervalSeconds [3600] and serverKeepAliveSeconds [60] : assignedClientIdentifier [null], reasonCode [SUCCESS], reasonString [null], responseInformation [null], sessionExpiryIntervalSeconds [3600], sessionPresent [true], serverKeepAliveSeconds [60], userProperties [] 15:47:31.157 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - disconnection with error code [5154] 15:47:31.157 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - disconnection with reason code [SESSION_TAKEN_OVER] and sessionExpiryIntervalSeconds [null]: null 15:49:31.158 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - attempting connect 15:49:31.471 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - connection success with clientID [connector-608] and rejoinedSession [true] and sessionExpiryIntervalSeconds [3600] and serverKeepAliveSeconds [60] : assignedClientIdentifier [null], reasonCode [SUCCESS], reasonString [null], responseInformation [null], sessionExpiryIntervalSeconds [3600], sessionPresent [true], serverKeepAliveSeconds [60], userProperties [] 15:49:32.455 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - disconnection with error code [5154] 15:49:32.455 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - disconnection with reason code [SESSION_TAKEN_OVER] and sessionExpiryIntervalSeconds [null]: null 15:50:55.170 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - attempting connect 15:50:55.534 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - connection success with clientID [connector-608] and rejoinedSession [true] and sessionExpiryIntervalSeconds [3600] and serverKeepAliveSeconds [60] : assignedClientIdentifier [null], reasonCode [SUCCESS], reasonString [null], responseInformation [null], sessionExpiryIntervalSeconds [3600], sessionPresent [true], serverKeepAliveSeconds [60], userProperties [] 15:50:56.696 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - disconnection with error code [5154] 15:50:56.696 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - disconnection with reason code [SESSION_TAKEN_OVER] and sessionExpiryIntervalSeconds [null]: null 15:51:08.092 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - attempting connect 15:51:08.392 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - connection success with clientID [connector-608] and rejoinedSession [true] and sessionExpiryIntervalSeconds [3600] and serverKeepAliveSeconds [60] : assignedClientIdentifier [null], reasonCode [SUCCESS], reasonString [null], responseInformation [null], sessionExpiryIntervalSeconds [3600], sessionPresent [true], serverKeepAliveSeconds [60], userProperties [] 15:51:08.939 [AwsEventLoop 2] DEBUG c.m.c.s.i.m.IoTCoreMqtt5ConnectionHandler - disconnection with error code [5154]
Reproduction Steps
Relevant source files were provided to AWS support. This issue will also be linked there.
Possible Solution
I suspect a resource leak.
Additional Information/Context
No response
SDK version used
1.20.1
Environment details (OS name and version, etc.)
Mac 14.3.1
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 24 (6 by maintainers)
The SESSION_TAKEN_OVER error code is sent in a disconnect packet from the server when the server gets a connect packet containing the same client id as a client that is currently connected to it. The server will accept the connection from the new client and disconnect the previously connected client.
You probably have two clients running concurrently using the same client id and they are taking turns disconnecting each other. To test whether this is the issue, try changing your createClient() function to use a UUID to generate a client id instead of using thing name at
.withClientId(thingName). If this resolves the SESSION_TAKEN_OVER issue, you can then look into your code to figure out how and why it’s creating more than one client.You can also setup and check Cloudwatch logs to see the incoming CONNECT packets to your MQTT endpoint. If you see more than one CONNECT packet with two different UUID client ids, that will also confirm that something is creating multiple clients and starting them up.