thingsboard-gateway: Error: Too many publishes in progress

When the tb-gateway (GW) loses the connection with the server for a period of time such that (I suspect) the number of messages not transmitted are greater than the “maxInFlight” parameter, then the connection is re-established the GW generate this error in the log:

2017-05-10 00:11:07,311 [MQTT Call: 452a5e28-9587-47bd-a61f-6d03124f6b21] WARN  o.t.g.service.MqttGatewayService - [Sensor_Verde][1370] Failed to publish device telemetry!
org.eclipse.paho.client.mqttv3.MqttException: Too many publishes in progress
	at org.eclipse.paho.client.mqttv3.internal.ClientState.send(ClientState.java:498)
	at org.eclipse.paho.client.mqttv3.internal.ClientComms.internalSend(ClientComms.java:136)
	at org.eclipse.paho.client.mqttv3.internal.ClientComms.sendNoWait(ClientComms.java:165)
	at org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:1126)
	at org.thingsboard.gateway.service.MqttGatewayService.publishAsync(MqttGatewayService.java:334)
	at org.thingsboard.gateway.service.MqttGatewayService.onDeviceTelemetry(MqttGatewayService.java:198)
	at org.thingsboard.gateway.extensions.mqtt.client.MqttBrokerMonitor.onDeviceData(MqttBrokerMonitor.java:181)
	at org.thingsboard.gateway.extensions.mqtt.client.listener.MqttTelemetryMessageListener.messageArrived(MqttTelemetryMessageListener.java:42)
	at org.eclipse.paho.client.mqttv3.internal.CommsCallback.deliverMessage(CommsCallback.java:469)
	at org.eclipse.paho.client.mqttv3.internal.CommsCallback.handleMessage(CommsCallback.java:380)
	at org.eclipse.paho.client.mqttv3.internal.CommsCallback.run(CommsCallback.java:184)
	at java.lang.Thread.run(Thread.java:745)

and in the server-side user interface Device telemetry data does not arrive.

Restart the tb-gateway service does not fix anything. The only way I found to fix the problem is to delete the GW and create it again in the Thingsboard console user interface, which has the effect of losing all historical data.

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 21 (3 by maintainers)

Most upvoted comments

Components: TB Gateway (v1.2.1) and a TB Server (v1.3.1).

I was doing some network captures with Tcpdump and I can see that of the 3 different types of MQTT PUBLISH (to topics: “v1/devices/me/telemetry”, “v1/gateway/attributes” and “v1/gateway/telemetry”) that the gateway makes to the TB Server only those made to the topic “v1/gateway/attributes” are acknowledged (Publish Ack). This agrees that in the storage folder I can see all the messages ID (s-xx.msg) of the unacknowledged messages and all correspond to the unacknowledged topics I saw in the network capture too, and any correspond to attributes message.

So, in other words, apparently only the messages of attributes (“v1/gateway/attributes”) are being acknowledged from the TB Server, but not those of telemetry (“v1/devices/me/telemetry” and “v1/gateway/telemetry”) and are those that end up filling the storage.

nmenoni on Oct 13, 2017