thingsboard: [Bug] CoAP observation drops

Hi @jairohg , I will provide more comments about this issue tomorrow morning. This is indeed related to NAT but not only to NAT. It is also about the routing tables on many load balancers. It is a long story but we have a solution. Stay tuned for updates.

Hi @WillNilges , my 2 cents about our “coap.thingsboard.cloud” setup: at the moment the LB is installed on AWS Ubuntu VMs with elastic IPs. It forwards the traffic to LwM2M pods using Node Port. The LB is “remembering” the routing table, which consists of: A) Source IP and port of the device. B) Dest IP and port of the Node; The LB is configured to remember the sessions for 1 hour. So, when the Node has an update, we make sure we push it from the correct LB IP and Port, and not from the AWS NAT Gateway.

Before the LB, we were still publishing the update from the node, but it was sent from the wrong IP (not from the LB IP, which received the packet, but from the AWS NAT Gateway). The client was ignoring the update since the IP of the originator of the CoAP/UDP packet was different.

The ThingsBoard UDP LB is only 8 days old. Is it ready for production use?

We have faced a similar issue during testing of some NB IoT devices which communicate over CoAP and LwM2M. Even if the device is not behind the NAT, the issue with the Load Balancer implementation is present.

AWS NLB During our experiments, we have found that the default AWS NLB implementation “remembers” the route between the client ip+port and the k8s instance for 2 minutes. After ~2-3 minutes of inactivity, whatever downlink we send to the client, is not delivered. We have not found any configuration property for the lifetime of the session that works for our case. In an attempt to solve the problem, we decided to try Nginx LB which has many configuration options.

Nginx LB We have allocated Nginx LB with the static IP address and forwarded traffic to our K8S services in the same VPC that are exposed as NodePorts. This solution allows you to configure the “session” expiration time or route table to 24 and more hours. This works well until your target servers (ThingsBoard CoAP Transport) are restarted due to an upgrade or outage. In such a case the Load Balancer should forward the traffic for existing “sessions” to the new server. Unfortunately, this was not the case. The traffic was forwarded for new but not old clients. Old clients traffic was forwarded to the wrong server and was never delivered. Theoretically, it is possible to configure Nginx to close the session using proxy_responses setting and close the connections using proxy_session_drop which is part of commercial subscription btw. But for our use case, these settings are not perfect for multiple reasons:

1. Clients may want to use non-confirmable packets for telemetry to optimize traffic consumption. In such a case, the proxy_responses setting does not work for us. The Load Balancer simply does not know should it close the session or not.
2. The amount of traffic should not increase due to server failures or restarts. That is why the only possible solution we found is to create our own UDP Load Balancer. It may be not as fast as Nginx or competitors, but the amount of traffic from the NB IoT devices is relatively low. We just need more control over “session” management.

ThingsBoard UDP LB Simple LB implementation which is open-source and powered by great frameworks. It has certain limitations but works best for our use case. A single server may handle up to 50K concurrent UDP sessions (since each session occupies one client port on the load balancer). The server reacts to the DNS address changes and updates the routing table and the data is forwarded to a new server(s). LwM2M transport remembers all active sessions so the clients are not affected. What we plan to do next:

1. Cache the routing table in Redis or local files. So the restart of the load balancer will not affect our setup.
2. Cache the subscriptions info on the ThingsBoard CoAP Transport. Same as for LwM2M. To remember the RPC/Attirubte subscriptions for the client. This way we optimize the traffic and the devices may be super constrained without any complex logic about reconnects/etc.