thingsboard-gateway: [BUG] Not all telemetry is sent to Thingsboard

Describe the bug We configured a tb-gateway on a raspberry to convert telemetries registered by shelly-em and send them to thingsboard (comunity edition, on premise).

Shelly EM reports data on various MQTT topics and we want to register a devices on TB for each measured dimension (eg: energy, power, voltage, …).

We created a custom converter that transform data based on topic

def convert(self, topic, body):
        try:
            matched = re.search(
                r'shellies/(shellyem-\w*?)/(relay|emeter)/(\d)/?(\w*)?', topic)
            if matched:
                name_with_channel = matched.group(1) + '-ch' + matched.group(3)
                self.dict_result["deviceType"] = "Shelly EM"
                self.dict_result["telemetry"] = []
                if matched.group(2) == 'emeter':
                    key = matched.group(4)
                else:
                    key = 'state'
                self.dict_result["deviceName"] = name_with_channel + "_" + key
                em_status = {key: body}
                self.dict_result["telemetry"].append(em_status)
                
            log.info('send %s [%s] telemetry for %s value %s',
                     self.dict_result["deviceName"], self.dict_result["deviceType"], key, body)
            return self.dict_result
        except Exception as e:
            log.exception('Error in converter message: \n%s\n', body)
            log.exception(e)

Everythings seems work fine because in the logs we see messages for each data received on various topics.

The devices have been registered on thingsboard but since shelly em send a message every 60 seconds on each topic I expect to see all the recorded telemetry but there is no device that has all the data saved. There are many jumps between one telemetry and another and some devices don’t even have one.

I think that the problem is in the configuration of the local storage (we use an in memory), but I can’t understand which could be the correct configuration. This is the current configuration:

storage:
  type: memory
  read_records_count: 100
  max_records_count: 100000

The only errors I saw in log are these

|ERROR| - [tb_gateway_service.py] - tb_gateway_service - __send_to_storage - 574 - 'str' object has no attribute 'get'"

|ERROR| - [tb_utility.py] - tb_utility - validate_converted_data - 67 - deviceName is empty in data: {"telemetry": [], "attributes": []}"
|ERROR| - [tb_gateway_service.py] - tb_gateway_service - __send_to_storage - 518 - Data from MQTT Broker Connector connector is invalid."

But I think they are irrelevant because appeare for each message that the converter translate (the first one disappear if I set the timestamp inside the converter, but telemetries are still loss).

I set the log level on DEBUG for each component, but no message clarify me where is the problem.

Thanks for support.

Connector name (If bug in the some connector): MQTT Connector

Versions (please complete the following information):

• OS: Raspbian GNU/Linux 11
• Thingsboard IoT Gateway version 3.0.1
• Python version 3.9.2