iotedge: Memory Leak in edgeHub module

Expected Behavior

The edgeHub module should be able to run for a long time and receive tons of messages without increasing memory consumption.

Current Behavior

Memory consumption of the edgeHub module increases constantly if messages are handled. Before Handling messages the memory consumption of edgeHub is about 115 MB. After 10.000 messages generated by tempSensorModule and passed through a SampleModule just piping the messages to IotHub, the memory consumption of edgeHub is about 150 MB. It does not decrease again after stopping sending messages but stays the same. Memory consumption of tempSensor and SampleModule increase just a little bit temporarily but will always decrease again.

The increasing Memory consumption occurs on both a Raspberry Pi and an ARM64 Virtual Machine on Azure. On the Raspberry Pi the edgeHub crashes at some point with std::badAlloc when the address space is exceeded.

Steps to Reproduce

Create new IotEdge Solution in Visual Studio Code using the C# Module template
Set MessageCount of tempSensor to 10000 and MessageDelay to “00:00:00.010”
Deploy to Device

Context (Environment)

Device (Host) Operating System

Architecture

Container Operating System

Runtime Versions

iotedged

Edge Agent

Edge Hub

Docker

Logs

Additional Information

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 15 (6 by maintainers)

Most upvoted comments

Hello @darobs

The “OptimizeForPerformance” setting is set to false (on both the Raspberry and the VirtualMachine). We also use a custom storage path for messages. Here is the deployment setting for the sample application for reproduction of the issue:

{
  "modulesContent": {
    "$edgeAgent": {
      "properties.desired": {
        "schemaVersion": "1.0",
        "runtime": {
          "type": "docker",
          "settings": {
            "minDockerVersion": "v1.25",
            "loggingOptions": "",
            "registryCredentials": {
              // removed from posting
              }
            }
          }
        },
        "systemModules": {
          "edgeAgent": {
            "type": "docker",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-agent:1.0",
              "createOptions": "{}"
            }
          },
          "edgeHub": {
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-hub:1.0",
              "createOptions": "{\"HostConfig\":{\"Binds\":[\"/etc/iotedge/storage/:/iotedge/storage/\"],\"PortBindings\":{\"5671/tcp\":[{\"HostPort\":\"5671\"}],\"8883/tcp\":[{\"HostPort\":\"8883\"}],\"443/tcp\":[{\"HostPort\":\"443\"}]}}}"
            },
            "env": {
              "OptimizeForPerformance": {
                "value": "false"
              },
              "storageFolder": {
                "value": "/iotedge/storage/"
              }
            }
          }
        },
        "modules": {
          "tempSensor": {
            "version": "1.0",
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-simulated-temperature-sensor:1.0",
              "createOptions": "{}"
            },
            "env": {
              "MessageCount": {
                "value": 10000
              },
              "MessageDelay": {
                "value": "00:00:00.010"
              }
            }
          },
          "SampleModule": {
            "version": "1.0",
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "bewaconnect.azurecr.io/samplemodule:0.0.1-amd64",
              "createOptions": "{}"
            }
          }
        }
      }
    },
    "$edgeHub": {
      "properties.desired": {
        "schemaVersion": "1.0",
        "routes": {
          "SampleModuleToIoTHub": "FROM /messages/modules/SampleModule/outputs/* INTO $upstream",
          "sensorToSampleModule": "FROM /messages/modules/tempSensor/outputs/temperatureOutput INTO BrokeredEndpoint(\"/modules/SampleModule/inputs/input1\")"
        },
        "storeAndForwardConfiguration": {
          "timeToLiveSecs": 7200
        }
      }
    }
  }
}

We are reaching a point where we have to question whether IotEdge can really be used in real life industrial use-cases. Our machines will be sending messages every 2 seconds and they have to be able to run 24/7 for long periods of time without worrying about loosing data. I’m under time pressure to get the messages to IotHub reliably so it would be great to get an estimation soon, if there is a way to use iotedge in real-world industry scenarios and if yes, which hardware and configuration has to be used to make it work.

I just found a report from March 18 stating there probably is a memory leak: https://github.com/Azure/iot-edge-v1/issues/537

Please let me know as soon as possible if there really is an issue and if it can be fixed soon. If it can’t I’ll have to look for an alternative. I’m still hoping that there is a configuration that makes it all work!

Thanks, René

BewaControl-ReneDivossen on Feb 8, 2019

@levi106 - The leaking of event handlers has been fixed in master - https://github.com/Azure/iotedge/commit/4f052756228bf84ac663e4f6171a0a47aa2025ae#diff-98a736262e0091d4cc8c5192d8ce48a8 It will be part of the 1.0.9 release.

varunpuranik on Oct 7, 2019