iotedge: OperationTimeout when sending messages from downstream device after updating IoT Edge images to 1.3.0

Expected Behavior

Sending messages from downstream devices to Edge using x509-certificate should not time out.

Current Behavior

After upgrading the IoT Edge runtime to 1.3.0 and edgeAgent/edgeHub module to 1.3.0 messages from downstream devices is only working randomly. Most of the time we get a timeout issue. Could not complete operation before timeout

The aziot-edged, edgeHub and edgeAgent logs are not showing any errors.

A workaround is to downgrade the edgeHub and edgeAgent images to 1.2.6. The az-iot runtime is still 1.3.0.

Test code

Based on code samples from: azure-iot-sdk-python

import os
import uuid
from azure.iot.device.aio import IoTHubDeviceClient
from azure.iot.device import Message, X509
from azure.iot.device.aio import ProvisioningDeviceClient
import asyncio
import threading
import time
import logging
import random
import json
import datetime
from asyncio import Queue, sleep
import socket

async def doTest():
 
    hostName=socket.gethostname()
    deviceId=f"{hostName}-controlsystem"

    x509 = X509(
        cert_file=f"/var/secrets/aziot/iot-device-{deviceId}-full-chain.cert.pem",
        key_file=f"/var/secrets/aziot/iot-device-{deviceId}.key.pem",
        pass_phrase="12345",
    )
    
    # # The client object is used to interact with your Azure IoT hub.
    device_client = IoTHubDeviceClient.create_from_x509_certificate(
        hostname=hostName, device_id=deviceId, x509=x509)
    print("Starting test...")
    
    messagecount = 1

    # # Connect the client.
    await device_client.connect()
    async def send_test_message(i):
        print("sending message #" + str(i))
        telemetryMessage = {
            "SensorName": "speed",
            "SensorValue": float(random.randint(0, 50)),
            "TimeStamp": datetime.datetime.now().isoformat()
            }
        jsonStr = json.dumps(telemetryMessage)
        print(jsonStr)
        msg = Message(jsonStr)
        msg.content_encoding = "utf-8"
        msg.content_type = "application/json"

        await device_client.send_message(msg)
        #print("done sending message #" + str(i))

    # send `messages_to_send` messages in parallel
    t1= time.time()
    await asyncio.gather(*[send_test_message(i) for i in range(1, int(messagecount) + 1)])
    t2 = time.time()
    print(t2-t1)
    print("Done!")

    await device_client.shutdown()
    



asyncio.run(doTest())

Exception when sending messages


ReconnectStage: DisconnectEvent received while in unexpected state - DISCONNECTED, Connected: False
Exception caught in background thread.  Unable to handle.
['azure.iot.device.common.transport_exceptions.ConnectionDroppedError: Unexpected disconnection\n']
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.9/site-packages/azure/iot/device/iothub/aio/async_clients.py", line 33, in handle_result
    return await callback.completion()
  File "/home/user/.local/lib/python3.9/site-packages/azure/iot/device/common/async_adapter.py", line 91, in completion
    return await self.future
azure.iot.device.common.pipeline.pipeline_exceptions.OperationTimeout: Transport timeout on connection operation

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/edge-install/iot-leaf-device-test-message.py", line 71, in <module>
    asyncio.run(doTest())
  File "/usr/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/home/user/edge-install/iot-leaf-device-test-message.py", line 41, in doTest
    await device_client.connect()
  File "/home/user/.local/lib/python3.9/site-packages/azure/iot/device/aio/patch_documentation.py", line 14, in connect
    return await super(IoTHubDeviceClient, self).connect()
  File "/home/user/.local/lib/python3.9/site-packages/azure/iot/device/iothub/aio/async_clients.py", line 231, in connect
    await handle_result(callback)
  File "/home/user/.local/lib/python3.9/site-packages/azure/iot/device/iothub/aio/async_clients.py", line 55, in handle_result
    raise exceptions.OperationTimeout("Could not complete operation before timeout") from e
azure.iot.device.exceptions.OperationTimeout: Could not complete operation before timeout`

Steps to Reproduce

Provide a detailed set of steps to reproduce the bug.

Install Azure IoT Edge 1.3.0 with X.509 certificate
Set up a IoT child device in IoT hub
Deploy edgeAgent=1.2.6 and egdeHub=1.2.6 docker images
Verify that sending messages from the downstream device works using az-iot-device sdk for Python
Deploy edgeAgent=1.3.0 and egdeHub=1.3.0 docker images
Send messages again
Exception azure.iot.device.exceptions.OperationTimeout: Could not complete operation before timeout is thrown

Context (Environment)

Output of `iotedge check`

Click here


Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK
√ aziot-identity-service package is up-to-date - OK
√ host time is close to reference time - OK
√ production readiness: identity certificates expiry - OK
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ identityd is running - OK
√ read all preloaded certificates from the Certificates Service - OK
√ read all preloaded key pairs from the Keys Service - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)
--------------------------------------------
‼ host can connect to and perform TLS handshake with iothub AMQP port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    Since no hostname is provided, all hub connectivity tests will be skipped.
‼ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    Since no hostname is provided, all hub connectivity tests will be skipped.
‼ host can connect to and perform TLS handshake with iothub MQTT port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    Since no hostname is provided, all hub connectivity tests will be skipped.
√ host can connect to and perform TLS handshake with DPS endpoint - OK

Configuration checks
--------------------
√ aziot-edged configuration is well-formed - OK
√ configuration up-to-date with config.toml - OK
√ container engine is installed and functional - OK
√ configuration has correct URIs for daemon mgmt endpoint - OK
√ aziot-edge package is up-to-date - OK
√ container time is close to host time - OK
√ DNS server - OK
√ production readiness: logs policy - OK
√ production readiness: Edge Agent's storage directory is persisted on the host filesystem - OK
√ production readiness: Edge Hub's storage directory is persisted on the host filesystem - OK
√ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK

Connectivity checks
-------------------
28 check(s) succeeded.
3 check(s) raised warnings. Re-run with --verbose for more details.
7 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details.

Device Information

Host OS - Raspbian GNU/Linux 11 (bullseye) armv7l
Architecture - arm32:
Container OS - Linux containers

Runtime Versions

aziot-edged [run iotedge version]: 1.3.0
Edge Agent [image tag (e.g. 1.0.0)]: 1.3.0
Edge Hub [image tag (e.g. 1.0.0)]: 1.3.0
Docker/Moby [run docker version]: 20.10.17+azure-1

Note: when using Windows containers on Windows, run docker -H npipe:////./pipe/iotedge_moby_engine version instead

Logs

aziot-edged logs


-- Journal begins at Mon 2022-04-04 14:05:58 CEST, ends at Tue 2022-08-16 12:49:56 CEST. --
Aug 16 12:46:59 ot-06 aziot-edged[10522]: 2022-08-16T10:46:59Z [INFO] - <-- GET /modules?api-version=2021-12-07 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Aug 16 12:46:59 ot-06 aziot-edged[10522]: 2022-08-16T10:46:59Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 16 12:47:04 ot-06 aziot-edged[10522]: 2022-08-16T10:47:04Z [INFO] - <-- GET /modules?api-version=2021-12-07 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Aug 16 12:47:04 ot-06 aziot-edged[10522]: 2022-08-16T10:47:04Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 16 12:47:09 ot-06 aziot-edged[10522]: 2022-08-16T10:47:09Z [INFO] - <-- GET /modules?api-version=2021-12-07 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Aug 16 12:47:09 ot-06 aziot-edged[10522]: 2022-08-16T10:47:09Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 16 12:47:14 ot-06 aziot-edged[10522]: 2022-08-16T10:47:14Z [INFO] - <-- GET /modules?api-version=2021-12-07 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Aug 16 12:47:14 ot-06 aziot-edged[10522]: 2022-08-16T10:47:14Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 16 12:47:17 ot-06 aziot-edged[10522]: 2022-08-16T10:47:17Z [INFO] - Watchdog checking Edge runtime status
Aug 16 12:47:17 ot-06 aziot-edged[10522]: 2022-08-16T10:47:17Z [INFO] - Edge runtime is running
Aug 16 12:47:19 ot-06 aziot-edged[10522]: 2022-08-16T10:47:19Z [INFO] - <-- GET /modules?api-version=2021-12-07 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Aug 16 12:47:19 ot-06 aziot-edged[10522]: 2022-08-16T10:47:19Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 16 12:47:24 ot-06 aziot-edged[10522]: 2022-08-16T10:47:24Z [INFO] - <-- GET /modules?api-version=2021-12-07 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Aug 16 12:47:24 ot-06 aziot-edged[10522]: 2022-08-16T10:47:24Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 16 12:47:29 ot-06 aziot-edged[10522]: 2022-08-16T10:47:29Z [INFO] - <-- GET /modules?api-version=2021-12-07 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Aug 16 12:47:30 ot-06 aziot-edged[10522]: 2022-08-16T10:47:30Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 16 12:47:35 ot-06 aziot-edged[10522]: 2022-08-16T10:47:35Z [INFO] - <-- GET /modules?api-version=2021-12-07 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Aug 16 12:47:35 ot-06 aziot-edged[10522]: 2022-08-16T10:47:35Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 16 12:47:40 ot-06 aziot-edged[10522]: 2022-08-16T10:47:40Z [INFO] - <-- GET /modules?api-version=2021-12-07 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Aug 16 12:47:40 ot-06 aziot-edged[10522]: 2022-08-16T10:47:40Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 16 12:47:45 ot-06 aziot-edged[10522]: 2022-08-16T10:47:45Z [INFO] - <-- GET /modules?api-version=2021-12-07 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Aug 16 12:47:45 ot-06 aziot-edged[10522]: 2022-08-16T10:47:45Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 16 12:47:50 ot-06 aziot-edged[10522]: 2022-08-16T10:47:50Z [INFO] - <-- GET /modules?api-version=2021-12-07 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Aug 16 12:47:50 ot-06 aziot-edged[10522]: 2022-08-16T10:47:50Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 16 12:47:50 ot-06 aziot-edged[10522]: 2022-08-16T10:47:50Z [INFO] - <-- GET /systeminfo/resources?api-version=2021-12-07 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Aug 16 12:47:50 ot-06 aziot-edged[10522]: 2022-08-16T10:47:50Z [INFO] - Querying system resources...
Aug 16 12:47:55 ot-06 aziot-edged[10522]: 2022-08-16T10:47:55Z [INFO] - <-- GET /modules?api-version=2021-12-07 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Aug 16 12:48:00 ot-06 aziot-edged[10522]: 2022-08-16T10:48:00Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 16 12:48:00 ot-06 aziot-edged[10522]: 2022-08-16T10:48:00Z [INFO] - --> 200 {"content-type": "application/json"}
Aug 16 12:48:05 ot-06 aziot-edged[10522]: 2022-08-16T10:48:05Z [INFO] - <-- GET /modules?api-version=2021-12-07 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Aug 16 12:48:05 ot-06 aziot-edged[10522]: 2022-08-16T10:48:05Z [INFO] - --> 200 {"content-type": "application/json"}

edge-agent logs


2022-08-16 10:31:18  Starting Edge Agent
2022-08-16 10:31:18  Changing ownership of storage folder: /iotedge/storage//edgeAgent to 13622
2022-08-16 10:31:18  Changing ownership of management socket: /var/run/iotedge/mgmt.sock
2022-08-16 10:31:18  Completed necessary setup. Starting Edge Agent.
2022-08-16 10:31:18.910 +00:00 Edge Agent Main()
<6> 2022-08-16 10:31:19.914 +00:00 [INF] - Initializing Edge Agent.
<6> 2022-08-16 10:31:20.903 +00:00 [INF] - Version - 1.3.0.57041647 (b022069058d21deb30c7760c4e384b637694f464)
<6> 2022-08-16 10:31:20.906 +00:00 [INF] -
        █████╗ ███████╗██╗   ██╗██████╗ ███████╗
       ██╔══██╗╚══███╔╝██║   ██║██╔══██╗██╔════╝
       ███████║  ███╔╝ ██║   ██║██████╔╝█████╗
       ██╔══██║ ███╔╝  ██║   ██║██╔═�╗██╔══╝
       ██║  ██║███████╗╚██████╔╝██║  ██║███████╗
       ╚═╝  ╚═╝╚══════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝

 ██╗ ██████╗ ████████╗    ███████╗██████╗  ██████╗ ███████╗
 ██║██╔═══██╗╚══██╔══╝    ██╔════╝██╔══██╗██╔════╝ ██╔════╝
 ██║██║   ██║   ██║       █████╗  ██║  ██║██║  ███╗█████╗
 ██║██║   ██║   ██║       ██╔══╝  ██║  ██║██║   ██║██╔══╝
 ██║╚██████╔╝   ██║       ███████╗██████╔╝╚██████╔╝███████╗
 ╚═╝ ╚═════╝    ╚═╝       ╚══════╝╚═════╝  ╚═════╝ ╚══════╝

<6> 2022-08-16 10:31:21.211 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false}
<6> 2022-08-16 10:31:21.939 +00:00 [INF] - Installing certificates [CN=ot-06-ca.ca:08/09/2023 13:43:08],[CN=Azure_IoT_Hub_Intermediate_Cert_Test_Only:06/03/2023 10:24:18],[CN=Azure_IoT_Hub_Evoy_dev_arm32_CA_Cert:06/03/2023 10:24:17] to Root
<6> 2022-08-16 10:31:22.930 +00:00 [INF] - Starting metrics listener on Host: *, Port: 9600, Suffix: metrics
<6> 2022-08-16 10:31:23.052 +00:00 [INF] - Updating performance metrics every 05m:00s
<6> 2022-08-16 10:31:23.098 +00:00 [INF] - Started operation Get system resources
<6> 2022-08-16 10:31:23.111 +00:00 [INF] - Collecting metadata metrics
<6> 2022-08-16 10:31:23.566 +00:00 [INF] - Set metadata metrics: 1.3.0.57041647 (b022069058d21deb30c7760c4e384b637694f464), {"Enabled":false,"DisableCloudSubscriptions":false}, {"OperatingSystemType":"Linux","Architecture":"arm","Version":"1.3.0","Provisioning":{"Type":"ProvisioningType","DynamicReprovisioning":false,"AlwaysReprovisionOnStartup":false},"ServerVersion":null,"KernelVersion":"#1538 SMP Thu Mar 31 19:38:48 BST 2022","OperatingSystem":"raspbian","NumCpus":4,"Virtualized":"no"}, True
<6> 2022-08-16 10:31:23.686 +00:00 [INF] - Started operation Checkpoint Availability
<6> 2022-08-16 10:31:23.706 +00:00 [INF] - Started operation refresh twin config
<6> 2022-08-16 10:31:23.826 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...
<6> 2022-08-16 10:31:25.447 +00:00 [INF] - Created persistent store at /iotedge/storage/edgeAgent
<6> 2022-08-16 10:31:25.851 +00:00 [INF] - Started operation Metrics Scrape
<6> 2022-08-16 10:31:25.852 +00:00 [INF] - Started operation Metrics Upload
Scraping frequency: 01:00:00
Upload Frequency: 1.00:00:00
<6> 2022-08-16 10:31:27.416 +00:00 [INF] - Registering request handler UploadModuleLogs
<6> 2022-08-16 10:31:27.417 +00:00 [INF] - Registering request handler GetModuleLogs
<6> 2022-08-16 10:31:27.418 +00:00 [INF] - Registering request handler UploadSupportBundle
<6> 2022-08-16 10:31:27.419 +00:00 [INF] - Registering request handler RestartModule
<6> 2022-08-16 10:31:28.294 +00:00 [INF] - Edge agent connected to IoT Hub via Amqp_Tcp_Only.
<6> 2022-08-16 10:31:28.864 +00:00 [INF] - Initialized new module client with subscriptions enabled
<6> 2022-08-16 10:31:29.109 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 14 and reported properties version 217.
<6> 2022-08-16 10:31:30.502 +00:00 [INF] - Plan execution started for deployment 14
<6> 2022-08-16 10:31:30.517 +00:00 [INF] - Executing command: "Command Group: (\n  [Stop module SensorRelay]\n  [Start module SensorRelay]\n  [Saving SensorRelay to store]\n)"
<6> 2022-08-16 10:31:30.544 +00:00 [INF] - Executing command: "Stop module SensorRelay"
<6> 2022-08-16 10:31:30.641 +00:00 [INF] - Executing command: "Start module SensorRelay"
<6> 2022-08-16 10:31:32.393 +00:00 [INF] - Executing command: "Saving SensorRelay to store"
<6> 2022-08-16 10:31:32.442 +00:00 [INF] - Executing command: "Command Group: (\n  [Stop module SensorHandler]\n  [Start module SensorHandler]\n  [Saving SensorHandler to store]\n)"
<6> 2022-08-16 10:31:32.443 +00:00 [INF] - Executing command: "Stop module SensorHandler"
<6> 2022-08-16 10:31:32.460 +00:00 [INF] - Executing command: "Start module SensorHandler"
<6> 2022-08-16 10:31:34.081 +00:00 [INF] - Executing command: "Saving SensorHandler to store"
<6> 2022-08-16 10:31:34.083 +00:00 [INF] - Executing command: "Command Group: (\n  [Stop module CustomMetrics]\n  [Start module CustomMetrics]\n  [Saving CustomMetrics to store]\n)"
<6> 2022-08-16 10:31:34.083 +00:00 [INF] - Executing command: "Stop module CustomMetrics"
<6> 2022-08-16 10:31:34.091 +00:00 [INF] - Executing command: "Start module CustomMetrics"
<6> 2022-08-16 10:31:36.443 +00:00 [INF] - Executing command: "Saving CustomMetrics to store"
<6> 2022-08-16 10:31:36.444 +00:00 [INF] - Executing command: "Command Group: (\n  [Stop module edgeHub]\n  [Start module edgeHub]\n  [Saving edgeHub to store]\n)"
<6> 2022-08-16 10:31:36.444 +00:00 [INF] - Executing command: "Stop module edgeHub"
<6> 2022-08-16 10:31:36.452 +00:00 [INF] - Executing command: "Start module edgeHub"
<6> 2022-08-16 10:31:39.736 +00:00 [INF] - Executing command: "Saving edgeHub to store"
<6> 2022-08-16 10:31:39.739 +00:00 [INF] - Plan execution ended for deployment 14
<6> 2022-08-16 10:31:40.691 +00:00 [INF] - Updated reported properties
<6> 2022-08-16 10:31:46.029 +00:00 [INF] - Updated reported properties
<6> 2022-08-16 10:41:35.154 +00:00 [INF] - HealthRestartPlanner is clearing restart stats for module 'SensorRelay' as it has been running healthy for 00:10:00.
<6> 2022-08-16 10:41:35.157 +00:00 [INF] - HealthRestartPlanner is clearing restart stats for module 'SensorHandler' as it has been running healthy for 00:10:00.
<6> 2022-08-16 10:41:35.159 +00:00 [INF] - Plan execution started for deployment 14
<6> 2022-08-16 10:41:35.162 +00:00 [INF] - Executing command: "Saving SensorRelay to store"
<6> 2022-08-16 10:41:35.187 +00:00 [INF] - Executing command: "Saving SensorHandler to store"
<6> 2022-08-16 10:41:35.190 +00:00 [INF] - Plan execution ended for deployment 14
<6> 2022-08-16 10:41:40.336 +00:00 [INF] - HealthRestartPlanner is clearing restart stats for module 'CustomMetrics' as it has been running healthy for 00:10:00.
<6> 2022-08-16 10:41:40.338 +00:00 [INF] - HealthRestartPlanner is clearing restart stats for module 'edgeHub' as it has been running healthy for 00:10:00.
<6> 2022-08-16 10:41:40.338 +00:00 [INF] - Plan execution started for deployment 14
<6> 2022-08-16 10:41:40.339 +00:00 [INF] - Executing command: "Saving CustomMetrics to store"
<6> 2022-08-16 10:41:40.339 +00:00 [INF] - Executing command: "Saving edgeHub to store"
<6> 2022-08-16 10:41:40.340 +00:00 [INF] - Plan execution ended for deployment 14
<6> 2022-08-16 10:41:40.627 +00:00 [INF] - Updated reported properties
<6> 2022-08-16 10:41:45.885 +00:00 [INF] - Updated reported properties

edge-hub logs


2022-08-16 10:31:39  Starting Edge Hub
2022-08-16 10:31:39  Starting Edge Hub
2022-08-16 10:31:40.116 +00:00 Edge Hub Main()
<6> 2022-08-16 10:31:45.846 +00:00 [INF] - Installing certificates [CN=ot-06-ca.ca:08/09/2023 13:43:08],[CN=Azure_IoT_Hub_Intermediate_Cert_Test_Only:06/03/2023 10:24:18],[CN=Azure_IoT_Hub_Evoy_dev_arm32_CA_Cert:06/03/2023 10:24:17] to Root
<6> 2022-08-16 10:31:46.249 +00:00 [INF] - Installing certificates [CN=ot-06-ca.ca:08/09/2023 13:43:08],[CN=Azure_IoT_Hub_Intermediate_Cert_Test_Only:06/03/2023 10:24:18],[CN=Azure_IoT_Hub_Evoy_dev_arm32_CA_Cert:06/03/2023 10:24:17] to Root
<6> 2022-08-16 10:31:46.374 +00:00 [INF] - Enabling SSL protocols: Tls12
<6> 2022-08-16 10:31:47.018 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false,"DisableConnectivityCheck":false}
<6> 2022-08-16 10:31:49.253 +00:00 [INF] - Created persistent store at /iotedge/storage/edgeHub
<6> 2022-08-16 10:31:49.778 +00:00 [INF] - Initializing Edge Hub
<6> 2022-08-16 10:31:49.779 +00:00 [INF] -
        █████╗ ███████╗██╗   ██╗██████╗ ███████╗
       ██╔══██╗╚══███╔╝██║   ██║██╔══██╗██╔════╝
       ███████║  ███╔╝ ██║   ██║██████╔╝█████╗
       ██╔══██║ ███╔╝  ██║   ██║██╔══██╗██╔══╝
       ██║  ██║███████╗╚██████╔╝██║  ██║███████╗
       ╚═╝  ╚═╝╚══════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝

 ██╗ ██████╗ ████████╗    ███████╗██████╗  ██████╗ ███████╗
 ██║██╔═══██╗╚══██╔══╝    ██╔════╝██╔══██╗██╔════╝ ██╔════╝
 ██║██║   ██║   ██║       █████╗  ██║  ██║██║  ███╗█████╗
 ██║██║   ██║   ██║       ██╔══╝  ██║  ██║██║   ██║██╔══╝
 ██║╚██████╔╝   ██║       ███████╗██████╔╝╚██████╔╝███████╗
 ╚═╝ ╚═════╝    ╚═╝       ╚══════╝╚═════╝  ╚═════╝ ╚══════╝

<6> 2022-08-16 10:31:49.785 +00:00 [INF] - Version - 1.3.0.57041647 (b022069058d21deb30c7760c4e384b637694f464)
<6> 2022-08-16 10:31:49.785 +00:00 [INF] - OptimizeForPerformance=False
<6> 2022-08-16 10:31:49.786 +00:00 [INF] - MessageAckTimeoutSecs=30
<6> 2022-08-16 10:31:49.797 +00:00 [INF] - Loaded server certificate with expiration date of "2022-09-15T10:31:44.0000000+00:00"
<6> 2022-08-16 10:31:49.846 +00:00 [INF] - Using Asp Net server for metrics
<6> 2022-08-16 10:31:50.206 +00:00 [INF] - Created new message store
<6> 2022-08-16 10:31:50.207 +00:00 [INF] - Started task to cleanup processed and stale messages
<6> 2022-08-16 10:31:50.635 +00:00 [INF] - Created DeviceConnectivityManager with connected check frequency 00:05:00 and disconnected check frequency 00:02:00
<6> 2022-08-16 10:31:50.679 +00:00 [INF] - Add node: ot-06
<6> 2022-08-16 10:31:50.694 +00:00 [INF] - Add node: ot-06-controlsystem
<6> 2022-08-16 10:31:50.696 +00:00 [INF] - Add node: ot-06/$edgeAgent
<6> 2022-08-16 10:31:50.697 +00:00 [INF] - Add node: ot-06/$edgeHub
<6> 2022-08-16 10:31:50.697 +00:00 [INF] - Add node: ot-06/CustomMetrics
<6> 2022-08-16 10:31:50.698 +00:00 [INF] - Add node: ot-06/DefenderIotMicroAgent
<6> 2022-08-16 10:31:50.698 +00:00 [INF] - Add node: ot-06/SensorHandler
<6> 2022-08-16 10:31:50.699 +00:00 [INF] - Add node: ot-06/SensorRelay
<6> 2022-08-16 10:31:50.700 +00:00 [INF] - Add node: ot-06/sensor-ingester
<6> 2022-08-16 10:31:50.719 +00:00 [INF] - Created device scope identities cache
<6> 2022-08-16 10:31:50.792 +00:00 [INF] - Starting refresh of device scope identities cache
<6> 2022-08-16 10:31:50.815 +00:00 [INF] - Initialized storing twin manager
<6> 2022-08-16 10:31:50.866 +00:00 [INF] - Updated node: ot-06/$edgeHub
<6> 2022-08-16 10:31:50.867 +00:00 [INF] - Initializing configuration
<6> 2022-08-16 10:31:50.948 +00:00 [INF] - New device connection for device ot-06/$edgeHub
<6> 2022-08-16 10:31:50.995 +00:00 [INF] - Client ot-06/$edgeHub connected to edgeHub, processing existing subscriptions.
<6> 2022-08-16 10:31:51.230 +00:00 [INF] - Attempting to connect to IoT Hub for client ot-06/$edgeHub via AMQP...
<6> 2022-08-16 10:31:52.006 +00:00 [INF] - Processing pending subscriptions for ot-06/$edgeHub
<6> 2022-08-16 10:31:53.376 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false,"DisableConnectivityCheck":false}
<6> 2022-08-16 10:31:53.582 +00:00 [INF] - Starting timer to authenticate connections with a period of 300 seconds
<6> 2022-08-16 10:31:53.810 +00:00 [INF] - Not changed node: ot-06-controlsystem
<6> 2022-08-16 10:31:53.811 +00:00 [INF] - Not changed node: ot-06
<6> 2022-08-16 10:31:53.814 +00:00 [INF] - Not changed node: ot-06/$edgeAgent
<6> 2022-08-16 10:31:53.815 +00:00 [INF] - Updated node: ot-06/$edgeHub
<6> 2022-08-16 10:31:53.922 +00:00 [INF] - Scheduling server certificate renewal for "2022-09-15T10:29:14.0005949Z".
<6> 2022-08-16 10:31:54.049 +00:00 [INF] - Service identity for ot-06/$edgeHub in device scope was updated.
<6> 2022-08-16 10:31:54.051 +00:00 [INF] - Not changed node: ot-06/SensorRelay
<6> 2022-08-16 10:31:54.051 +00:00 [INF] - Not changed node: ot-06/SensorHandler
<6> 2022-08-16 10:31:54.052 +00:00 [INF] - Not changed node: ot-06/CustomMetrics
<6> 2022-08-16 10:31:54.052 +00:00 [INF] - Not changed node: ot-06/sensor-ingester
<6> 2022-08-16 10:31:54.053 +00:00 [INF] - Not changed node: ot-06/DefenderIotMicroAgent
<6> 2022-08-16 10:31:54.070 +00:00 [INF] - Done refreshing device scope identities cache. Waiting for 60 minutes.
<6> 2022-08-16 10:31:54.821 +00:00 [INF] - Created cloud endpoint iothub with max batch size 10 and fan-out factor of 10.
<6> 2022-08-16 10:31:55.989 +00:00 [INF] - Set the following 6 route(s) in edge hub
<6> 2022-08-16 10:31:55.990 +00:00 [INF] - SensorIngesterToSensorRelay: FROM /messages/modules/sensor-ingester/outputs/* INTO BrokeredEndpoint("/modules/SensorRelay/inputs/deviceInput")
<6> 2022-08-16 10:31:55.990 +00:00 [INF] - ControlSystemToSensorRelay: FROM /messages/* WHERE ENDS_WITH($connectionDeviceId,'-controlsystem') INTO BrokeredEndpoint("/modules/SensorRelay/inputs/deviceInput")
<6> 2022-08-16 10:31:55.992 +00:00 [INF] - SensorRelayToSensorHandler: FROM /messages/modules/SensorRelay/outputs/* INTO BrokeredEndpoint("/modules/SensorHandler/inputs/input1")
<6> 2022-08-16 10:31:55.993 +00:00 [INF] - SensorRelayToIoTHub: FROM /messages/modules/SensorRelay/outputs/hubOutput INTO $upstream
<6> 2022-08-16 10:31:55.993 +00:00 [INF] - CustomMetricsToIoTHub: FROM /messages/modules/CustomMetrics/outputs/* INTO $upstream
<6> 2022-08-16 10:31:55.993 +00:00 [INF] - SensorHandlerToIoTHub: FROM /messages/modules/SensorHandler/outputs/* INTO $upstream
<6> 2022-08-16 10:31:56.001 +00:00 [INF] - Updated message store TTL to 2592000 seconds
<6> 2022-08-16 10:31:56.004 +00:00 [INF] - Updated the edge hub store and forward configuration
<6> 2022-08-16 10:31:56.241 +00:00 [INF] - Started operation Get EdgeHub config
<6> 2022-08-16 10:31:56.243 +00:00 [INF] - Initialized edge hub configuration
<6> 2022-08-16 10:31:56.256 +00:00 [INF] - Starting protocol heads - (MQTT, AMQP, HTTP)
<6> 2022-08-16 10:31:56.283 +00:00 [INF] - Starting MQTT head
<6> 2022-08-16 10:31:56.532 +00:00 [INF] - Initializing TLS endpoint on port 8883 for MQTT head.
<6> 2022-08-16 10:31:56.572 +00:00 [INF] - Exiting disconnected state
<6> 2022-08-16 10:31:56.680 +00:00 [INF] - Device connected to cloud, processing subscriptions for connected clients.
<6> 2022-08-16 10:31:56.687 +00:00 [INF] - Processing subscriptions for client ot-06/$edgeHub on device connected to cloud.
<6> 2022-08-16 10:31:56.689 +00:00 [INF] - Skipping ot-06/$edgeHub for subscription processing, as it is currently being processed.
<6> 2022-08-16 10:31:56.703 +00:00 [INF] - Received device connected callback
<6> 2022-08-16 10:31:56.714 +00:00 [INF] - Skipping cache refresh, waiting 117 seconds until refreshing again.
<6> 2022-08-16 10:31:56.768 +00:00 [INF] - Entering connected state
<6> 2022-08-16 10:31:56.788 +00:00 [INF] - Cloud connection for ot-06/$edgeHub is True
<6> 2022-08-16 10:31:56.802 +00:00 [INF] - Connection status for ot-06/$edgeHub changed to ConnectionEstablished
<6> 2022-08-16 10:31:56.808 +00:00 [INF] - Client ot-06/$edgeHub connected to cloud, processing existing subscriptions.
<6> 2022-08-16 10:31:56.809 +00:00 [INF] - Skipping ot-06/$edgeHub for subscription processing, as it is currently being processed.
<6> 2022-08-16 10:31:56.820 +00:00 [INF] - Created cloud proxy for client ot-06/$edgeHub via AMQP, with client operation timeout 20 seconds.
<6> 2022-08-16 10:31:56.835 +00:00 [INF] - Starting AMQP head
<6> 2022-08-16 10:31:56.838 +00:00 [INF] - Initialized cloud proxy 3c55ccb7-9070-4bef-9a0a-47bc0614972f for ot-06/$edgeHub
<6> 2022-08-16 10:31:56.847 +00:00 [INF] - Created cloud connection for client ot-06/$edgeHub
<6> 2022-08-16 10:31:56.863 +00:00 [INF] - Started MQTT head
<6> 2022-08-16 10:31:57.470 +00:00 [INF] - Started AMQP head
<6> 2022-08-16 10:31:57.478 +00:00 [INF] - Starting HTTP head
<6> 2022-08-16 10:31:58.494 +00:00 [INF] - Updated reported properties for ot-06/$edgeHub
<6> 2022-08-16 10:31:58.839 +00:00 [INF] - Processing pending subscriptions for ot-06/$edgeHub
<4> 2022-08-16 10:31:59.623 +00:00 [WRN] - Overriding address(es) '"http://+:80"'. Binding to endpoints defined via IConfiguration and/or UseKestrel() instead.
<6> 2022-08-16 10:31:59.678 +00:00 [INF] - Obtained edge hub config from module twin
<6> 2022-08-16 10:31:59.774 +00:00 [INF] - Started HTTP head
<6> 2022-08-16 10:31:59.849 +00:00 [INF] - Set the following 6 route(s) in edge hub
<6> 2022-08-16 10:31:59.849 +00:00 [INF] - SensorIngesterToSensorRelay: FROM /messages/modules/sensor-ingester/outputs/* INTO BrokeredEndpoint("/modules/SensorRelay/inputs/deviceInput")
<6> 2022-08-16 10:31:59.849 +00:00 [INF] - ControlSystemToSensorRelay: FROM /messages/* WHERE ENDS_WITH($connectionDeviceId,'-controlsystem') INTO BrokeredEndpoint("/modules/SensorRelay/inputs/deviceInput")
<6> 2022-08-16 10:31:59.849 +00:00 [INF] - SensorRelayToSensorHandler: FROM /messages/modules/SensorRelay/outputs/* INTO BrokeredEndpoint("/modules/SensorHandler/inputs/input1")
<6> 2022-08-16 10:31:59.849 +00:00 [INF] - SensorRelayToIoTHub: FROM /messages/modules/SensorRelay/outputs/hubOutput INTO $upstream
<6> 2022-08-16 10:31:59.850 +00:00 [INF] - CustomMetricsToIoTHub: FROM /messages/modules/CustomMetrics/outputs/* INTO $upstream
<6> 2022-08-16 10:31:59.850 +00:00 [INF] - SensorHandlerToIoTHub: FROM /messages/modules/SensorHandler/outputs/* INTO $upstream
<6> 2022-08-16 10:31:59.850 +00:00 [INF] - Updated message store TTL to 2592000 seconds
<6> 2022-08-16 10:31:59.850 +00:00 [INF] - Updated the edge hub store and forward configuration
<6> 2022-08-16 10:32:00.407 +00:00 [INF] - Client ot-06/CustomMetrics in device scope authenticated locally.
<6> 2022-08-16 10:32:00.461 +00:00 [INF] - Successfully generated identity for clientId ot-06/CustomMetrics and username ot-06/ot-06/CustomMetrics/?api-version=2020-09-30&DeviceClientType=.NET%2F1.41.0%20%28.NET%206.0.7%3B%20Linux%205.15.32-v7%2B%20%231538%20SMP%20Thu%20Mar%2031%2019%3A38%3A48%20BST%202022%3B%20Arm%29
<6> 2022-08-16 10:32:00.489 +00:00 [INF] - "ClientAuthenticated, ot-06/CustomMetrics, 2d80987b"
<6> 2022-08-16 10:32:00.662 +00:00 [INF] - New device connection for device ot-06/CustomMetrics
<6> 2022-08-16 10:32:00.666 +00:00 [INF] - Client ot-06/CustomMetrics connected to edgeHub, processing existing subscriptions.
<6> 2022-08-16 10:32:00.750 +00:00 [INF] - Bind device proxy for device ot-06/CustomMetrics
<6> 2022-08-16 10:32:00.766 +00:00 [INF] - Binding message channel for device Id ot-06/CustomMetrics
<6> 2022-08-16 10:32:00.786 +00:00 [INF] - Processing pending subscriptions for ot-06/CustomMetrics
<6> 2022-08-16 10:32:00.802 +00:00 [INF] - Attempting to connect to IoT Hub for client ot-06/CustomMetrics via AMQP...
<6> 2022-08-16 10:32:00.955 +00:00 [INF] - Processing subscriptions TwinResponse, ModuleMessages, Methods for client ot-06/CustomMetrics.
<6> 2022-08-16 10:32:01.001 +00:00 [INF] - Cloud connection for ot-06/CustomMetrics is True
<6> 2022-08-16 10:32:01.001 +00:00 [INF] - Connection status for ot-06/CustomMetrics changed to ConnectionEstablished
<6> 2022-08-16 10:32:01.001 +00:00 [INF] - Client ot-06/CustomMetrics connected to cloud, processing existing subscriptions.
<6> 2022-08-16 10:32:01.002 +00:00 [INF] - Skipping ot-06/CustomMetrics for subscription processing, as it is currently being processed.
<6> 2022-08-16 10:32:01.002 +00:00 [INF] - Created cloud proxy for client ot-06/CustomMetrics via AMQP, with client operation timeout 20 seconds.
<6> 2022-08-16 10:32:01.002 +00:00 [INF] - Initialized cloud proxy 7dc3c2fa-b288-4167-9f56-924c2b12a0c3 for ot-06/CustomMetrics
<6> 2022-08-16 10:32:01.003 +00:00 [INF] - Created cloud connection for client ot-06/CustomMetrics
<6> 2022-08-16 10:32:01.020 +00:00 [INF] - Set subscriptions from session state for ot-06/CustomMetrics
<6> 2022-08-16 10:32:01.068[INF] - Processing pending subscriptions for ot-06/CustomMetrics
<6> 2022-08-16 10:32:01.068 +00:00 [INF] - Processing pending subscriptions for ot-06/CustomMetrics
<6> 2022-08-16 10:32:01.069 +00:00 [INF] - Processing pending subscriptions for ot-06/CustomMetrics
<6> 2022-08-16 10:32:01.359 +00:00 [INF] - Processing subscriptions TwinResponse, ModuleMessages, Methods for client ot-06/CustomMetrics.
<6> 2022-08-16 10:32:01.361 +00:00 [INF] - Processing pending subscriptions for ot-06/CustomMetrics
<6> 2022-08-16 10:32:01.383 +00:00 [INF] - Set subscriptions from session state for ot-06/CustomMetrics
<6> 2022-08-16 10:32:01.476 +00:00 [INF] - Processing subscriptions TwinResponse, ModuleMessages, Methods for client ot-06/CustomMetrics.
<6> 2022-08-16 10:32:01.482 +00:00 [INF] - Set subscriptions from session state for ot-06/CustomMetrics
<6> 2022-08-16 10:32:01.496 +00:00 [INF] - Processing pending subscriptions for ot-06/CustomMetrics
<6> 2022-08-16 10:32:04.361 +00:00 [INF] - Updated reported properties for ot-06/$edgeHub
<6> 2022-08-16 10:32:07.854 +00:00 [INF] - New token received on the Cbs link
<6> 2022-08-16 10:32:08.482 +00:00 [INF] - Client ot-06/SensorHandler in device scope authenticated locally.
<6> 2022-08-16 10:32:08.538 +00:00 [INF] - New device connection for device ot-06/SensorHandler
<6> 2022-08-16 10:32:08.540 +00:00 [INF] - Client ot-06/SensorHandler connected to edgeHub, processing existing subscriptions.
<6> 2022-08-16 10:32:08.542 +00:00 [INF] - Attempting to connect to IoT Hub for client ot-06/SensorHandler via AMQP...
<6> 2022-08-16 10:32:08.549 +00:00 [INF] - Bind device proxy for device ot-06/SensorHandler
<6> 2022-08-16 10:32:08.559 +00:00 [INF] - Initialized device listener in the AMQP protocol head for ot-06/SensorHandler
<6> 2022-08-16 10:32:08.600 +00:00 [INF] - Opened link Events for ot-06/SensorHandler
<6> 2022-08-16 10:32:08.702 +00:00 [INF] - Cloud connection for ot-06/SensorHandler is True
<6> 2022-08-16 10:32:08.702 +00:00 [INF] - Connection status for ot-06/SensorHandler changed to ConnectionEstablished
<6> 2022-08-16 10:32:08.703 +00:00 [INF] - Client ot-06/SensorHandler connected to cloud, processing existing subscriptions.
<6> 2022-08-16 10:32:08.703 +00:00 [INF] - Skipping ot-06/SensorHandler for subscription processing, as it is currently being processed.
<6> 2022-08-16 10:32:08.703 +00:00 [INF] - Created cloud proxy for client ot-06/SensorHandler via AMQP, with client operation timeout 20 seconds.
<6> 2022-08-16 10:32:08.703 +00:00 [INF] - Initialized cloud proxy 7baac2d6-8075-46e0-9e57-959668f1c952 for ot-06/SensorHandler
<6> 2022-08-16 10:32:08.704 +00:00 [INF] - Created cloud connection for client ot-06/SensorHandler
<6> 2022-08-16 10:32:09.082 +00:00 [INF] - Opened link TwinReceiving for ot-06/SensorHandler
<6> 2022-08-16 10:32:09.091 +00:00 [INF] - Processing pending subscriptions for ot-06/SensorHandler
<6> 2022-08-16 10:32:09.105 +00:00 [INF] - Opened link TwinSending for ot-06/SensorHandler
<6> 2022-08-16 10:32:09.338 +00:00 [INF] - Opened link MethodSending for ot-06/SensorHandler
<6> 2022-08-16 10:32:09.342 +00:00 [INF] - Processing pending subscriptions for ot-06/SensorHandler
<6> 2022-08-16 10:32:09.362 +00:00 [INF] - Opened link MethodReceiving for ot-06/SensorHandler
<6> 2022-08-16 10:32:10.003 +00:00 [INF] - New token received on the Cbs link
<6> 2022-08-16 10:32:10.190 +00:00 [INF] - Updated reported properties for ot-06/$edgeHub
<6> 2022-08-16 10:32:10.356 +00:00 [INF] - Client ot-06/SensorRelay in device scope authenticated locally.
<6> 2022-08-16 10:32:10.358 +00:00 [INF] - New device connection for device ot-06/SensorRelay
<6> 2022-08-16 10:32:10.359 +00:00 [INF] - Client ot-06/SensorRelay connected to edgeHub, processing existing subscriptions.
<6> 2022-08-16 10:32:10.360 +00:00 [INF] - Attempting to connect to IoT Hub for client ot-06/SensorRelay via AMQP...
<6> 2022-08-16 10:32:10.361 +00:00 [INF] - Bind device proxy for device ot-06/SensorRelay
<6> 2022-08-16 10:32:10.361 +00:00 [INF] - Initialized device listener in the AMQP protocol head for ot-06/SensorRelay
<6> 2022-08-16 10:32:10.362 +00:00 [INF] - Opened link Events for ot-06/SensorRelay
<6> 2022-08-16 10:32:10.525 +00:00 [INF] - Cloud connection for ot-06/SensorRelay is True
<6> 2022-08-16 10:32:10.526 +00:00 [INF] - Connection status for ot-06/SensorRelay changed to ConnectionEstablished
<6> 2022-08-16 10:32:10.526 +00:00 [INF] - Client ot-06/SensorRelay connected to cloud, processing existing subscriptions.
<6> 2022-08-16 10:32:10.526 +00:00 [INF] - Skipping ot-06/SensorRelay for subscription processing, as it is currently being processed.
<6> 2022-08-16 10:32:10.526 +00:00 [INF] - Created cloud proxy for client ot-06/SensorRelay via AMQP, with client operation timeout 20 seconds.
<6> 2022-08-16 10:32:10.527 +00:00 [INF] - Initialized cloud proxy 054d1bba-9021-43d8-b28d-59a453fa94a0 for ot-06/SensorRelay
<6> 2022-08-16 10:32:10.527 +00:00 [INF] - Created cloud connection for client ot-06/SensorRelay
<6> 2022-08-16 10:32:10.706 +00:00 [INF] - Opened link TwinSending for ot-06/SensorRelay
<6> 2022-08-16 10:32:10.709 +00:00 [INF] - Processing pending subscriptions for ot-06/SensorRelay
<6> 2022-08-16 10:32:10.797 +00:00 [INF] - Opened link TwinReceiving for ot-06/SensorRelay
<6> 2022-08-16 10:32:14.138 +00:00 [INF] - Processing pending subscriptions for ot-06/SensorHandler
<6> 2022-08-16 10:32:14.140 +00:00 [INF] - Opened link ModuleMessages for ot-06/SensorHandler
<6> 2022-08-16 10:32:15.520 +00:00 [INF] - Opened link ModuleMessages for ot-06/SensorRelay
<6> 2022-08-16 10:32:15.520 +00:00 [INF] - Processing pending subscriptions for ot-06/SensorRelay
<6> 2022-08-16 10:32:16.375 +00:00 [INF] - Updated reported properties for ot-06/$edgeHub
<6> 2022-08-16 10:32:17.298 +00:00 [INF] - Updated reported properties for ot-06/SensorHandler
<6> 2022-08-16 10:32:18.035 +00:00 [INF] - Updated reported properties for ot-06/SensorRelay
<6> 2022-08-16 10:36:53.600 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<6> 2022-08-16 10:41:53.576 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<6> 2022-08-16 10:46:53.578 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<6> 2022-08-16 10:51:53.579 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<6> 2022-08-16 10:56:53.575 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<6> 2022-08-16 11:01:50.295 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint ot-06/SensorHandler/input1
<6> 2022-08-16 11:01:50.321 +00:00 [INF] - Cleaned up 0 messages from queue for endpoint ot-06/SensorHandler/input1 and 0 messages from message store.
<6> 2022-08-16 11:01:50.321 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint iothub
<6> 2022-08-16 11:01:50.323 +00:00 [INF] - Cleaned up 0 messages from queue for endpoint iothub and 0 messages from message store.
<6> 2022-08-16 11:01:50.323 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint ot-06/SensorRelay/deviceInput
<6> 2022-08-16 11:01:50.325 +00:00 [INF] - Cleaned up 0 messages from queue for endpoint ot-06/SensorRelay/deviceInput and 0 messages from message store.

Additional Information

Please provide any additional information that may be helpful in understanding the issue.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 15 (10 by maintainers)

Most upvoted comments

Varun said he might know what’s going on, but he’s OOF till next week, so I’m waiting for him to come back.

arsing on Sep 1, 2022