iotedge: Cannot resolve edgeAgent and edgeHub storage warning

Expected Behavior

iotedge check command should display storage settings resolved and mark ok (green).

Current Behavior

I still get thses warnings :

‼ production readiness: Edge Agent's storage directory is persisted on the host filesystem - Warning
    The edgeAgent module is not configured to persist its C:\Windows\Temp\edgeAgent directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.
‼ production readiness: Edge Hub's storage directory is persisted on the host filesystem - Warning
    The edgeHub module is not configured to persist its C:\\Windows\\Temp\edgeHub directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.

Steps to Reproduce

Provide a detailed set of steps to reproduce the bug.

  1. create a normal deployment with no custom modules
  2. define binds like example in the provided link. here is my deployment json file:
{
    "modulesContent": {
        "$edgeAgent": {
            "properties.desired": {
                "schemaVersion": "1.0",
                "runtime": {
                    "type": "docker",
                    "settings": {
                        "minDockerVersion": "v1.25",
                    }
                },
                "systemModules": {
                    "edgeAgent": {
                        "settings": {
                            "image": "mcr.microsoft.com/azureiotedge-agent:1.0.9",
                            "createOptions": "{\"HostConfig\": {\"Binds\":[\"C:\\\\temp\\\\edgeAgent:C:\\\\Windows\\\\Temp\\\\edgeAgent\"]}}"
                        },
                        "type": "docker",
                        "env": {
                            "experimentalfeatures__enabled": {
                                "value": "true"
                            },
                            "experimentalfeatures__enableMetrics": {
                                "value": "true"
                            },
                            "experimentalfeatures__enableGetLogs": {
                                "value": "true"
                            },
                            "experimentalfeatures__enableUploadLogs": {
                                "value": "true"
                            },
                            "storageFolder": {
                                "value": "C:\\\\Windows\\\\Temp"
                            }
                        }
                    },
                    "edgeHub": {
                        "settings": {
                            "image": "mcr.microsoft.com/azureiotedge-hub:1.0.9",
                            "createOptions": "{\"HostConfig\":{\"Binds\":[\"C:\\\\temp\\\\edgeHub:C:\\\\Windows\\\\Temp\\\\edgeHub\"],\"PortBindings\":{\"8883/tcp\":[{\"HostPort\":\"8883\"}],\"5671/tcp\":[{\"HostPort\":\"5671\"}],\"443/tcp\":[{\"HostPort\":\"443\"}]}}}"
                        },
                        "type": "docker",
                        "status": "running",
                        "restartPolicy": "always",
                        "env": {
                            "storageFolder": {
                                "value": "C:\\\\Windows\\\\Temp"
                            }
                        }
                    }
                },
                "modules":{}
            }
        },
        "$edgeHub": {
            "properties.desired": {
                "routes": {},
                "schemaVersion": "1.0",
                "storeAndForwardConfiguration": {
                    "timeToLiveSecs": 7200
                }
            }
        }
    }
}
  1. after edge device received the deployment, run iotedge check

Context (Environment)

Windows 10

Output of iotedge check

√ config.yaml is well-formed - OK
‼ config.yaml has well-formed connection string - Warning
    Device not configured with manual provisioning, in this configuration 'iotedge check' is not able to discover the device's backing IoT Hub.
    To run connectivity checks in this configuration please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    If no hostname is provided, all hub connectivity tests will be skipped.
√ container engine is installed and functional - OK
√ Windows host version is supported - OK
√ config.yaml has correct hostname - OK
√ config.yaml has correct URIs for daemon mgmt endpoint - OK
√ latest security daemon - OK
√ host time is close to real time - OK
√ container time is close to host time - OK
√ DNS server - OK
√ production readiness: certificates - OK
√ production readiness: container engine - OK
√ production readiness: logs policy - OK
‼ production readiness: Edge Agent's storage directory is persisted on the host filesystem - Warning
    The edgeAgent module is not configured to persist its C:\Windows\Temp\edgeAgent directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.
‼ production readiness: Edge Hub's storage directory is persisted on the host filesystem - Warning
    The edgeHub module is not configured to persist its C:\\Windows\\Temp\edgeHub directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.

Connectivity checks
-------------------
√ host can connect to and perform TLS handshake with DPS endpoint - OK

13 check(s) succeeded.
3 check(s) raised warnings. Re-run with --verbose for more details.
6 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details.

Device Information

  • Host OS : Windows 10
  • Architecture : amd64
  • Container OS : Windows containers

Runtime Versions

  • iotedged : iotedge 1.0.9 (1f7cf6a21dfa365c8f21475553c544f4df02dd44)
  • Edge Agent [image tag (e.g. 1.0.0)]:
  • Edge Hub [image tag (e.g. 1.0.0)]:
  • Docker/Moby [run docker version]: 3.0.5

Note: when using Windows containers on Windows, run docker -H npipe:////./pipe/iotedge_moby_engine version instead

Logs

iotedged logs 

<Paste here between the triple backticks>
edge-agent logs 

2020-04-24 10:13:11.112 +00:00 Edge Agent Main()
<6> 2020-04-24 10:13:11.990 +00:00 [INF] - Initializing Edge Agent.
<6> 2020-04-24 10:13:12.693 +00:00 [INF] - Version - 1.0.9.29532150 (1f7cf6a21dfa365c8f21475553c544f4df02dd44)
<6> 2020-04-24 10:13:12.694 +00:00 [INF] -
        █████╗ ███████╗██╗   ██╗██████╗ ███████╗
       ██╔══██╗╚══███╔╝██║   ██║██╔══██╗██╔════╝
       ███████║  ███╔╝ ██║   ██║██████╔╝█████╗
       ██╔══██║ ███╔╝  ██║   ██║██╔══██╗██╔══╝
       ██║  ██║███████╗╚██████╔╝██║  ██║███████╗
       ╚═╝  ╚═╝╚══════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝

 ██╗ ██████╗ ████████╗    ███████╗██████╗  ██████╗ ███████╗
 ██║██╔═══██╗╚══██╔══╝    ██╔════╝██╔══██╗██╔════╝ ██╔════╝
 ██║██║   ██║   ██║       █████╗  ██║  ██║██║  ███╗█████╗
 ██║██║   ██║   ██║       ██╔══╝  ██║  ██║██║   ██║██╔══╝
 ██║╚██████╔╝   ██║       ███████╗██████╔╝╚██████╔╝███████╗
 ╚═╝ ╚═════╝    ╚═╝       ╚══════╝╚═════╝  ╚═════╝ ╚══════╝

<6> 2020-04-24 10:13:13.008 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false,"EnableUploadLogs":false,"EnableGetLogs":false,"EnableMetrics":false}
<6> 2020-04-24 10:13:13.745 +00:00 [INF] - Installing certificates [CN=Azure IoT CA TestOnly Root CA:5/22/2020 8:37:51 AM] to CertificateAuthority
<6> 2020-04-24 10:13:15.687 +00:00 [INF] - Created persistent store at C:\Windows\TEMP\edgeAgent
<6> 2020-04-24 10:13:16.108 +00:00 [INF] - Started operation refresh twin config
<6> 2020-04-24 10:13:16.283 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...
<6> 2020-04-24 10:13:16.509 +00:00 [INF] - Registering request handler RestartModule
<6> 2020-04-24 10:13:19.151 +00:00 [INF] - Edge agent connected to IoT Hub via Amqp_Tcp_Only.
<6> 2020-04-24 10:13:19.524 +00:00 [INF] - Initialized new module client with subscriptions enabled
<6> 2020-04-24 10:13:19.764 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 46 and reported properties version 617.
<6> 2020-04-24 10:13:20.976 +00:00 [INF] - Plan execution started for deployment 46
<6> 2020-04-24 10:13:21.058 +00:00 [INF] - Executing command: "Command Group: (\n  [Create module edgeHub]\n  [Start module edgeHub]\n)"
<6> 2020-04-24 10:13:21.085 +00:00 [INF] - Executing command: "Create module edgeHub"
<6> 2020-04-24 10:13:35.589 +00:00 [INF] - Executing command: "Start module edgeHub"
<6> 2020-04-24 10:13:39.428 +00:00 [INF] - Plan execution ended for deployment 46
<6> 2020-04-24 10:13:40.379 +00:00 [INF] - Updated reported properties
<6> 2020-04-24 10:13:45.792 +00:00 [INF] - Updated reported properties
edge-hub logs 

2020-04-24 10:13:41.172 +00:00 Edge Hub Main()
<6> 2020-04-24 10:13:45.495 +00:00 [INF] - Installing certificates [CN=iotedged workload ca:5/22/2020 8:39:54 AM],[CN=MyEdgeDeviceCA.ca:5/22/2020 8:39:54 AM],[CN=Azure IoT CA TestOnly Intermediate CA:5/22/2020 8:38:02 AM],[CN=Azure IoT CA TestOnly Root CA:5/22/2020 8:37:51 AM] to CertificateAuthority
<6> 2020-04-24 10:13:45.693 +00:00 [INF] - Installing certificates [CN=Azure IoT CA TestOnly Root CA:5/22/2020 8:37:51 AM] to CertificateAuthority
<6> 2020-04-24 10:13:49.147 +00:00 [INF] - Created persistent store at C:\\Windows\\Temp\edgeHub
<6> 2020-04-24 10:13:49.347 +00:00 [INF] - Initializing Edge Hub
<6> 2020-04-24 10:13:49.384 +00:00 [INF] -
        █████╗ ███████╗██╗   ██╗██████╗ ███████╗
       ██╔══██╗╚══███╔╝██║   ██║██╔══██╗██╔════╝
       ███████║  ███╔╝ ██║   ██║██████╔╝█████╗
       ██╔══██║ ███╔╝  ██║   ██║██╔══██╗██╔══╝
       ██║  ██║███████╗╚██████╔╝██║  ██║███████╗
       ╚═╝  ╚═╝╚══════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝

 ██╗ ██████╗ ████████╗    ███████╗██████╗  ██████╗ ███████╗
 ██║██╔═══██╗╚══██╔══╝    ██╔════╝██╔══██╗██╔════╝ ██╔════╝
 ██║██║   ██║   ██║       █████╗  ██║  ██║██║  ███╗█████╗
 ██║██║   ██║   ██║       ██╔══╝  ██║  ██║██║   ██║██╔══╝
 ██║╚██████╔╝   ██║       ███████╗██████╔╝╚██████╔╝███████╗
 ╚═╝ ╚═════╝    ╚═╝       ╚══════╝╚═════╝  ╚═════╝ ╚══════╝

<6> 2020-04-24 10:13:49.393 +00:00 [INF] - Version - 1.0.9.29532150 (1f7cf6a21dfa365c8f21475553c544f4df02dd44)
<6> 2020-04-24 10:13:49.393 +00:00 [INF] - OptimizeForPerformance=True
<6> 2020-04-24 10:13:49.403 +00:00 [INF] - Loaded server certificate with expiration date of "2020-05-22T08:39:54.0000000+00:00"
<6> 2020-04-24 10:13:49.623 +00:00 [INF] - Created new message store
<6> 2020-04-24 10:13:49.694 +00:00 [INF] - Created device scope identities cache
<6> 2020-04-24 10:13:49.733 +00:00 [INF] - Starting refresh of device scope identities cache
<6> 2020-04-24 10:13:50.113 +00:00 [INF] - Started task to cleanup processed and stale messages
<6> 2020-04-24 10:13:50.315 +00:00 [INF] - Created DeviceConnectivityManager with connected check frequency 00:05:00 and disconnected check frequency 00:02:00
<6> 2020-04-24 10:13:50.716 +00:00 [INF] - Initialized storing twin manager
<6> 2020-04-24 10:13:50.765 +00:00 [INF] - Initializing configuration
<6> 2020-04-24 10:13:50.831 +00:00 [INF] - New device connection for device p2-az-220-hu/$edgeHub
<6> 2020-04-24 10:13:50.867 +00:00 [INF] - Processing pending subscriptions for p2-az-220-hu/$edgeHub
<6> 2020-04-24 10:13:51.197 +00:00 [INF] - Attempting to connect to IoT Hub for client p2-az-220-hu/$edgeHub via AMQP...
<4> 2020-04-24 10:13:51.474 +00:00 [WRN] - Empty edge hub configuration received. Ignoring...
<6> 2020-04-24 10:13:53.577 +00:00 [INF] - Exiting disconnected state
<6> 2020-04-24 10:13:53.587 +00:00 [INF] - Received device connected callback
<6> 2020-04-24 10:13:53.624 +00:00 [INF] - Device connected to cloud, processing subscriptions for connected clients.
<6> 2020-04-24 10:13:53.634 +00:00 [INF] - Processing subscriptions for client p2-az-220-hu/$edgeHub on device connected.
<6> 2020-04-24 10:13:53.690 +00:00 [INF] - Entering connected state
<6> 2020-04-24 10:13:53.694 +00:00 [INF] - Cloud connection for p2-az-220-hu/$edgeHub is True
<6> 2020-04-24 10:13:53.713 +00:00 [INF] - Connection status for p2-az-220-hu/$edgeHub changed to ConnectionEstablished
<6> 2020-04-24 10:13:53.721 +00:00 [INF] - Client p2-az-220-hu/$edgeHub connected to cloud, processing existing subscriptions.
<6> 2020-04-24 10:13:53.732 +00:00 [INF] - Created cloud proxy for client p2-az-220-hu/$edgeHub via AMQP, with client operation timeout 20 seconds.
<6> 2020-04-24 10:13:53.751 +00:00 [INF] - Initialized cloud proxy dbc04422-26bb-4e01-b710-7cfc7581628e for p2-az-220-hu/$edgeHub
<6> 2020-04-24 10:13:53.752 +00:00 [INF] - Created cloud connection for client p2-az-220-hu/$edgeHub
<6> 2020-04-24 10:13:54.360 +00:00 [INF] - Processing pending subscriptions for p2-az-220-hu/$edgeHub
<6> 2020-04-24 10:13:54.695 +00:00 [INF] - Updated reported properties for p2-az-220-hu/$edgeHub
<6> 2020-04-24 10:13:54.737 +00:00 [INF] - Obtained edge hub config from module twin
<6> 2020-04-24 10:13:54.780 +00:00 [INF] - No routes set in the edge hub
<6> 2020-04-24 10:13:54.781 +00:00 [INF] - Updated message store TTL to 7200 seconds
<6> 2020-04-24 10:13:54.781 +00:00 [INF] - Updated the edge hub store and forward configuration
<6> 2020-04-24 10:13:54.805 +00:00 [INF] - Started operation Get EdgeHub config
<6> 2020-04-24 10:13:54.806 +00:00 [INF] - Initialized edge hub configuration
<6> 2020-04-24 10:13:54.843 +00:00 [INF] - Starting timer to authenticate connections with a period of 300 seconds
<6> 2020-04-24 10:13:55.137 +00:00 [INF] - Scheduling server certificate renewal for "2020-05-22T08:37:24.0000865Z".
<6> 2020-04-24 10:13:55.139 +00:00 [INF] - Starting protocol heads - (MQTT, AMQP, HTTP)
<6> 2020-04-24 10:13:55.148 +00:00 [INF] - Starting MQTT head
<6> 2020-04-24 10:13:55.251 +00:00 [INF] - Initializing TLS endpoint on port 8883 for MQTT head.
<6> 2020-04-24 10:13:55.476 +00:00 [INF] - Starting AMQP head
<6> 2020-04-24 10:13:55.536 +00:00 [INF] - Started MQTT head
<6> 2020-04-24 10:13:55.848 +00:00 [INF] - Updated reported properties for p2-az-220-hu/$edgeHub
<6> 2020-04-24 10:13:55.867 +00:00 [INF] - Started AMQP head
<6> 2020-04-24 10:13:55.876 +00:00 [INF] - Starting HTTP head
<6> 2020-04-24 10:13:56.745 +00:00 [INF] - User profile is available. Using '"C:\\Users\\ContainerUser\\AppData\\Local\\ASP.NET\\DataProtection-Keys"' as key repository and Windows DPAPI to encrypt keys at rest.
<6> 2020-04-24 10:13:56.938 +00:00 [INF] - Creating key {4e0ce94c-9512-4e55-855f-f43d5960e435} with creation date 2020-04-24 10:13:56Z, activation date 2020-04-24 10:13:56Z, and expiration date 2020-07-23 10:13:56Z.
<6> 2020-04-24 10:13:57.297 +00:00 [INF] - Writing data to file '"C:\\Users\\ContainerUser\\AppData\\Local\\ASP.NET\\DataProtection-Keys\\key-4e0ce94c-9512-4e55-855f-f43d5960e435.xml"'.
<4> 2020-04-24 10:13:58.666 +00:00 [WRN] - Overriding address(es) '"http://+:80"'. Binding to endpoints defined in "UseKestrel()" instead.
<6> 2020-04-24 10:13:58.743 +00:00 [INF] - Started HTTP head
<6> 2020-04-24 10:18:54.874 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<6> 2020-04-24 10:23:54.867 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<6> 2020-04-24 10:28:08.382 +00:00 [INF] - Request starting HTTP/1.1 GET https://52.178.152.23/cgi-bin/luci
<6> 2020-04-24 10:28:08.710 +00:00 [INF] - Request finished in 347.0905ms 401 text/html
<6> 2020-04-24 10:28:11.314 +00:00 [INF] - Request starting HTTP/1.1 GET https://52.178.152.23/dana-na/auth/url_default/welcome.cgi
<6> 2020-04-24 10:28:11.314 +00:00 [INF] - Request finished in 0.363ms 401 text/html
<6> 2020-04-24 10:28:15.003 +00:00 [INF] - Request starting HTTP/1.1 GET https://52.178.152.23/remote/login?lang=en
<6> 2020-04-24 10:28:15.003 +00:00 [INF] - Request finished in 0.2092ms 401 text/html
<6> 2020-04-24 10:28:25.260 +00:00 [INF] - HttpExtensionConnectionAdapter authentication failed
<6> 2020-04-24 10:28:54.884 +00:00 [INF] - Entering periodic task to reauthenticate connected clients

Additional Information

I also tried docker inspect, on edgeHub the bind is listed and also the environment variable exists. but on edgeHub not affected.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 22 (5 by maintainers)

Most upvoted comments

Same… @veyalla perhaps this should be re-opened and closed when the docs are better or more complete?

+1
kpm-at-hfi on Sep 2, 2020

Hi, I’m running on the exact same issue on linux machine… I’ve tried the proposed remediation steps without success… any update on this??

+1
ralarcon on Aug 14, 2020

I have the same problem on Linux. I went through Give modules access to a device’s local storage and updated my deployment templates. Afterwards the edgeHub storage warning went away but not edgeAgent. They both have the same settings in deployment template but only edgeHub gets the env var set not edgeAgent.

"systemModules": {
    "edgeAgent": {
        "type": "docker",
        "settings": {
            "image": "mcr.microsoft.com/azureiotedge-agent:1.0",
            "createOptions": {
                "HostConfig": {
                    "Binds": [
                        "/etc/iotedge/storage/:/iotedge/storage/"
                    ]
                }
            }
        },
        "env": {
            "storageFolder": {
                "value": "/iotedge/storage/"
            }
        }
    },
    "edgeHub": {
        "type": "docker",
        "status": "running",
        "restartPolicy": "always",
        "settings": {
            "image": "mcr.microsoft.com/azureiotedge-hub:1.0",
            "createOptions": {
                "HostConfig": {
                    "Binds": [
                        "/etc/iotedge/storage/:/iotedge/storage/"
                    ],
                    "PortBindings": {
                        "5671/tcp": [
                            {
                                "HostPort": "5671"
                            }
                        ],
                        "8883/tcp": [
                            {
                                "HostPort": "8883"
                            }
                        ],
                        "443/tcp": [
                            {
                                "HostPort": "443"
                            }
                        ]
                    }
                }
            }
        },
        "env": {
            "storageFolder": {
                "value": "/iotedge/storage/"
            }
        }
    }
}

I also tried config.yaml tweaks, same behavior.

agent:
  name: "edgeAgent"
  type: "docker"
  env:
    storageFolder: "/etc/iotedge/storage"
  config:
    image: "mcr.microsoft.com/azureiotedge-agent:1.0"
    auth: {}
+1
thnk2wn on May 7, 2020
Read more comments on GitHub
← iotedge: Automatic provisioning with DPS using X.509 not working using iotedge v1.0.9.3
iotedge: Connections refused (err 111) connecting to unix:///var/run/iotedge/workload.sock after edgeAgent restart →