iotedge: EdgeHub module turns in backoff state with error code 132 over and over again

Expected Behavior

EdgeHub module should start, run normally and send messages to IoT Hub periodically (approx. 3-4 messages per minute) without errors. Messages are being delivered to edgeHub from two custom IoT Edge modules written in Python.

Current Behavior

EdgeHub module starts and changes its state to “running” for approx. 30 seconds. Then it turns into “backoff” state with exit code 132. After a while, edgeHub is restarted and whole situation repeats (i.e. edgeHub module starts and changes its state to “running” for approx. 30 seconds. Then it turns into “backoff” state with exit code 132)

Steps to Reproduce

  1. systemctl disable ntp
  2. systemctl disable iotedge
  3. date -s “2016-11-11”
  4. hwclock -w
  5. disable internet connection
  6. reboot
  7. systemctl start iotedge
  8. wait ~1 minute
  9. enable internet connection
  10. systemctl start ntp
  11. wait ~1minute
  12. edgeHub module should fall into backoff state with error code 132 and restart over and over again

Context (Environment)

IoT Edge runtime is installed on single-board-computer with Debian 9 on it. Its product name is Moxa-UC-8112-LX - device similar to Raspberry Pi 3B. Moxa has hardware GSM/LTE module plugged-in, which allows Internet connection.

Output of iotedge check --verbose

Click here

Configuration checks
--------------------
√ config.yaml is well-formed - OK
√ config.yaml has well-formed connection string - OK
√ container engine is installed and functional - OK
√ config.yaml has correct hostname - OK
√ config.yaml has correct URIs for daemon mgmt endpoint - OK
√ latest security daemon - OK
√ host time is close to real time - OK
√ container time is close to host time - OK
‼ DNS server - Warning
    Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
    Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
    You can ignore this warning if you are setting DNS server per module in the Edge deployment.
‼ production readiness: certificates - Warning
    Device is using self-signed, automatically generated certs.
    Please see https://aka.ms/iotedge-prod-checklist-certs for best practices.
× production readiness: certificates expiry - Error
    Device CA certificate in /var/lib/iotedge/hsm/certs/device_ca_aliasazxVrrdEVxd7kvKvne1pOEyuSHF8EXSowNDhMzl30jI_.cert.pem expired at 2017-02-01 17:17:27 UTC
√ production readiness: container engine - OK
√ production readiness: logs policy - OK

Connectivity checks
-------------------
√ host can connect to and perform TLS handshake with IoT Hub AMQP port - OK
√ host can connect to and perform TLS handshake with IoT Hub HTTPS / WebSockets port - OK
√ host can connect to and perform TLS handshake with IoT Hub MQTT port - OK
√ container on the default network can connect to IoT Hub AMQP port - OK
√ container on the default network can connect to IoT Hub HTTPS / WebSockets port - OK
√ container on the default network can connect to IoT Hub MQTT port - OK
√ container on the IoT Edge module network can connect to IoT Hub AMQP port - OK
√ container on the IoT Edge module network can connect to IoT Hub HTTPS / WebSockets port - OK
√ container on the IoT Edge module network can connect to IoT Hub MQTT port - OK
√ Edge Hub can bind to ports on host - OK

20 check(s) succeeded.
2 check(s) raised warnings.
1 check(s) raised errors.


Device Information

  • Host OS: Debian 9:
  • Architecture: armv7hf,
  • Container OS: Linux containers

Runtime Versions

  • iotedged: 1.0.8
  • Edge Agent (logs provided for version 1.0, but already tryied all of them): 1.0, 1.0.5, 1.0.6, 1.0.7, 1.0.8
  • Edge Hub: (logs provided for version 1.0, but already tryied all of them): 1.0, 1.0.5, 1.0.6, 1.0.7, 1.0.8
  • Docker/Moby: 3.0.6

Logs

iotedged logs
Oct 03 09:53:50 moxa iotedged[1281]: 2019-10-03T07:53:50Z [INFO] - Successfully started module edgeHub
Oct 03 09:53:50 moxa iotedged[1281]: 2019-10-03T07:53:50Z [INFO] - [mgmt] - - - [2019-10-03 07:53:50.354029765 UTC] "POST /modules/edgeHub/start?api-version=2019-01-30 HTTP/1.1" 204 No Content - "-" "-" auth_id(-)
Oct 03 09:53:51 moxa iotedged[1281]: 2019-10-03T07:53:51Z [INFO] - [work] - - - [2019-10-03 07:53:51.626729826 UTC] "POST /modules/%24edgeAgent/genid/637055217905681836/encrypt?api-version=2019-01-30 HTTP/1.1" 200 OK 2169 "-" "-" auth_id(-)
Oct 03 09:53:57 moxa iotedged[1281]: 2019-10-03T07:53:57Z [INFO] - [mgmt] - - - [2019-10-03 07:53:57.501095375 UTC] "GET /modules?api-version=2019-01-30 HTTP/1.1" 200 OK 2049 "-" "-" auth_id(-)
[ 7110.243075] br-096d9f841933: port 1(vethdb33d43) entered forwarding state
Oct 03 09:54:03 moxa iotedged[1281]: 2019-10-03T07:54:03Z [INFO] - [mgmt] - - - [2019-10-03 07:54:03.813832203 UTC] "GET /modules?api-version=2019-01-30 HTTP/1.1" 200 OK 2049 "-" "-" auth_id(-)
Oct 03 09:54:09 moxa iotedged[1281]: 2019-10-03T07:54:09Z [INFO] - [mgmt] - - - [2019-10-03 07:54:09.485045309 UTC] "GET /modules?api-version=2019-01-30 HTTP/1.1" 200 OK 2049 "-" "-" auth_id(-)
Oct 03 09:54:15 moxa iotedged[1281]: 2019-10-03T07:54:15Z [INFO] - [mgmt] - - - [2019-10-03 07:54:15.465957047 UTC] "GET /modules?api-version=2019-01-30 HTTP/1.1" 200 OK 2049 "-" "-" auth_id(-)
Oct 03 09:54:31 moxa iotedged[1281]: 2019-10-03T07:54:31Z [ERR!] - Internal server error: Could not get server cert
Oct 03 09:54:31 moxa iotedged[1281]:         caused by: An error occurred creating the certificate
Oct 03 09:54:31 moxa iotedged[1281]:         caused by: HSM failure
Oct 03 09:54:31 moxa iotedged[1281]:         caused by: HSM API returned an invalid null response
Oct 03 09:54:31 moxa iotedged[1281]: 2019-10-03T07:54:31Z [INFO] - [work] - - - [2019-10-03 07:54:31.084605277 UTC] "POST /modules/%24edgeHub/genid/637055217905681836/certificate/server?api-version=2019-01-30 HTTP/1.1" 500 Internal Server Error 178 "-" "-" auth_id(-)
Oct 03 09:54:31 moxa iotedged[1281]: 2019-10-03T07:54:31Z [INFO] - [mgmt] - - - [2019-10-03 07:54:31.340886805 UTC] "GET /modules?api-version=2019-01-30 HTTP/1.1" 200 OK 2049 "-" "-" auth_id(-)
Oct 03 09:54:37 moxa iotedged[1281]: ure-iot-hsm-c/src/edge_pki_openssl.c:validate_certificate_expiration:806) Certificate has expired
Oct 03 09:54:37 moxa iotedged[1281]: 2019-10-03T07:52:45Z [ERR!] (/project/hsm-sys/azure-iot-hsm-c/src/edge_pki_openssl.c:cert_set_expiration:849) Issuer certificate expiration failure. Status 0, verify status: 1
Oct 03 09:54:37 moxa iotedged[1281]: 2019-10-03T07:52:45Z [ERR!] (/project/hsm-sys/azure-iot-hsm-c/src/edge_pki_openssl.c:generate_evp_certificate:1347) Failure setting certificate validity period
Oct 03 09:54:37 moxa iotedged[1281]: 2019-10-03T07:52:45Z [ERR!] (/project/hsm-sys/azure-iot-hsm-c/src/edge_pki_openssl.c:generate_pki_cert_and_key_helper:1525) Could not generate certificate create request
Oct 03 09:54:37 moxa iotedged[1281]: 2019-10-03T07:52:45Z [ERR!] (/project/hsm-sys/azure-iot-hsm-c/src/edge_hsm_client_store.c:edge_hsm_client_store_create_pki_cert_internal:2633) Could not create PKI certificate and key for $edgeHub637055217905681836server
Oct 03 09:54:37 moxa iotedged[1281]: 2019-10-03T07:52:45Z [ERR!] (/project/hsm-sys/azure-iot-hsm-c/src/edge_hsm_client_store.c:edge_hsm_client_store_create_pki_cert:2709) Could not create certificate and key for alias $edgeHub637055217905681836server
Oct 03 09:54:37 moxa iotedged[1281]: 2019-10-03T07:52:45Z [ERR!] (/project/hsm-sys/azure-iot-hsm-c/src/edge_hsm_client_crypto.c:edge_hsm_client_create_certificate:285) Could not create certificate in the store
Oct 03 09:54:37 moxa iotedged[1281]: 2019-10-03T07:54:20Z [ERR!] (/project/hsm-sys/azure-iot-hsm-c/src/edge_hsm_client_store.c:remove_if_cert_and_key_exist_by_alias:1443) Certificate and key file for alias do not exist $edgeHub637055217905681836server
Oct 03 09:54:37 moxa iotedged[1281]: 2019-10-03T07:54:31Z [ERR!] (/project/hsm-sys/azure-iot-hsm-c/src/edge_pki_openssl.c:validate_certificate_expiration:806) Certificate has expired
Oct 03 09:54:37 moxa iotedged[1281]: 2019-10-03T07:54:31Z [ERR!] (/project/hsm-sys/azure-iot-hsm-c/src/edge_pki_openssl.c:cert_set_expiration:849) Issuer certificate expiration failure. Status 0, verify status: 1
Oct 03 09:54:37 moxa iotedged[1281]: 2019-10-03T07:54:31Z [ERR!] (/project/hsm-sys/azure-iot-hsm-c/src/edge_pki_openssl.c:generate_evp_certificate:1347) Failure setting certificate validity period
Oct 03 09:54:37 moxa iotedged[1281]: 2019-10-03T07:54:31Z [ERR!] (/project/hsm-sys/azure-iot-hsm-c/src/edge_pki_openssl.c:generate_pki_cert_and_key_helper:1525) Could not generate certificate create request
Oct 03 09:54:37 moxa iotedged[1281]: 2019-10-03T07:54:31Z [ERR!] (/project/hsm-sys/azure-iot-hsm-c/src/edge_hsm_client_store.c:edge_hsm_client_store_create_pki_cert_internal:2633) Could not create PKI certificate and key for $edgeHub637055217905681836server
Oct 03 09:54:37 moxa iotedged[1281]: 2019-10-03T07:54:31Z [ERR!] (/project/hsm-sys/azure-iot-hsm-c/src/edge_hsm_client_store.c:edge_hsm_client_store_create_pki_cert:2709) Could not create certificate and key for alias $edgeHub637055217905681836server
Oct 03 09:54:37 moxa iotedged[1281]: 2019-10-03T07:54:31Z [ERR!] (/project/hsm-sys/azure-iot-hsm-c/src/edge_hsm_client_crypto.c:edge_hsm_client_create_certificate:285) Could not create certificate in the store
Oct 03 09:54:37 moxa iotedged[1281]: 2019-10-03T07:54:35Z [ERR!] (/project/hsm-sys/azure-iot-hsm-c/src/edge_hsm_client_store.c:remove_if_cert_and_key_exist_by_alias:1443) Certificate and key file for alias do not exist $edgeHub637055217905681836server
Oct 03 09:54:37 moxa iotedged[1281]: 2019-10-03T07:54:37Z [ERR!] (/project/hsm-sys/azure-iot-hsm-c/src/edge_pki_openssl.c:validate_certificate_expiration:806) Certificate has expired
Oct 03 09:54:37 moxa iotedged[1281]: 2019-10-03T07:54:37Z [ERR!] (/project/hsm-sys/azure-iot-hsm-c/src/edge_pki_openssl.c:cert_set_expiration:849) Issuer certificate expiration failure. Status 0, verify status: 1
Oct 03 09:54:37 moxa iotedged[1281]: 2019-10-03T07:54:37Z [ERR!] (/project/hsm-sys/azure-iot-hsm-c/src/edge_pki_openssl.c:generate_evp_certificate:1347) Failure setting certificate validity period
Oct 03 09:54:37 moxa iotedged[1281]: 2019-10-03T07:54:37Z [ERR!] (/project/hsm-sys/azure-iot-hsm-c/src/edge_pki_openssl.c:generate_pki_cert_and_key_helper:1525) Could not generate certificate create request
Oct 03 09:54:37 moxa iotedged[1281]: 2019-10-03T07:54:37Z [ERR!] (/project/hsm-sys/azure-iot-hsm-c/src/edge_hsm_client_store.c:edge_hsm_client_store_create_pki_cert_internal:2633) Could not create PKI certificate and key for $edgeHub637055217905681836server
Oct 03 09:54:37 moxa iotedged[1281]: 2019-10-03T07:54:37Z [ERR!] (/project/hsm-sys/azure-iot-hsm-c/src/edge_hsm_client_store.c:edge_hsm_client_store_create_pki_cert:2709) Could not create certificate and key for alias $edgeHub637055217905681836server
Oct 03 09:54:37 moxa iotedged[1281]: 2019-10-03T07:54:37<3>2019-10-03T07:54:37Z [ERR!] - Internal server error: Could not get server cert
edge-agent logs
{"log":"\u003c6\u003e 2019-10-03 07:31:29.975 +00:00 [INF] - Plan execution started for deployment 15\n","stream":"stdout","time":"2019-10-03T07:31:29.97623601Z"}
{"log":"\u003c6\u003e 2019-10-03 07:31:29.977 +00:00 [INF] - Executing command: \"Command Group: (\\n  [Stop module edgeHub]\\n  [Start module edgeHub]\\n  [Saving edgeHub to store]\\n)\"\n","stream":"stdout","time":"2019-10-03T07:31:29.97965085Z"}
{"log":"\u003c6\u003e 2019-10-03 07:31:29.981 +00:00 [INF] - Executing command: \"Stop module edgeHub\"\n","stream":"stdout","time":"2019-10-03T07:31:29.982418219Z"}
{"log":"\u003c6\u003e 2019-10-03 07:31:30.040 +00:00 [INF] - Executing command: \"Start module edgeHub\"\n","stream":"stdout","time":"2019-10-03T07:31:30.042043815Z"}
{"log":"\u003c6\u003e 2019-10-03 07:31:41.355 +00:00 [INF] - Executing command: \"Saving edgeHub to store\"\n","stream":"stdout","time":"2019-10-03T07:31:41.35840133Z"}
{"log":"\u003c6\u003e 2019-10-03 07:31:41.369 +00:00 [INF] - Plan execution ended for deployment 15\n","stream":"stdout","time":"2019-10-03T07:31:41.371722178Z"}
{"log":"\u003c6\u003e 2019-10-03 07:31:42.360 +00:00 [INF] - Updated reported properties\n","stream":"stdout","time":"2019-10-03T07:31:42.361903021Z"}
{"log":"\u003c6\u003e 2019-10-03 07:31:48.126 +00:00 [INF] - Updated reported properties\n","stream":"stdout","time":"2019-10-03T07:31:48.130162958Z"}
{"log":"\u003c6\u003e 2019-10-03 07:32:43.170 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (04m:55s left).\n","stream":"stdout","time":"2019-10-03T07:32:43.193508974Z"}
{"log":"\u003c6\u003e 2019-10-03 07:32:43.445 +00:00 [INF] - Updated reported properties\n","stream":"stdout","time":"2019-10-03T07:32:43.446793249Z"}
{"log":"\u003c6\u003e 2019-10-03 07:32:49.066 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (04m:49s left).\n","stream":"stdout","time":"2019-10-03T07:32:49.070454558Z"}
{"log":"\u003c6\u003e 2019-10-03 07:32:54.504 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (04m:44s left).\n","stream":"stdout","time":"2019-10-03T07:32:54.505838634Z"}
{"log":"\u003c6\u003e 2019-10-03 07:33:00.269 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (04m:38s left).\n","stream":"stdout","time":"2019-10-03T07:33:00.277929993Z"}
{"log":"\u003c6\u003e 2019-10-03 07:33:05.865 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (04m:32s left).\n","stream":"stdout","time":"2019-10-03T07:33:05.867748648Z"}
{"log":"\u003c6\u003e 2019-10-03 07:33:11.215 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (04m:27s left).\n","stream":"stdout","time":"2019-10-03T07:33:11.216567724Z"}
{"log":"\u003c6\u003e 2019-10-03 07:33:16.795 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (04m:21s left).\n","stream":"stdout","time":"2019-10-03T07:33:16.797294904Z"}
{"log":"\u003c6\u003e 2019-10-03 07:33:22.169 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (04m:16s left).\n","stream":"stdout","time":"2019-10-03T07:33:22.171503272Z"}
{"log":"\u003c6\u003e 2019-10-03 07:33:27.461 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (04m:11s left).\n","stream":"stdout","time":"2019-10-03T07:33:27.462761086Z"}
{"log":"\u003c6\u003e 2019-10-03 07:33:38.838 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (03m:59s left).\n","stream":"stdout","time":"2019-10-03T07:33:38.840005963Z"}
{"log":"\u003c6\u003e 2019-10-03 07:33:44.131 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (03m:54s left).\n","stream":"stdout","time":"2019-10-03T07:33:44.133270133Z"}
{"log":"\u003c6\u003e 2019-10-03 07:33:49.739 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (03m:48s left).\n","stream":"stdout","time":"2019-10-03T07:33:49.742963891Z"}
{"log":"\u003c6\u003e 2019-10-03 07:33:55.125 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (03m:43s left).\n","stream":"stdout","time":"2019-10-03T07:33:55.127026666Z"}
{"log":"\u003c6\u003e 2019-10-03 07:34:00.457 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (03m:38s left).\n","stream":"stdout","time":"2019-10-03T07:34:00.459392094Z"}
{"log":"\u003c6\u003e 2019-10-03 07:34:06.053 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (03m:32s left).\n","stream":"stdout","time":"2019-10-03T07:34:06.060472288Z"}
{"log":"\u003c6\u003e 2019-10-03 07:34:11.454 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (03m:27s left).\n","stream":"stdout","time":"2019-10-03T07:34:11.456005908Z"}
{"log":"\u003c6\u003e 2019-10-03 07:34:16.798 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (03m:21s left).\n","stream":"stdout","time":"2019-10-03T07:34:16.799974615Z"}
{"log":"\u003c6\u003e 2019-10-03 07:34:22.101 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (03m:16s left).\n","stream":"stdout","time":"2019-10-03T07:34:22.103386965Z"}
{"log":"\u003c6\u003e 2019-10-03 07:34:27.377 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (03m:11s left).\n","stream":"stdout","time":"2019-10-03T07:34:27.379510773Z"}
{"log":"\u003c6\u003e 2019-10-03 07:34:32.937 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (03m:05s left).\n","stream":"stdout","time":"2019-10-03T07:34:32.94025275Z"}
{"log":"\u003c6\u003e 2019-10-03 07:34:38.339 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (03m:00s left).\n","stream":"stdout","time":"2019-10-03T07:34:38.340887558Z"}
{"log":"\u003c6\u003e 2019-10-03 07:34:43.696 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (02m:54s left).\n","stream":"stdout","time":"2019-10-03T07:34:43.697876202Z"}
{"log":"\u003c6\u003e 2019-10-03 07:34:49.359 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (02m:49s left).\n","stream":"stdout","time":"2019-10-03T07:34:49.363227411Z"}
{"log":"\u003c6\u003e 2019-10-03 07:34:54.794 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (02m:43s left).\n","stream":"stdout","time":"2019-10-03T07:34:54.796419667Z"}
{"log":"\u003c6\u003e 2019-10-03 07:35:00.155 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (02m:38s left).\n","stream":"stdout","time":"2019-10-03T07:35:00.15706879Z"}
{"log":"\u003c6\u003e 2019-10-03 07:35:06.049 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (02m:32s left).\n","stream":"stdout","time":"2019-10-03T07:35:06.05797886Z"}
{"log":"\u003c6\u003e 2019-10-03 07:35:11.714 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (02m:26s left).\n","stream":"stdout","time":"2019-10-03T07:35:11.716550004Z"}
{"log":"\u003c6\u003e 2019-10-03 07:35:17.508 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (02m:21s left).\n","stream":"stdout","time":"2019-10-03T07:35:17.515895473Z"}
{"log":"\u003c6\u003e 2019-10-03 07:35:22.993 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (02m:15s left).\n","stream":"stdout","time":"2019-10-03T07:35:22.996064271Z"}
{"log":"\u003c6\u003e 2019-10-03 07:35:28.366 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (02m:10s left).\n","stream":"stdout","time":"2019-10-03T07:35:28.36832547Z"}
{"log":"\u003c6\u003e 2019-10-03 07:35:34.154 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (02m:04s left).\n","stream":"stdout","time":"2019-10-03T07:35:34.159475921Z"}
{"log":"\u003c6\u003e 2019-10-03 07:35:39.536 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (01m:58s left).\n","stream":"stdout","time":"2019-10-03T07:35:39.537970889Z"}
{"log":"\u003c6\u003e 2019-10-03 07:35:45.187 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (01m:53s left).\n","stream":"stdout","time":"2019-10-03T07:35:45.19453427Z"}
{"log":"\u003c6\u003e 2019-10-03 07:35:50.979 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (01m:47s left).\n","stream":"stdout","time":"2019-10-03T07:35:50.981427657Z"}
{"log":"\u003c6\u003e 2019-10-03 07:35:56.315 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (01m:42s left).\n","stream":"stdout","time":"2019-10-03T07:35:56.31675844Z"}
{"log":"\u003c6\u003e 2019-10-03 07:36:01.889 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (01m:36s left).\n","stream":"stdout","time":"2019-10-03T07:36:01.89775679Z"}
{"log":"\u003c6\u003e 2019-10-03 07:36:07.283 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (01m:31s left).\n","stream":"stdout","time":"2019-10-03T07:36:07.285315923Z"}
{"log":"\u003c6\u003e 2019-10-03 07:36:12.626 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (01m:25s left).\n","stream":"stdout","time":"2019-10-03T07:36:12.629821028Z"}
{"log":"\u003c6\u003e 2019-10-03 07:36:18.426 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (01m:20s left).\n","stream":"stdout","time":"2019-10-03T07:36:18.442493622Z"}
{"log":"\u003c6\u003e 2019-10-03 07:36:23.840 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (01m:14s left).\n","stream":"stdout","time":"2019-10-03T07:36:23.842293688Z"}
{"log":"\u003c6\u003e 2019-10-03 07:36:29.156 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (01m:09s left).\n","stream":"stdout","time":"2019-10-03T07:36:29.157915879Z"}
{"log":"\u003c6\u003e 2019-10-03 07:36:34.644 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (01m:03s left).\n","stream":"stdout","time":"2019-10-03T07:36:34.645358478Z"}
{"log":"\u003c6\u003e 2019-10-03 07:36:40.013 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (58s left).\n","stream":"stdout","time":"2019-10-03T07:36:40.015666151Z"}
{"log":"\u003c6\u003e 2019-10-03 07:36:45.781 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (52s left).\n","stream":"stdout","time":"2019-10-03T07:36:45.794774733Z"}
{"log":"\u003c6\u003e 2019-10-03 07:36:51.235 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (47s left).\n","stream":"stdout","time":"2019-10-03T07:36:51.237282909Z"}
{"log":"\u003c6\u003e 2019-10-03 07:36:56.610 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (41s left).\n","stream":"stdout","time":"2019-10-03T07:36:56.611794154Z"}
{"log":"\u003c6\u003e 2019-10-03 07:37:02.204 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (36s left).\n","stream":"stdout","time":"2019-10-03T07:37:02.208275129Z"}
{"log":"\u003c6\u003e 2019-10-03 07:37:07.635 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (30s left).\n","stream":"stdout","time":"2019-10-03T07:37:07.637541819Z"}
{"log":"\u003c6\u003e 2019-10-03 07:37:13.005 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (25s left).\n","stream":"stdout","time":"2019-10-03T07:37:13.006146575Z"}
{"log":"\u003c6\u003e 2019-10-03 07:37:18.534 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (19s left).\n","stream":"stdout","time":"2019-10-03T07:37:18.540197664Z"}
{"log":"\u003c6\u003e 2019-10-03 07:37:23.908 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (14s left).\n","stream":"stdout","time":"2019-10-03T07:37:23.909476591Z"}
{"log":"\u003c6\u003e 2019-10-03 07:37:29.233 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (09s left).\n","stream":"stdout","time":"2019-10-03T07:37:29.234271532Z"}
{"log":"\u003c6\u003e 2019-10-03 07:37:34.968 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (03s left).\n","stream":"stdout","time":"2019-10-03T07:37:34.969883282Z"}
{"log":"\u003c6\u003e 2019-10-03 07:37:40.360 +00:00 [INF] - Plan execution started for deployment 15\n","stream":"stdout","time":"2019-10-03T07:37:40.363034285Z"}
{"log":"\u003c6\u003e 2019-10-03 07:37:40.364 +00:00 [INF] - Executing command: \"Command Group: (\\n  [Stop module edgeHub]\\n  [Start module edgeHub]\\n  [Saving edgeHub to store]\\n)\"\n","stream":"stdout","time":"2019-10-03T07:37:40.366686847Z"}
{"log":"\u003c6\u003e 2019-10-03 07:37:40.368 +00:00 [INF] - Executing command: \"Stop module edgeHub\"\n","stream":"stdout","time":"2019-10-03T07:37:40.369759695Z"}
{"log":"\u003c6\u003e 2019-10-03 07:37:40.438 +00:00 [INF] - Executing command: \"Start module edgeHub\"\n","stream":"stdout","time":"2019-10-03T07:37:40.440026081Z"}
{"log":"\u003c6\u003e 2019-10-03 07:37:48.399 +00:00 [INF] - Executing command: \"Saving edgeHub to store\"\n","stream":"stdout","time":"2019-10-03T07:37:48.400970054Z"}
{"log":"\u003c6\u003e 2019-10-03 07:37:48.443 +00:00 [INF] - Plan execution ended for deployment 15\n","stream":"stdout","time":"2019-10-03T07:37:48.447767566Z"}
{"log":"\u003c6\u003e 2019-10-03 07:37:56.150 +00:00 [INF] - Updated reported properties\n","stream":"stdout","time":"2019-10-03T07:37:56.151733039Z"}
{"log":"\u003c6\u003e 2019-10-03 07:39:00.472 +00:00 [INF] - Module 'edgeHub' scheduled to restart after 05m:00s (04m:53s left).\n","stream":"stdout","time":"2019-10-03T07:39:00.481865217Z"}
{"log":"\u003c6\u003e 2019-10-03 07:39:00.789 +00:00 [INF] - Updated reported properties\n","stream":"stdout","time":"2019-10-03T07:39:00.790638007Z"}
edge-hub logs
{"log":"2019-10-03 06:00:06 +00:00 Starting Edge Hub\n","stream":"stdout","time":"2019-10-03T06:00:06.14081052Z"}
{"log":"2019-10-03 06:00:08.060 +00:00 Edge Hub Main()\n","stream":"stdout","time":"2019-10-03T06:00:08.18361Z"}
{"log":"\n","stream":"stderr","time":"2019-10-03T06:01:03.6346802Z"}
{"log":"Unhandled Exception: System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:01:04.8154524Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:01:04.81574404Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:01:04.81583752Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response) ---\u003e Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException: Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:01:04.8159058Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:01:04.81597112Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:01:04.81605032Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response\n","stream":"stderr","time":"2019-10-03T06:01:04.81610864Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 106\n","stream":"stderr","time":"2019-10-03T06:01:04.8161666Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 65\n","stream":"stderr","time":"2019-10-03T06:01:04.81622476Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 35\n","stream":"stderr","time":"2019-10-03T06:01:04.81628272Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 249\n","stream":"stderr","time":"2019-10-03T06:01:04.8163482Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 53\n","stream":"stderr","time":"2019-10-03T06:01:04.81646392Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 48\n","stream":"stderr","time":"2019-10-03T06:01:04.81653872Z"}
{"log":"   --- End of inner exception stack trace ---\n","stream":"stderr","time":"2019-10-03T06:01:04.81660304Z"}
{"log":"   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\n","stream":"stderr","time":"2019-10-03T06:01:04.81665828Z"}
{"log":"   at System.Threading.Tasks.Task`1.get_Result()\n","stream":"stderr","time":"2019-10-03T06:01:04.81671176Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 29\n","stream":"stderr","time":"2019-10-03T06:01:04.81947144Z"}
{"log":"2019-10-03 06:01:23 +00:00 Starting Edge Hub\n","stream":"stdout","time":"2019-10-03T06:01:23.9675744Z"}
{"log":"2019-10-03 06:01:27.709 +00:00 Edge Hub Main()\n","stream":"stdout","time":"2019-10-03T06:01:28.16560116Z"}
{"log":"\n","stream":"stderr","time":"2019-10-03T06:02:12.732636Z"}
{"log":"Unhandled Exception: System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:02:13.90643976Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:02:13.90672912Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:02:13.906819Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response) ---\u003e Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException: Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:02:13.90688636Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:02:13.90914232Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:02:13.90923504Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response\n","stream":"stderr","time":"2019-10-03T06:02:13.90932388Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 106\n","stream":"stderr","time":"2019-10-03T06:02:13.90938796Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 65\n","stream":"stderr","time":"2019-10-03T06:02:13.90944848Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 35\n","stream":"stderr","time":"2019-10-03T06:02:13.90951252Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 249\n","stream":"stderr","time":"2019-10-03T06:02:13.90958176Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 53\n","stream":"stderr","time":"2019-10-03T06:02:13.90967828Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 48\n","stream":"stderr","time":"2019-10-03T06:02:13.90976204Z"}
{"log":"   --- End of inner exception stack trace ---\n","stream":"stderr","time":"2019-10-03T06:02:13.90983052Z"}
{"log":"   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\n","stream":"stderr","time":"2019-10-03T06:02:13.90988684Z"}
{"log":"   at System.Threading.Tasks.Task`1.get_Result()\n","stream":"stderr","time":"2019-10-03T06:02:13.90994164Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 29\n","stream":"stderr","time":"2019-10-03T06:02:13.90999628Z"}
{"log":"2019-10-03 06:02:38 +00:00 Starting Edge Hub\n","stream":"stdout","time":"2019-10-03T06:02:38.2840432Z"}
{"log":"2019-10-03 06:02:40.304 +00:00 Edge Hub Main()\n","stream":"stdout","time":"2019-10-03T06:02:40.50664192Z"}
{"log":"\n","stream":"stderr","time":"2019-10-03T06:03:25.0193566Z"}
{"log":"Unhandled Exception: System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:03:25.56529756Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:03:25.56566112Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:03:25.56575396Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response) ---\u003e Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException: Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:03:25.56581488Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:03:25.56587484Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:03:25.565928Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response\n","stream":"stderr","time":"2019-10-03T06:03:25.56598028Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 106\n","stream":"stderr","time":"2019-10-03T06:03:25.56603492Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 65\n","stream":"stderr","time":"2019-10-03T06:03:25.5661048Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 35\n","stream":"stderr","time":"2019-10-03T06:03:25.56616352Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 249\n","stream":"stderr","time":"2019-10-03T06:03:25.56622308Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 53\n","stream":"stderr","time":"2019-10-03T06:03:25.56629828Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 48\n","stream":"stderr","time":"2019-10-03T06:03:25.56636868Z"}
{"log":"   --- End of inner exception stack trace ---\n","stream":"stderr","time":"2019-10-03T06:03:25.56644932Z"}
{"log":"   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\n","stream":"stderr","time":"2019-10-03T06:03:25.566504Z"}
{"log":"   at System.Threading.Tasks.Task`1.get_Result()\n","stream":"stderr","time":"2019-10-03T06:03:25.56656504Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 29\n","stream":"stderr","time":"2019-10-03T06:03:25.56856636Z"}
{"log":"2019-10-03 06:04:12 +00:00 Starting Edge Hub\n","stream":"stdout","time":"2019-10-03T06:04:12.220436221Z"}
{"log":"2019-10-03 06:04:15.754 +00:00 Edge Hub Main()\n","stream":"stdout","time":"2019-10-03T06:04:16.030334486Z"}
{"log":"\n","stream":"stderr","time":"2019-10-03T06:04:55.449438324Z"}
{"log":"Unhandled Exception: System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:04:56.001114298Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:04:56.001402407Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:04:56.001595839Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response) ---\u003e Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException: Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:04:56.001675956Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:04:56.001742274Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:04:56.001797831Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response\n","stream":"stderr","time":"2019-10-03T06:04:56.001851869Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 106\n","stream":"stderr","time":"2019-10-03T06:04:56.001907427Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 65\n","stream":"stderr","time":"2019-10-03T06:04:56.001974504Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 35\n","stream":"stderr","time":"2019-10-03T06:04:56.002033102Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 249\n","stream":"stderr","time":"2019-10-03T06:04:56.0020923Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 53\n","stream":"stderr","time":"2019-10-03T06:04:56.002166537Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 48\n","stream":"stderr","time":"2019-10-03T06:04:56.002237334Z"}
{"log":"   --- End of inner exception stack trace ---\n","stream":"stderr","time":"2019-10-03T06:04:56.002296292Z"}
{"log":"   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\n","stream":"stderr","time":"2019-10-03T06:04:56.00235049Z"}
{"log":"   at System.Threading.Tasks.Task`1.get_Result()\n","stream":"stderr","time":"2019-10-03T06:04:56.002404247Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 29\n","stream":"stderr","time":"2019-10-03T06:04:56.004143059Z"}
{"log":"2019-10-03 06:06:25 +00:00 Starting Edge Hub\n","stream":"stdout","time":"2019-10-03T06:06:25.194691139Z"}
{"log":"2019-10-03 06:06:27.230 +00:00 Edge Hub Main()\n","stream":"stdout","time":"2019-10-03T06:06:27.385029094Z"}
{"log":"\n","stream":"stderr","time":"2019-10-03T06:06:54.881198579Z"}
{"log":"Unhandled Exception: System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:06:55.470963029Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:06:55.471370634Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:06:55.471569837Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response) ---\u003e Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException: Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:06:55.473355981Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:06:55.473647985Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:06:55.473781227Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response\n","stream":"stderr","time":"2019-10-03T06:06:55.473942429Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 106\n","stream":"stderr","time":"2019-10-03T06:06:55.47405283Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 65\n","stream":"stderr","time":"2019-10-03T06:06:55.474160792Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 35\n","stream":"stderr","time":"2019-10-03T06:06:55.474289793Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 249\n","stream":"stderr","time":"2019-10-03T06:06:55.474402315Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 53\n","stream":"stderr","time":"2019-10-03T06:06:55.474533157Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 48\n","stream":"stderr","time":"2019-10-03T06:06:55.474700559Z"}
{"log":"   --- End of inner exception stack trace ---\n","stream":"stderr","time":"2019-10-03T06:06:55.47480512Z"}
{"log":"   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\n","stream":"stderr","time":"2019-10-03T06:06:55.474906162Z"}
{"log":"   at System.Threading.Tasks.Task`1.get_Result()\n","stream":"stderr","time":"2019-10-03T06:06:55.475005203Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 29\n","stream":"stderr","time":"2019-10-03T06:06:55.477391395Z"}
{"log":"2019-10-03 06:09:47 +00:00 Starting Edge Hub\n","stream":"stdout","time":"2019-10-03T06:09:47.301817638Z"}
{"log":"2019-10-03 06:09:49.860 +00:00 Edge Hub Main()\n","stream":"stdout","time":"2019-10-03T06:09:50.026210359Z"}
{"log":"\n","stream":"stderr","time":"2019-10-03T06:10:17.014563473Z"}
{"log":"Unhandled Exception: System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:10:17.503270098Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:10:17.503671903Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:10:17.503874705Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response) ---\u003e Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException: Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:10:17.504000507Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:10:17.504411991Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:10:17.504553033Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response\n","stream":"stderr","time":"2019-10-03T06:10:17.504725155Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 106\n","stream":"stderr","time":"2019-10-03T06:10:17.504832356Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 65\n","stream":"stderr","time":"2019-10-03T06:10:17.504941558Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 35\n","stream":"stderr","time":"2019-10-03T06:10:17.505068599Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 249\n","stream":"stderr","time":"2019-10-03T06:10:17.50518316Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 53\n","stream":"stderr","time":"2019-10-03T06:10:17.505320002Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 48\n","stream":"stderr","time":"2019-10-03T06:10:17.505494244Z"}
{"log":"   --- End of inner exception stack trace ---\n","stream":"stderr","time":"2019-10-03T06:10:17.505602605Z"}
{"log":"   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\n","stream":"stderr","time":"2019-10-03T06:10:17.505705606Z"}
{"log":"   at System.Threading.Tasks.Task`1.get_Result()\n","stream":"stderr","time":"2019-10-03T06:10:17.505806248Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 29\n","stream":"stderr","time":"2019-10-03T06:10:17.50777055Z"}
{"log":"2019-10-03 06:15:29 +00:00 Starting Edge Hub\n","stream":"stdout","time":"2019-10-03T06:15:29.346280474Z"}
{"log":"2019-10-03 06:15:31.386 +00:00 Edge Hub Main()\n","stream":"stdout","time":"2019-10-03T06:15:31.575159438Z"}
{"log":"\n","stream":"stderr","time":"2019-10-03T06:16:10.621454142Z"}
{"log":"Unhandled Exception: System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:16:11.204909415Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:16:11.205188001Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:16:11.205341274Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response) ---\u003e Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException: Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:16:11.205412511Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:16:11.205576783Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:16:11.205652179Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response\n","stream":"stderr","time":"2019-10-03T06:16:11.205708336Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 106\n","stream":"stderr","time":"2019-10-03T06:16:11.205763534Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 65\n","stream":"stderr","time":"2019-10-03T06:16:11.205820851Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 35\n","stream":"stderr","time":"2019-10-03T06:16:11.205877688Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 249\n","stream":"stderr","time":"2019-10-03T06:16:11.205945205Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 53\n","stream":"stderr","time":"2019-10-03T06:16:11.206013922Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 48\n","stream":"stderr","time":"2019-10-03T06:16:11.206089238Z"}
{"log":"   --- End of inner exception stack trace ---\n","stream":"stderr","time":"2019-10-03T06:16:11.206148035Z"}
{"log":"   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\n","stream":"stderr","time":"2019-10-03T06:16:11.206201753Z"}
{"log":"   at System.Threading.Tasks.Task`1.get_Result()\n","stream":"stderr","time":"2019-10-03T06:16:11.20625527Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 29\n","stream":"stderr","time":"2019-10-03T06:16:11.20833501Z"}
{"log":"2019-10-03 06:21:19 +00:00 Starting Edge Hub\n","stream":"stdout","time":"2019-10-03T06:21:19.291246959Z"}
{"log":"2019-10-03 06:21:22.646 +00:00 Edge Hub Main()\n","stream":"stdout","time":"2019-10-03T06:21:22.806102608Z"}
{"log":"\n","stream":"stderr","time":"2019-10-03T06:21:55.519971408Z"}
{"log":"Unhandled Exception: System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:21:56.088924227Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:21:56.089207222Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:21:56.0892955Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response) ---\u003e Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException: Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:21:56.089361939Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:21:56.089427298Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:21:56.089579735Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response\n","stream":"stderr","time":"2019-10-03T06:21:56.089646414Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 106\n","stream":"stderr","time":"2019-10-03T06:21:56.089705973Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 65\n","stream":"stderr","time":"2019-10-03T06:21:56.089766212Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 35\n","stream":"stderr","time":"2019-10-03T06:21:56.089828291Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 249\n","stream":"stderr","time":"2019-10-03T06:21:56.08989925Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 53\n","stream":"stderr","time":"2019-10-03T06:21:56.089968328Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 48\n","stream":"stderr","time":"2019-10-03T06:21:56.090038287Z"}
{"log":"   --- End of inner exception stack trace ---\n","stream":"stderr","time":"2019-10-03T06:21:56.090102286Z"}
{"log":"   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\n","stream":"stderr","time":"2019-10-03T06:21:56.090158125Z"}
{"log":"   at System.Threading.Tasks.Task`1.get_Result()\n","stream":"stderr","time":"2019-10-03T06:21:56.090211644Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 29\n","stream":"stderr","time":"2019-10-03T06:21:56.092282647Z"}
{"log":"2019-10-03 06:27:07 +00:00 Starting Edge Hub\n","stream":"stdout","time":"2019-10-03T06:27:07.582230563Z"}
{"log":"2019-10-03 06:27:10.713 +00:00 Edge Hub Main()\n","stream":"stdout","time":"2019-10-03T06:27:10.921837253Z"}
{"log":"\n","stream":"stderr","time":"2019-10-03T06:27:43.621379674Z"}
{"log":"Unhandled Exception: System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:27:44.279613968Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:27:44.279884924Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:27:44.279970842Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response) ---\u003e Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException: Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:27:44.280030681Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:27:44.28009316Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:27:44.280148919Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response\n","stream":"stderr","time":"2019-10-03T06:27:44.280202478Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 106\n","stream":"stderr","time":"2019-10-03T06:27:44.280352196Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 65\n","stream":"stderr","time":"2019-10-03T06:27:44.280427835Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 35\n","stream":"stderr","time":"2019-10-03T06:27:44.280489874Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 249\n","stream":"stderr","time":"2019-10-03T06:27:44.280557633Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 53\n","stream":"stderr","time":"2019-10-03T06:27:44.280630151Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 48\n","stream":"stderr","time":"2019-10-03T06:27:44.28070199Z"}
{"log":"   --- End of inner exception stack trace ---\n","stream":"stderr","time":"2019-10-03T06:27:44.280759989Z"}
{"log":"   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\n","stream":"stderr","time":"2019-10-03T06:27:44.280821748Z"}
{"log":"   at System.Threading.Tasks.Task`1.get_Result()\n","stream":"stderr","time":"2019-10-03T06:27:44.280876427Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 29\n","stream":"stderr","time":"2019-10-03T06:27:44.282776515Z"}
{"log":"2019-10-03 06:32:52 +00:00 Starting Edge Hub\n","stream":"stdout","time":"2019-10-03T06:32:52.731421023Z"}
{"log":"2019-10-03 06:32:57.048 +00:00 Edge Hub Main()\n","stream":"stdout","time":"2019-10-03T06:32:57.562876242Z"}
{"log":"\n","stream":"stderr","time":"2019-10-03T06:33:43.35469851Z"}
{"log":"Unhandled Exception: System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:33:44.919305972Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:33:44.919785849Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:33:44.921955276Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response) ---\u003e Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException: Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:33:44.922162115Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:33:44.922283754Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:33:44.922397473Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response\n","stream":"stderr","time":"2019-10-03T06:33:44.922505312Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 106\n","stream":"stderr","time":"2019-10-03T06:33:44.922626232Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 65\n","stream":"stderr","time":"2019-10-03T06:33:44.92288587Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 35\n","stream":"stderr","time":"2019-10-03T06:33:44.923022149Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 249\n","stream":"stderr","time":"2019-10-03T06:33:44.923176708Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 53\n","stream":"stderr","time":"2019-10-03T06:33:44.923310467Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 48\n","stream":"stderr","time":"2019-10-03T06:33:44.923439907Z"}
{"log":"   --- End of inner exception stack trace ---\n","stream":"stderr","time":"2019-10-03T06:33:44.923550946Z"}
{"log":"   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\n","stream":"stderr","time":"2019-10-03T06:33:44.927530801Z"}
{"log":"   at System.Threading.Tasks.Task`1.get_Result()\n","stream":"stderr","time":"2019-10-03T06:33:44.92779176Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 29\n","stream":"stderr","time":"2019-10-03T06:33:44.927919839Z"}
{"log":"2019-10-03 06:38:57 +00:00 Starting Edge Hub\n","stream":"stdout","time":"2019-10-03T06:38:57.406992883Z"}
{"log":"2019-10-03 06:38:59.896 +00:00 Edge Hub Main()\n","stream":"stdout","time":"2019-10-03T06:39:00.067070799Z"}
{"log":"\n","stream":"stderr","time":"2019-10-03T06:39:57.870502006Z"}
{"log":"Unhandled Exception: System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:39:58.508492692Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:39:58.508915968Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:39:58.509155565Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response) ---\u003e Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException: Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:39:58.509284124Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:39:58.509447922Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:39:58.509555641Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response\n","stream":"stderr","time":"2019-10-03T06:39:58.50966224Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 106\n","stream":"stderr","time":"2019-10-03T06:39:58.509768719Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 65\n","stream":"stderr","time":"2019-10-03T06:39:58.510139755Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 35\n","stream":"stderr","time":"2019-10-03T06:39:58.510280753Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 249\n","stream":"stderr","time":"2019-10-03T06:39:58.510456791Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 53\n","stream":"stderr","time":"2019-10-03T06:39:58.510679509Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 48\n","stream":"stderr","time":"2019-10-03T06:39:58.510815547Z"}
{"log":"   --- End of inner exception stack trace ---\n","stream":"stderr","time":"2019-10-03T06:39:58.510929186Z"}
{"log":"   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\n","stream":"stderr","time":"2019-10-03T06:39:58.511035865Z"}
{"log":"   at System.Threading.Tasks.Task`1.get_Result()\n","stream":"stderr","time":"2019-10-03T06:39:58.511143584Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 29\n","stream":"stderr","time":"2019-10-03T06:39:58.51433347Z"}
{"log":"2019-10-03 06:45:12 +00:00 Starting Edge Hub\n","stream":"stdout","time":"2019-10-03T06:45:12.14005415Z"}
{"log":"2019-10-03 06:45:14.329 +00:00 Edge Hub Main()\n","stream":"stdout","time":"2019-10-03T06:45:14.534221686Z"}
{"log":"\n","stream":"stderr","time":"2019-10-03T06:46:00.810390328Z"}
{"log":"Unhandled Exception: System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:46:01.337999636Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:46:01.338345951Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:46:01.33843431Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response) ---\u003e Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException: Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:46:01.338527789Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:46:01.338593508Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:46:01.338648267Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response\n","stream":"stderr","time":"2019-10-03T06:46:01.338702707Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 106\n","stream":"stderr","time":"2019-10-03T06:46:01.338759986Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 65\n","stream":"stderr","time":"2019-10-03T06:46:01.338817905Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 35\n","stream":"stderr","time":"2019-10-03T06:46:01.338982263Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 249\n","stream":"stderr","time":"2019-10-03T06:46:01.339059862Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 53\n","stream":"stderr","time":"2019-10-03T06:46:01.339154501Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 48\n","stream":"stderr","time":"2019-10-03T06:46:01.3392355Z"}
{"log":"   --- End of inner exception stack trace ---\n","stream":"stderr","time":"2019-10-03T06:46:01.339296499Z"}
{"log":"   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\n","stream":"stderr","time":"2019-10-03T06:46:01.339350779Z"}
{"log":"   at System.Threading.Tasks.Task`1.get_Result()\n","stream":"stderr","time":"2019-10-03T06:46:01.339404178Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 29\n","stream":"stderr","time":"2019-10-03T06:46:01.341286075Z"}
{"log":"2019-10-03 06:55:36 +00:00 Starting Edge Hub\n","stream":"stdout","time":"2019-10-03T06:55:36.188261453Z"}
{"log":"2019-10-03 06:55:38.399 +00:00 Edge Hub Main()\n","stream":"stdout","time":"2019-10-03T06:55:38.609871022Z"}
{"log":"\n","stream":"stderr","time":"2019-10-03T06:56:20.162852791Z"}
{"log":"Unhandled Exception: System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:56:21.360672241Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:56:21.361517635Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:56:21.361797312Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response) ---\u003e Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException: Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T06:56:21.361924111Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T06:56:21.36206655Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T06:56:21.362178349Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response\n","stream":"stderr","time":"2019-10-03T06:56:21.362284989Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 106\n","stream":"stderr","time":"2019-10-03T06:56:21.362388508Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 65\n","stream":"stderr","time":"2019-10-03T06:56:21.362492467Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 35\n","stream":"stderr","time":"2019-10-03T06:56:21.365339844Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 249\n","stream":"stderr","time":"2019-10-03T06:56:21.365723081Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 53\n","stream":"stderr","time":"2019-10-03T06:56:21.366044758Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 48\n","stream":"stderr","time":"2019-10-03T06:56:21.366263517Z"}
{"log":"   --- End of inner exception stack trace ---\n","stream":"stderr","time":"2019-10-03T06:56:21.366381556Z"}
{"log":"   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\n","stream":"stderr","time":"2019-10-03T06:56:21.366488155Z"}
{"log":"   at System.Threading.Tasks.Task`1.get_Result()\n","stream":"stderr","time":"2019-10-03T06:56:21.366590114Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 29\n","stream":"stderr","time":"2019-10-03T06:56:21.369743449Z"}
{"log":"2019-10-03 07:01:31 +00:00 Starting Edge Hub\n","stream":"stdout","time":"2019-10-03T07:01:31.092511462Z"}
{"log":"2019-10-03 07:01:33.333 +00:00 Edge Hub Main()\n","stream":"stdout","time":"2019-10-03T07:01:33.522215862Z"}
{"log":"\n","stream":"stderr","time":"2019-10-03T07:02:09.381619322Z"}
{"log":"Unhandled Exception: System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T07:02:10.026372802Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T07:02:10.02666648Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T07:02:10.026761679Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response) ---\u003e Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException: Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T07:02:10.026828079Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T07:02:10.026893118Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T07:02:10.027376355Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response\n","stream":"stderr","time":"2019-10-03T07:02:10.027505194Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 106\n","stream":"stderr","time":"2019-10-03T07:02:10.027669473Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 65\n","stream":"stderr","time":"2019-10-03T07:02:10.027776792Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 35\n","stream":"stderr","time":"2019-10-03T07:02:10.027882191Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 249\n","stream":"stderr","time":"2019-10-03T07:02:10.028194069Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 53\n","stream":"stderr","time":"2019-10-03T07:02:10.028402068Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 48\n","stream":"stderr","time":"2019-10-03T07:02:10.028533987Z"}
{"log":"   --- End of inner exception stack trace ---\n","stream":"stderr","time":"2019-10-03T07:02:10.028653546Z"}
{"log":"   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\n","stream":"stderr","time":"2019-10-03T07:02:10.028758185Z"}
{"log":"   at System.Threading.Tasks.Task`1.get_Result()\n","stream":"stderr","time":"2019-10-03T07:02:10.028858865Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 29\n","stream":"stderr","time":"2019-10-03T07:02:10.03239312Z"}
{"log":"2019-10-03 07:07:22 +00:00 Starting Edge Hub\n","stream":"stdout","time":"2019-10-03T07:07:22.211629316Z"}
{"log":"2019-10-03 07:07:24.837 +00:00 Edge Hub Main()\n","stream":"stdout","time":"2019-10-03T07:07:25.075291216Z"}
{"log":"\n","stream":"stderr","time":"2019-10-03T07:08:13.586521338Z"}
{"log":"Unhandled Exception: System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T07:08:14.697175224Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T07:08:14.697452459Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T07:08:14.697540177Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response) ---\u003e Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException: Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T07:08:14.697604056Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T07:08:14.697668335Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T07:08:14.697726654Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response\n","stream":"stderr","time":"2019-10-03T07:08:14.697807812Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 106\n","stream":"stderr","time":"2019-10-03T07:08:14.697869011Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 65\n","stream":"stderr","time":"2019-10-03T07:08:14.69792801Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 35\n","stream":"stderr","time":"2019-10-03T07:08:14.697987169Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 249\n","stream":"stderr","time":"2019-10-03T07:08:14.698124926Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 53\n","stream":"stderr","time":"2019-10-03T07:08:14.698241604Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 48\n","stream":"stderr","time":"2019-10-03T07:08:14.698316603Z"}
{"log":"   --- End of inner exception stack trace ---\n","stream":"stderr","time":"2019-10-03T07:08:14.698385042Z"}
{"log":"   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\n","stream":"stderr","time":"2019-10-03T07:08:14.69844116Z"}
{"log":"   at System.Threading.Tasks.Task`1.get_Result()\n","stream":"stderr","time":"2019-10-03T07:08:14.698495879Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 29\n","stream":"stderr","time":"2019-10-03T07:08:14.700766758Z"}
{"log":"2019-10-03 07:13:23 +00:00 Starting Edge Hub\n","stream":"stdout","time":"2019-10-03T07:13:23.766027338Z"}
{"log":"2019-10-03 07:13:27.992 +00:00 Edge Hub Main()\n","stream":"stdout","time":"2019-10-03T07:13:28.669577289Z"}
{"log":"\n","stream":"stderr","time":"2019-10-03T07:14:17.016378749Z"}
{"log":"Unhandled Exception: System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T07:14:18.927773081Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T07:14:18.928103957Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T07:14:18.928190755Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response) ---\u003e Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException: Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T07:14:18.928251995Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T07:14:18.928313074Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T07:14:18.928368913Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response\n","stream":"stderr","time":"2019-10-03T07:14:18.928422193Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 106\n","stream":"stderr","time":"2019-10-03T07:14:18.928489112Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 65\n","stream":"stderr","time":"2019-10-03T07:14:18.928549871Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 35\n","stream":"stderr","time":"2019-10-03T07:14:18.92860759Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 249\n","stream":"stderr","time":"2019-10-03T07:14:18.92866695Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 53\n","stream":"stderr","time":"2019-10-03T07:14:18.928806828Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 48\n","stream":"stderr","time":"2019-10-03T07:14:18.928887707Z"}
{"log":"   --- End of inner exception stack trace ---\n","stream":"stderr","time":"2019-10-03T07:14:18.928948466Z"}
{"log":"   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\n","stream":"stderr","time":"2019-10-03T07:14:18.929010185Z"}
{"log":"   at System.Threading.Tasks.Task`1.get_Result()\n","stream":"stderr","time":"2019-10-03T07:14:18.929064865Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 29\n","stream":"stderr","time":"2019-10-03T07:14:18.929118224Z"}
{"log":"2019-10-03 07:19:33 +00:00 Starting Edge Hub\n","stream":"stdout","time":"2019-10-03T07:19:33.877770741Z"}
{"log":"2019-10-03 07:19:36.482 +00:00 Edge Hub Main()\n","stream":"stdout","time":"2019-10-03T07:19:36.763051579Z"}
{"log":"\n","stream":"stderr","time":"2019-10-03T07:20:25.98422083Z"}
{"log":"Unhandled Exception: System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T07:20:26.821375296Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T07:20:26.821671852Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T07:20:26.821786731Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response) ---\u003e Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException: Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T07:20:26.82185113Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T07:20:26.821913169Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T07:20:26.821968088Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response\n","stream":"stderr","time":"2019-10-03T07:20:26.822022087Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 106\n","stream":"stderr","time":"2019-10-03T07:20:26.822078526Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 65\n","stream":"stderr","time":"2019-10-03T07:20:26.822148166Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 35\n","stream":"stderr","time":"2019-10-03T07:20:26.822209205Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 249\n","stream":"stderr","time":"2019-10-03T07:20:26.822268844Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 53\n","stream":"stderr","time":"2019-10-03T07:20:26.822429162Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 48\n","stream":"stderr","time":"2019-10-03T07:20:26.822512561Z"}
{"log":"   --- End of inner exception stack trace ---\n","stream":"stderr","time":"2019-10-03T07:20:26.8225732Z"}
{"log":"   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\n","stream":"stderr","time":"2019-10-03T07:20:26.822627719Z"}
{"log":"   at System.Threading.Tasks.Task`1.get_Result()\n","stream":"stderr","time":"2019-10-03T07:20:26.822690718Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 29\n","stream":"stderr","time":"2019-10-03T07:20:26.825043606Z"}
{"log":"2019-10-03 07:25:36 +00:00 Starting Edge Hub\n","stream":"stdout","time":"2019-10-03T07:25:36.617821203Z"}
{"log":"2019-10-03 07:25:39.428 +00:00 Edge Hub Main()\n","stream":"stdout","time":"2019-10-03T07:25:39.697185786Z"}
{"log":"\n","stream":"stderr","time":"2019-10-03T07:26:27.817369884Z"}
{"log":"Unhandled Exception: System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T07:26:28.649962572Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T07:26:28.650277849Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T07:26:28.650388607Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response) ---\u003e Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException: Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T07:26:28.650455366Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T07:26:28.650518645Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T07:26:28.650573525Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response\n","stream":"stderr","time":"2019-10-03T07:26:28.650626804Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 106\n","stream":"stderr","time":"2019-10-03T07:26:28.650682043Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 65\n","stream":"stderr","time":"2019-10-03T07:26:28.650747483Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 35\n","stream":"stderr","time":"2019-10-03T07:26:28.650806322Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 249\n","stream":"stderr","time":"2019-10-03T07:26:28.650865281Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 53\n","stream":"stderr","time":"2019-10-03T07:26:28.651040759Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 48\n","stream":"stderr","time":"2019-10-03T07:26:28.651124878Z"}
{"log":"   --- End of inner exception stack trace ---\n","stream":"stderr","time":"2019-10-03T07:26:28.651184237Z"}
{"log":"   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\n","stream":"stderr","time":"2019-10-03T07:26:28.651238916Z"}
{"log":"   at System.Threading.Tasks.Task`1.get_Result()\n","stream":"stderr","time":"2019-10-03T07:26:28.651292796Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 29\n","stream":"stderr","time":"2019-10-03T07:26:28.653284131Z"}
{"log":"2019-10-03 07:31:41 +00:00 Starting Edge Hub\n","stream":"stdout","time":"2019-10-03T07:31:41.145405925Z"}
{"log":"2019-10-03 07:31:44.000 +00:00 Edge Hub Main()\n","stream":"stdout","time":"2019-10-03T07:31:44.289200363Z"}
{"log":"\n","stream":"stderr","time":"2019-10-03T07:32:37.573422085Z"}
{"log":"Unhandled Exception: System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T07:32:38.411801453Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T07:32:38.41208165Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T07:32:38.412164449Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response) ---\u003e Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException: Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T07:32:38.412232688Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T07:32:38.412320687Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T07:32:38.412408406Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response\n","stream":"stderr","time":"2019-10-03T07:32:38.412468085Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 106\n","stream":"stderr","time":"2019-10-03T07:32:38.412525725Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 65\n","stream":"stderr","time":"2019-10-03T07:32:38.412584764Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 35\n","stream":"stderr","time":"2019-10-03T07:32:38.412651603Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 249\n","stream":"stderr","time":"2019-10-03T07:32:38.412712682Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 53\n","stream":"stderr","time":"2019-10-03T07:32:38.412778962Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 48\n","stream":"stderr","time":"2019-10-03T07:32:38.413445154Z"}
{"log":"   --- End of inner exception stack trace ---\n","stream":"stderr","time":"2019-10-03T07:32:38.413560033Z"}
{"log":"   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\n","stream":"stderr","time":"2019-10-03T07:32:38.413620432Z"}
{"log":"   at System.Threading.Tasks.Task`1.get_Result()\n","stream":"stderr","time":"2019-10-03T07:32:38.413675872Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 29\n","stream":"stderr","time":"2019-10-03T07:32:38.415749328Z"}
{"log":"2019-10-03 07:37:47 +00:00 Starting Edge Hub\n","stream":"stdout","time":"2019-10-03T07:37:47.878917617Z"}
{"log":"2019-10-03 07:37:53.269 +00:00 Edge Hub Main()\n","stream":"stdout","time":"2019-10-03T07:37:53.526893645Z"}
{"log":"\n","stream":"stderr","time":"2019-10-03T07:38:51.374436568Z"}
{"log":"Unhandled Exception: System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T07:38:53.570616903Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T07:38:53.570871341Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T07:38:53.57095398Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response) ---\u003e Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException: Error calling CreateServerCertificateAsync: Could not get server cert\n","stream":"stderr","time":"2019-10-03T07:38:53.571019099Z"}
{"log":"\u0009caused by: An error occurred creating the certificate\n","stream":"stderr","time":"2019-10-03T07:38:53.571104618Z"}
{"log":"\u0009caused by: HSM failure\n","stream":"stderr","time":"2019-10-03T07:38:53.571211217Z"}
{"log":"\u0009caused by: HSM API returned an invalid null response\n","stream":"stderr","time":"2019-10-03T07:38:53.571271737Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 106\n","stream":"stderr","time":"2019-10-03T07:38:53.571330896Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 65\n","stream":"stderr","time":"2019-10-03T07:38:53.571390135Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 35\n","stream":"stderr","time":"2019-10-03T07:38:53.571455935Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 249\n","stream":"stderr","time":"2019-10-03T07:38:53.571516654Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 53\n","stream":"stderr","time":"2019-10-03T07:38:53.571584573Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 48\n","stream":"stderr","time":"2019-10-03T07:38:53.571725692Z"}
{"log":"   --- End of inner exception stack trace ---\n","stream":"stderr","time":"2019-10-03T07:38:53.571792771Z"}
{"log":"   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\n","stream":"stderr","time":"2019-10-03T07:38:53.571848051Z"}
{"log":"   at System.Threading.Tasks.Task`1.get_Result()\n","stream":"stderr","time":"2019-10-03T07:38:53.57190285Z"}
{"log":"   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 29\n","stream":"stderr","time":"2019-10-03T07:38:53.57195593Z"}

Additional Information

  • Unexpected behaviour appeared suprisingly - no changes in code of custom modules nor in IoT Edge runtime and Docker/Moby. Everything worked fine for tens of hours and crashed completely unexpectingly.
  • Moxa has GSM/LTE module plugged-in, due to that it sometimes loses Internet connection and tryies to restart it.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 25 (25 by maintainers)

Most upvoted comments

Hello @kajakIYD,

Sorry, I have not been monitoring closed issues, and this didn’t pass my email filters. To my knowledge, there has been not work done to handle this case. We have still been operating under the assumption that the OS will have the correct time set when iotedged starts. There may be another systemd target that may suit your needs.