iotedge: IoTedge release 1.0.10 custom certificates edgeAgent Failes

Expected Behavior

We are upgrading the IoT edge runtime on some of our devices from 1.0.9.5 to 1.0.10 and everything should work like before where the device can onboard through DPS (with certificates) and running with production certificates

Current Behavior

Currently edgeAgent is not starting up because of an Hsm failure. We use custom production certificates which are valid according to iotedge check. DPS seems to work as expected. Downgrading to version 1.0.9.5 or using the self generated test certificates by Iot Edge solves the problem.

Steps to Reproduce

Provide a detailed set of steps to reproduce the bug.

  1. Install lasted version of IoT edge on Raspbian (sudo install -y iotedge)
  2. Install production certificates
  3. Edit config.yaml to use the production certificates
  4. Restart iotedge

Context (Environment)

Output of iotedge check

Click here
Configuration checks
--------------------
√ config.yaml is well-formed - OK
√ config.yaml has well-formed connection string - OK
√ container engine is installed and functional - OK
√ config.yaml has correct hostname - OK
√ config.yaml has correct URIs for daemon mgmt endpoint - OK
√ latest security daemon - OK
√ host time is close to real time - OK
√ container time is close to host time - OK
√ DNS server - OK
√ production readiness: identity certificates expiry - OK
√ production readiness: certificates - OK
√ production readiness: container engine - OK
√ production readiness: logs policy - OK
‼ production readiness: Edge Agent's storage directory is persisted on the host filesystem - Warning
    The edgeAgent module is not configured to persist its /tmp/edgeAgent directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.
× production readiness: Edge Hub's storage directory is persisted on the host filesystem - Error
    Could not check current state of edgeHub container

Connectivity checks
-------------------
√ host can connect to and perform TLS handshake with DPS endpoint - OK
√ host can connect to and perform TLS handshake with IoT Hub AMQP port - OK
√ host can connect to and perform TLS handshake with IoT Hub HTTPS / WebSockets port - OK
√ host can connect to and perform TLS handshake with IoT Hub MQTT port - OK
√ container on the default network can connect to IoT Hub AMQP port - OK
√ container on the default network can connect to IoT Hub HTTPS / WebSockets port - OK
√ container on the default network can connect to IoT Hub MQTT port - OK
√ container on the IoT Edge module network can connect to IoT Hub AMQP port - OK
√ container on the IoT Edge module network can connect to IoT Hub HTTPS / WebSockets port - OK
√ container on the IoT Edge module network can connect to IoT Hub MQTT port - OK

23 check(s) succeeded.
1 check(s) raised warnings. Re-run with --verbose for more details.
1 check(s) raised errors. Re-run with --verbose for more details.

Device Information

  • Host OS: Raspbian stretch
  • Architecture: arm64
  • Container OS: Linux containers

Runtime Versions

  • iotedged: 1.0.10-1
  • Edge Agent: 1.0.10
  • Edge Hub: never deployed
  • Docker/Moby : moby-engine (3.0.7)

Logs

sudo iotedge list

NAME             STATUS           DESCRIPTION                CONFIG
edgeAgent        failed           Failed (1) 43 seconds ago  mcr.microsoft.com/azureiotedge-agent:1.0
iotedged logs

-- Logs begin at Thu 2020-10-22 11:01:35 BST. --
Oct 22 11:07:29 dg-p061701-200917-009 iotedged[766]: 2020-10-22T10:07:29Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hs                              m-c/src/certificate_info.c:parse_cer<3>2020-10-22T10:07:29Z [ERR!] - Internal server error: Could not get trust bundle
Oct 22 11:07:29 dg-p061701-200917-009 iotedged[766]:         caused by: An error occurred getting the certificate
Oct 22 11:07:29 dg-p061701-200917-009 iotedged[766]:         caused by: HSM failure
Oct 22 11:07:29 dg-p061701-200917-009 iotedged[766]:         caused by: HSM API returned an invalid null response
Oct 22 11:07:29 dg-p061701-200917-009 iotedged[766]: 2020-10-22T10:07:29Z [INFO] - [work] - - - [2020-10-22 10:07:29.63                              7351901 UTC] "GET /trust-bundle?api-version=2019-01-30 HTTP/1.1" 500 Internal Server Error 178 "-" "-" auth_id(-)
Oct 22 11:07:32 dg-p061701-200917-009 iotedged[766]: 2020-10-22T10:07:32Z [ERR!] - Internal server error: Could not get                               trust bundle
Oct 22 11:07:32 dg-p061701-200917-009 iotedged[766]:         caused by: An error occurred getting the certificate
Oct 22 11:07:32 dg-p061701-200917-009 iotedged[766]:         caused by: HSM failure
Oct 22 11:07:32 dg-p061701-200917-009 iotedged[766]:         caused by: HSM API returned an invalid null response
Oct 22 11:07:32 dg-p061701-200917-009 iotedged[766]: 2020-10-22T10:07:32Z [INFO] - [work] - - - [2020-10-22 10:07:32.67                              9744492 UTC] "GET /trust-bundle?api-version=2019-01-30 HTTP/1.1" 500 Internal Server Error 178 "-" "-" auth_id(-)

-- Logs begin at Thu 2020-10-22 11:01:35 BST. --
Oct 22 11:11:12 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:12Z [INFO] - Configuring the Device private key using "/etc/iotedge/certs/Device_CA.key.pem".
Oct 22 11:11:12 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:12Z [INFO] - Configuring the trusted CA certificates using "/etc/iotedge/certs/Root_CA.pem".
Oct 22 11:11:12 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:12Z [INFO] - Finished configuring provisioning environment variables and certificates.
Oct 22 11:11:12 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:12Z [INFO] - Initializing hsm...
Oct 22 11:11:12 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:12Z [INFO] - Finished initializing hsm.
Oct 22 11:11:12 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:12Z [INFO] - Initializing hsm X509 interface...
Oct 22 11:11:12 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:12Z [INFO] - Finished initializing hsm X509 interface...
Oct 22 11:11:12 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:12Z [INFO] - Provisioning edge device...
Oct 22 11:11:12 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:12Z [INFO] - Starting provisioning edge device via X509 provisioning...
Oct 22 11:11:12 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:12Z [INFO] - Starting DPS registration with scope_id "0ne0014273D", registration_id "DG-P061701-200917-009"
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - DPS registration assigned device "DG-P061701-200917-009" in hub "vd-prd-iothub.azure-devices.net"
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Successful DPS provisioning.
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Finished provisioning edge device.
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Initializing the module runtime...
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Initializing module runtime...
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Using runtime network id azure-iot-edge
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Successfully initialized module runtime
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Finished initializing the module runtime.
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Stopping all modules...
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Stopping module edgeAgent...
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [WARN] - Could not stop module edgeAgent
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [WARN] -         caused by: Target of operation already in this state
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Finished stopping modules.
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Reprovisioning status InitialAssignment will trigger reconfiguration of modules.
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Removing module edgeAgent...
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Successfully removed module edgeAgent
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Detecting if configuration file has changed...
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - No change to configuration file detected.
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Edge issuer CA expiration date: 2025-10-21T09:50:47Z
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Obtaining workload CA succeeded.
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Starting management API...
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Starting workload API...
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Starting watchdog with 60 second frequency...
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Listening on fd://iotedge.mgmt.socket/ with 1 thread for management API.
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Listening on fd://iotedge.socket/ with 1 thread for workload API.
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Checking edge runtime status
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Creating and starting edge runtime module edgeAgent
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Updating identity for module $edgeAgent
Oct 22 11:11:23 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:23Z [INFO] - Pulling image mcr.microsoft.com/azureiotedge-agent:1.0...
Oct 22 11:11:24 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:24Z [INFO] - Successfully pulled image mcr.microsoft.com/azureiotedge-agent:1.0
Oct 22 11:11:24 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:24Z [INFO] - Creating module edgeAgent...
Oct 22 11:11:24 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:24Z [INFO] - Successfully created module edgeAgent
Oct 22 11:11:24 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:24Z [INFO] - Starting module edgeAgent...
Oct 22 11:11:26 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:26Z [INFO] - Successfully started module edgeAgent
Oct 22 11:11:28 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:28Z [ERR!] - Internal server error: Could not get trust bundle
Oct 22 11:11:28 dg-p061701-200917-009 iotedged[7464]:         caused by: An error occurred getting the certificate
Oct 22 11:11:28 dg-p061701-200917-009 iotedged[7464]:         caused by: HSM failure
Oct 22 11:11:28 dg-p061701-200917-009 iotedged[7464]:         caused by: HSM API returned an invalid null response
Oct 22 11:11:28 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:28Z [INFO] - [work] - - - [2020-10-22 10:11:28.393531343 UTC] "GET /trust-bundle?api-version=2019-01-30 HTTP/1.1" 500 Internal Server Error 178 "-" "-" auth_id(-)
Oct 22 11:11:28 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:28Z [ERR!] - Internal server error: Could not get trust bundle
Oct 22 11:11:28 dg-p061701-200917-009 iotedged[7464]:         caused by: An error occurred getting the certificate
Oct 22 11:11:28 dg-p061701-200917-009 iotedged[7464]:         caused by: HSM failure
Oct 22 11:11:28 dg-p061701-200917-009 iotedged[7464]:         caused by: HSM API returned an invalid null response
Oct 22 11:11:28 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:28Z [INFO] - [work] - - - [2020-10-22 10:11:28.590992263 UTC] "GET /trust-bundle?api-version=2019-01-30 HTTP/1.1" 500 Internal Server Error 178 "-" "-" auth_id(-)
Oct 22 11:11:31 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:31Z [ERR!] - Internal server error: Could not get trust bundle
Oct 22 11:11:31 dg-p061701-200917-009 iotedged[7464]:         caused by: An error occurred getting the certificate
Oct 22 11:11:31 dg-p061701-200917-009 iotedged[7464]:         caused by: HSM failure
Oct 22 11:11:31 dg-p061701-200917-009 iotedged[7464]:         caused by: HSM API returned an invalid null response
Oct 22 11:11:31 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:31Z [INFO] - [work] - - - [2020-10-22 10:11:31.674695998 UTC] "GET /trust-bundle?api-version=2019-01-30 HTTP/1.1" 500 Internal Server Error 178 "-" "-" auth_id(-)
Oct 22 11:11:34 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:34Z [ERR!] - Internal server error: Could not get trust bundle
Oct 22 11:11:34 dg-p061701-200917-009 iotedged[7464]:         caused by: An error occurred getting the certificate
Oct 22 11:11:34 dg-p061701-200917-009 iotedged[7464]:         caused by: HSM failure
Oct 22 11:11:34 dg-p061701-200917-009 iotedged[7464]:         caused by: HSM API returned an invalid null response
Oct 22 11:11:34 dg-p061701-200917-009 iotedged[7464]: 2020-10-22T10:11:34Z [INFO] - [work] - - - [2020-10-22 10:11:34.704802018 UTC] "GET /trust-bundle?api-version=2019-01-30 HTTP/1.1" 500 Internal Server Error 178 "-" "-" auth_id(-)

Additional Information

In the end we found some more details on the error

iotedged logs

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 47 (23 by maintainers)

Most upvoted comments

I have found some time to test this. Indeed when using the test certificates and including my own root certificate, it is indeed all working fine but when using my own device device certificate it fails.

I again also run with my own certificates and received some more details on the error. Again very strange because on version 9 the same certificates are working an on 10 they are not.

Detials

Dec 16 15:34:34 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:34Z [DBUG] - [edgelet_http] accepted new connection (unknown)
Dec 16 15:34:34 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:34Z [DBUG] - [edgelet_http_mgmt::server::module::list] List modules
Dec 16 15:34:34 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:34Z [DBUG] - [edgelet_docker::runtime] Listing modules...
Dec 16 15:34:34 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:34Z [DBUG] - [edgelet_docker::runtime] Successfully listed modules
Dec 16 15:34:34 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:34Z [INFO] - [mgmt] - - - [2020-12-16 15:34:34.731057953 UTC] "GET /modules?api-version=2020-07-07 HTTP/1.1" 200 OK 607 "-" "iotedge/0.1.0" auth_id(-)
Dec 16 15:34:47 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:47Z [INFO] - Checking edge runtime status
Dec 16 15:34:47 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:47Z [DBUG] - [edgelet_docker::runtime] Listing modules...
Dec 16 15:34:47 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:47Z [DBUG] - [edgelet_docker::runtime] Successfully listed modules
Dec 16 15:34:47 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:47Z [INFO] - Edge runtime status is failed, starting module now...
Dec 16 15:34:47 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:47Z [INFO] - Starting module edgeAgent...
Dec 16 15:34:48 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:48Z [INFO] - Successfully started module edgeAgent
Dec 16 15:34:50 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:50Z [DBUG] - [edgelet_http] accepted new connection (unknown)
Dec 16 15:34:51 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:51Z [ERR!] - Internal server error: Could not get trust bundle
Dec 16 15:34:51 dg-0000000-191104-001 iotedged[16366]:         caused by: An error occurred getting the certificate
Dec 16 15:34:51 dg-0000000-191104-001 iotedged[16366]:         caused by: HSM failure
Dec 16 15:34:51 dg-0000000-191104-001 iotedged[16366]:         caused by: HSM API returned an invalid null response
Dec 16 15:34:51 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:51Z [INFO] - [work] - - - [2020-12-16 15:34:51.059240846 UTC] "GET /trust-bundle?api-version=2019-01-30 HTTP/1.1" 500 Internal Server Error 178 "-" "-" auth_id(-)
Dec 16 15:34:51 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:51Z [DBUG] - [edgelet_http] accepted new connection (unknown)
Dec 16 15:34:51 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:51Z [ERR!] - Internal server error: Could not get trust bundle
Dec 16 15:34:51 dg-0000000-191104-001 iotedged[16366]:         caused by: An error occurred getting the certificate
Dec 16 15:34:51 dg-0000000-191104-001 iotedged[16366]:         caused by: HSM failure
Dec 16 15:34:51 dg-0000000-191104-001 iotedged[16366]:         caused by: HSM API returned an invalid null response
Dec 16 15:34:51 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:51Z [INFO] - [work] - - - [2020-12-16 15:34:51.288420210 UTC] "GET /trust-bundle?api-version=2019-01-30 HTTP/1.1" 500 Internal Server Error 178 "-" "-" auth_id(-)
Dec 16 15:34:54 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:54Z [DBUG] - [edgelet_http] accepted new connection (unknown)
Dec 16 15:34:54 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:54Z [ERR!] - Internal server error: Could not get trust bundle
Dec 16 15:34:54 dg-0000000-191104-001 iotedged[16366]:         caused by: An error occurred getting the certificate
Dec 16 15:34:54 dg-0000000-191104-001 iotedged[16366]:         caused by: HSM failure
Dec 16 15:34:54 dg-0000000-191104-001 iotedged[16366]:         caused by: HSM API returned an invalid null response
Dec 16 15:34:54 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:54Z [INFO] - [work] - - - [2020-12-16 15:34:54.367514591 UTC] "GET /trust-bundle?api-version=2019-01-30 HTTP/1.1" 500 Internal Server Error 178 "-" "-" auth_id(-)
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:57Z [DBUG] - [edgelet_http] accepted new connection (unknown)
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: eate:500) Failure parsing certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:33:51Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:parse_validity_timestamps:305) Could not parse 'not after' timestamp from certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:33:51Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:parse_certificate_details:381) Error obtaining validity timestamps from certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:33:51Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:parse_certificate:418) Failure obtaining first certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:33:51Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:certificate_info_create:500) Failure parsing certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:33:54Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:parse_validity_timestamps:305) Could not parse 'not after' timestamp from certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:33:54Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:parse_certificate_details:381) Error obtaining validity timestamps from certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:33:54Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:parse_certificate:418) Failure obtaining first certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:33:54Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:certificate_info_create:500) Failure parsing certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:33:57Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:parse_validity_timestamps:305) Could not parse 'not after' timestamp from certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:33:57Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:parse_certificate_details:381) Error obtaining validity timestamps from certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:33:57Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:parse_certificate:418) Failure obtaining first certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:33:57Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:certificate_info_create:500) Failure parsing certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:51Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:parse_validity_timestamps:305) Could not parse 'not after' timestamp from certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:51Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:parse_certificate_details:381) Error obtaining validity timestamps from certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:51Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:parse_certificate:418) Failure obtaining first certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:51Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:certificate_info_create:500) Failure parsing certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:51Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:parse_validity_timestamps:305) Could not parse 'not after' timestamp from certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:51Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:parse_certificate_details:381) Error obtaining validity timestamps from certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:51Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:parse_certificate:418) Failure obtaining first certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:51Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:certificate_info_create:500) Failure parsing certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:54Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:parse_validity_timestamps:305) Could not parse 'not after' timestamp from certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:54Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:parse_certificate_details:381) Error obtaining validity timestamps from certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:54Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:parse_certificate:418) Failure obtaining first certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:54Z [ERR!] (/project/edgelet/hsm-sys/azure-iot-hsm-c/src/certificate_info.c:certificate_info_create:500) Failure parsing certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16<3>2020-12-16T15:34:57Z [ERR!] - Internal server error: Could not get trust bundle
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]:         caused by: An error occurred getting the certificate
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]:         caused by: HSM failure
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]:         caused by: HSM API returned an invalid null response
Dec 16 15:34:57 dg-0000000-191104-001 iotedged[16366]: 2020-12-16T15:34:57Z [INFO] - [work] - - - [2020-12-16 15:34:57.396989446 UTC] "GET /trust-bundle?api-version=2019-01-30 HTTP/1.1" 500 Internal Server Error 178 "-" "-" auth_id(-)

Should be “your trusted cert expires post 2037

@and-rewsmith, just wanted to say thank you for the information here. Learning this this week rather than a week or two later makes a huge difference on the project I’m working on.

@kpm-at-hfi Unfortunately the support ticket contains confidential information from the reporting customer, so we aren’t allowed to share details.

Yes indeed the device CA cert. In my lasted test I completely left out the DPS setup and just used the connection string.

I will check to maybe open a support ticket to share further information.

Yes we first tried it on 1.0.10 and then we had the issue with EdgeAgent throwing the error. Moving back to 1.0.9.5 solved it.

Well the dots in the paths are just hiding where the certificate is for the post here. Normally the absolute path is in the file for every certificate.

We selected to keep the config.yaml file, have check with the new version and doesn’t seem to have changed. So just to be clear, we installed the device with the latest version of IoT Edge as we have always done and since 1.0.10 things don’t start properly even if we deliver all the same settings as before (config, certificates, etc.). Without making any changes to settings and just downgrading to 1.0.9.5, keeping the original config file, and everything worked again as it should.

Config.yaml
###############################################################################
#                      IoT Edge Daemon configuration
###############################################################################
#
# This file configures the IoT Edge daemon. The daemon must be restarted to
# pick up any configuration changes.
#
# Note - this file is yaml. Learn more here: http://yaml.org/refcard.html
#
###############################################################################

###############################################################################
# Provisioning mode and settings
###############################################################################
#
# Configures the identity provisioning mode of the daemon.
#
# Supported modes:
#     manual   - using an iothub connection string
#     dps      - using dps for provisioning
#     external - the device has been provisioned externally.
#                Uses an external provisioning endpoint to get device specific information.
#
# DPS Settings
#     scope_id        - Required. Value of a specific DPS instance's ID scope
#     registration_id - Required for TPM and symmetric key provisioning flows.
#                       Optional for X.509 provisioning. Registration ID of a
#                       specific device in DPS.
#                       For more information regarding DPS registration ids
#                       please see https://docs.microsoft.com/en-us/azure/iot-dps/concepts-device#registration-id
#     symmetric_key   - Optional. This entry should only be specified when
#                       provisioning devices configured for symmetric key
#                       attestation. Device specific symmetric key.
#     identity_cert   - Optional. The Edge device identity X.509 certificate
#                       entry should only be specified when provisioning
#                       an Edge device configured for X.509 attestation.
#                       The value should be specified as a URI.
#                       Ex. when specifying a PEM encoded certificate file, the URI
#                       should be specified as file:///path/identity_certificate.pem
#     identity_pk     - Optional. The Edge device identity private key
#                       entry should only be specified when provisioning
#                       an Edge device configured for X.509 attestation.
#                       The value should be specified as a URI.
#                       Ex. when specifying a PEM encoded private key file, the URI
#                       should be specified as file:///path/identity_key.pem
#
# External Settings
#     endpoint - Required. Value of the endpoint used to retrieve device specific
#                information such as its IoT hub connection information.
###############################################################################

# Manual provisioning configuration
# provisioning:
#   source: "manual"
#   device_connection_string: "<ADD DEVICE CONNECTION STRING HERE>"

# DPS TPM provisioning configuration
# provisioning:
#   source: "dps"
#   global_endpoint: "https://global.azure-devices-provisioning.net"
#   scope_id: "<SCOPE_ID>"
#   attestation:
#     method: "tpm"
#     registration_id: "<REGISTRATION_ID>"

# DPS symmetric key provisioning configuration
# provisioning:
#   source: "dps"
#   global_endpoint: "https://global.azure-devices-provisioning.net"
#   scope_id: "<SCOPE_ID>"
#   attestation:
#     method: "symmetric_key"
#     registration_id: "<REGISTRATION_ID>"
#     symmetric_key: "<SYMMETRIC_KEY>"

# DPS X.509 provisioning configuration
provisioning:
  source: "dps"
  global_endpoint: "https://global.azure-devices-provisioning.net"
  scope_id: "<<ID>>"
  attestation:
    method: "x509"
    registration_id: "<<ID>>"
    identity_cert: "file:///<<HIDE>>/Device_Identity.pem"
    identity_pk: "file:///<<HIDE>>/Device_Identity.key.pem"

# External provisioning configuration
# provisioning:
#   source: "external"
#   endpoint: "http://localhost:9999"

###############################################################################
# Certificate settings
###############################################################################
#
# Configures the certificates required to operate the IoT Edge
# runtime as a gateway which enables external leaf devices to securely
# communicate with the Edge Hub. If not specified, the required certificates
# are auto generated for quick start scenarios which are not intended for
# production environments.
#
# Settings:
#     device_ca_cert   - URI of the device ca certificate and its chain.
#                        Optionally can be specified as a file path.
#     device_ca_pk     - URI of the device ca private key file.
#                        Optionally can be specified as a file path.
#     trusted_ca_certs - URI containing all the trusted CA
#                        certificates required for Edge module communication
#                        Optionally can be specified as a file path.
#
# Note:
# The values of all of these fields must be specified as a
# "file" scheme URI such as "file:///path/cert_key.pem"
###############################################################################

certificates:
  device_ca_cert: "file:///<<HIDE>>/Device_CA.pem"
  device_ca_pk: "file:///<<HIDE>>/Device_CA.key.pem"
  trusted_ca_certs: "file:///<<HIDE>>/Root_CA.pem"

###############################################################################
# Edge Agent module spec
###############################################################################
#
# Configures the initial Edge Agent module.
#
# The daemon uses this definition to bootstrap the system. The Edge Agent can
# then update itself based on the Edge Agent module definition present in the
# deployment in IoT Hub.
#
# Settings:
#     name     - name of the edge agent module. Expected to be "edgeAgent".
#     type     - type of module. Always "docker".
#     env      - Any environment variable that needs to be set for edge agent module.
#     config   - type specific configuration for edge agent module.
#       image  - (docker) Modules require a docker image tag.
#       auth   - (docker) Modules may need authoriation to connect to container registry.
#
# Adding environment variables:
# replace "env: {}" with
#  env:
#    key: "value"
#
# Adding container registry authorization:
# replace "auth: {}" with
#    auth:
#      username: "username"
#      password: "password"
#      serveraddress: "serveraddress"
#
###############################################################################

agent:
  name: "edgeAgent"
  type: "docker"
  env: {}
  config:
    image: "mcr.microsoft.com/azureiotedge-agent:1.0"
    auth: {}

###############################################################################
# Edge device hostname
###############################################################################
#
# Configures the environment variable 'IOTEDGE_GATEWAYHOSTNAME' injected into
# modules. Regardless of case the hostname is specified below, a lower case
# value is used to configure the Edge Hub server hostname as well as the
# environment variable specified above.
#
# It is important to note that when connecting downstream devices to the
# Edge Hub that the lower case value of this hostname be used in the
# 'GatewayHostName' field of the device's connection string URI.
###############################################################################

hostname: "<<Serialnumber>>"

###############################################################################
# Watchdog settings
###############################################################################
#
# The IoT edge daemon has a watchdog that periodically checks the health of the
# Edge Agent module and restarts it if it's down.
#
# max_retries - Configures the number of retry attempts that the IoT edge daemon
#               should make for failed operations before failing with a fatal error.
#
#               If this configuration is not specified, the daemon keeps retrying
#               on errors and doesn't fail fatally.
#
#               On a fatal failure, the daemon returns an exit code which
#               signifies the kind of error encountered. Currently, the following
#               error codes are returned by the daemon -
#
#               150 - Invalid Device ID specified.
#               151 - Invalid IoT hub configuration.
#               152 - Invalid SAS token used to call IoT hub.
#                     This could signal an invalid SAS key.
#               1 - All other errors.
###############################################################################

#watchdog:
#  max_retries: 2

###############################################################################
# Connect settings
###############################################################################
#
#
#Configures URIs used by clients of the management and workload APIs
#     management_uri - used by the Edge Agent and 'iotedge' CLI to start,
#                      stop, and manage modules
#     workload_uri   - used by modules to retrieve tokens and certificates
#
# The following uri schemes are supported:
#     http - connect over TCP
#     unix - connect over Unix domain socket
#
###############################################################################

connect:
  management_uri: "unix:///var/run/iotedge/mgmt.sock"
  workload_uri: "unix:///var/run/iotedge/workload.sock"

###############################################################################
# Listen settings
###############################################################################
#
# Configures the listen addresses for the daemon.
#     management_uri - used by the Edge Agent and 'iotedge' CLI to start,
#                      stop, and manage modules
#     workload_uri   - used by modules to retrieve tokens and certificates
#
# The following uri schemes are supported:
#     http - listen over TCP
#     unix - listen over Unix domain socket
#     fd   - listen using systemd socket activation
#
# These values can be different from the connect URIs. For instance, when
# using the fd:// scheme for systemd:
#     listen address is fd://iotedge.workload,
#     connect address is unix:///var/run/iotedge/workload.sock
#
###############################################################################

listen:
  management_uri: "fd://iotedge.mgmt.socket"
  workload_uri: "fd://iotedge.socket"

###############################################################################
# Home Directory
###############################################################################
#
# Configures the home directory for the daemon.
#
###############################################################################

homedir: "/var/lib/iotedge"

###############################################################################
# Moby Container Runtime settings
###############################################################################
#
# uri - configures the uri for the container runtime.
# network - configures the network on which the containers will be created.
#
# Additional container network configuration such as enabling IPv6 networking
# and providing the IPAM settings can be achieved by specifying the relevant
# configuration in the network settings.
#
# network:
#   name: "azure-iot-edge"
#   ipv6: true
#   ipam:
#     config:
#       -
#           gateway: '172.18.0.1'
#           subnet: '172.18.0.0/16'
#           ip_range: '172.18.0.0/16'
#       -
#           gateway: '2021:ffff:e0:3b1:1::1'
#           subnet: '2021:ffff:e0:3b1:1::/80'
#           ip_range: '2021:ffff:e0:3b1:1::/80'
###############################################################################

moby_runtime:
  uri: "unix:///var/run/docker.sock"
  # network: "azure-iot-edge"
  #
  # network:
  #   name: "azure-iot-edge"
  #   ipv6: true
  #   ipam:
  #     config:
  #       -
  #           gateway: '172.18.0.1'
  #           subnet: '172.18.0.0/16'
  #           ip_range: '172.18.0.0/16'
  #       -
  #           gateway: '2021:ffff:e0:3b1:1::1'
  #           subnet: '2021:ffff:e0:3b1:1::/80'
  #           ip_range: '2021:ffff:e0:3b1:1::/80'