iotedge: TPM Provisioning doesn't work for iotedge version 1.4 unless upgrade from version 1.3

Expected Behavior

With version 1.3 we are able to deploy iotedge on a clean installation of ubuntu device with the next script using TPM provisioning:

#!/bin/bash
# Variable
SCOPE_ID=0ne000XXXXX
REGISTRATION_ID=YYYYYYYYYYYY

#Distro management
UBUNTU20=20.04
UBUNTU18=18.04/multiarch
 
DISTRO="$UBUNTU20"
 
# Prerequisites
wget https://packages.microsoft.com/config/ubuntu/$DISTRO/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
 
sudo dpkg -i packages-microsoft-prod.deb
rm packages-microsoft-prod.deb

# Installation forcing version 1.3.0
sudo apt-get update
sudo apt-get install moby-engine
sudo apt-get install aziot-identity-service=1.3.0-1 aziot-edge=1.3.0-1

# Provisioning
STARTLINE=$(sudo awk '/DPS provisioning with TPM/{ print NR; exit }' /etc/aziot/config.toml.edge.template)
ENDLINE=$((STARTLINE+9))	

sudo sed "$STARTLINE"','"$ENDLINE"'s:^# ::' /etc/aziot/config.toml.edge.template > config.toml
sed -i 's:^id_scope.*:id_scope = "'"$SCOPE_ID"'":' config.toml
sed -i 's:^registration_id.*:registration_id = "'"$REGISTRATION_ID"'":' config.toml
sudo cp ./config.toml /etc/aziot/
rm ./config.toml

# Give IoT Edge access to the TPM
echo '# allow aziottpm access to tpm0 and tpmrm0' >> tpmaccess.rules
echo 'KERNEL=="tpm0", SUBSYSTEM=="tpm", OWNER="aziottpm", MODE="0660"' >> tpmaccess.rules
echo 'KERNEL=="tpmrm0", SUBSYSTEM=="tpmrm", OWNER="aziottpm", MODE="0660"' >> tpmaccess.rules

sudo cp -f tpmaccess.rules /etc/udev/rules.d/
rm tpmaccess.rules

/bin/udevadm trigger --subsystem-match=tpm --subsystem-match=tpmrm

# Apply configuration changes
sudo iotedge config apply

and this is the result in the /etc/aziot/config.toml file

## DPS provisioning with TPM
[provisioning]
source = "dps"
global_endpoint = "https://global.azure-devices-provisioning.net"
id_scope = "0ne000XXXXX"
#
[provisioning.attestation]
method = "tpm"
registration_id = "YYYYYYYYYYYY"

And this the result of iotedge system status

> sudo iotedge system status
System services:
    aziot-edged             Running
    aziot-identityd         Running
    aziot-keyd              Running
    aziot-certd             Running
    aziot-tpmd              Running

Use 'iotedge system logs' to check for non-fatal errors.
Use 'iotedge check' to diagnose connectivity and configuration issues.

Of course on the DPS an individual enrollment with TPM method and the right Endorsement key has been prepared in advance.

Current Behavior

With version 1.4 the config.toml file slightly changed the format adding 2 rows in the TPM provisioning section where it is possible to send a custom payload during DPS registration… We have no need to do this (I suppose, since no change in DPS) so let them commented out. The script is identical with few changes on how config.toml is craeted (temporary solution… in the future will manage better) and of course no version forced. Here the differences

# Installation using last version 1.4
sudo apt-get update
sudo apt-get install moby-engine
sudo apt-get install aziot-edge

# Provisioning with new config.toml format
STARTLINE=$(sudo awk '/DPS provisioning with TPM/{ print NR; exit }' /etc/aziot/config.toml.edge.template)
ENDLINE1=$((STARTLINE+4))
STARTLINE2=$((ENDLINE1+4))
ENDLINE=$((STARTLINE2+3))

sudo sed "$STARTLINE"','"$ENDLINE1"'s:^# ::' /etc/aziot/config.toml.edge.template > config.toml
sed -i "$STARTLINE2"','"$ENDLINE"'s:^# ::' config.toml
sed -i 's:^id_scope.*:id_scope = "'"$SCOPE_ID"'":' config.toml
sed -i 's:^registration_id.*:registration_id = "'"$REGISTRATION_ID"'":' config.toml
sudo cp ./config.toml /etc/aziot/
rm ./config.toml

and this is the result in the /etc/aziot/config.toml file

## DPS provisioning with TPM
[provisioning]
source = "dps"
global_endpoint = "https://global.azure-devices-provisioning.net"
id_scope = "0ne000XXXXX"
#
## Uncomment to send a custom payload during DPS registration
# payload = { uri = "file:///var/secrets/aziot/identityd/dps-additional-data.json" }
#
[provisioning.attestation]
method = "tpm"
registration_id = "YYYYYYYYYYYY"

Ok, this looks like perfectly identical to the one of version 1.3… greats… it shuold work… Also everything is following documentation But if running on Ubuntu clean installation, it doesn’t works

Here the result of iotedge systen status

sudo iotedge system status
System services:
    aziot-edged             Running
    aziot-identityd         Running
    aziot-keyd              Ready
    aziot-certd             Ready
    aziot-tpmd              Down - activating

aziot-tpmd is in a bad state because:
aziot-tpmd.service: Down - activating : Printing the last 10 log lines.
-- Logs begin at Mon 2022-09-05 15:29:55 UTC, end at Mon 2022-09-05 15:44:39 UTC. --
Sep 05 15:44:39 edgelinux aziot-tpmd[4228]: ERROR:tcti:src/tss2-tcti/tctildr-dl.c:150:tcti_from_file() Could not initialize TCTI file: device
Sep 05 15:44:39 edgelinux aziot-tpmd[4228]: ERROR:tcti:src/tss2-tcti/tctildr.c:418:Tss2_TctiLdr_Initialize_Ex() Failed to instantiate TCTI
Sep 05 15:44:39 edgelinux aziot-tpmd[4228]: 2022-09-05T15:44:39Z [ERR!] - service encountered an error
Sep 05 15:44:39 edgelinux aziot-tpmd[4228]: 2022-09-05T15:44:39Z [ERR!] - caused by: internal error
Sep 05 15:44:39 edgelinux aziot-tpmd[4228]: 2022-09-05T15:44:39Z [ERR!] - caused by: could not initialize TPM
Sep 05 15:44:39 edgelinux aziot-tpmd[4228]: 2022-09-05T15:44:39Z [ERR!] - caused by: tcti:IO failure
Sep 05 15:44:39 edgelinux aziot-tpmd[4228]: 2022-09-05T15:44:39Z [ERR!] -    0: <unknown>
Sep 05 15:44:39 edgelinux aziot-tpmd[4228]:    1: <unknown>
Sep 05 15:44:39 edgelinux systemd[1]: aziot-tpmd.service: Main process exited, code=exited, status=1/FAILURE
Sep 05 15:44:39 edgelinux systemd[1]: aziot-tpmd.service: Failed with result 'exit-code'.

Use 'iotedge system logs' to check for non-fatal errors.
Use 'iotedge check' to diagnose connectivity and configuration issues. 

Steps to Reproduce

1. On DPS create an Individual enrolment with TPM method and correct endorsement key retrieved by physical device
2. New clean installation of Ubuntu server 20.04TLS with security update on phisical device
3. Install iotedge following next steps (as described in documentation and shown is script above)

sudo apt-get update
sudo apt-get install moby-engine
sudo apt-get install aziot-edge

1. Give IoT Edge access to the TPM as described in documentation and in the script above
2. Apply changes (sudo iotedge config apply) in order to provision the iotedge and reboot in order to ensure everything starts clean

Context (Environment)

Output of iotedge check

iotedge check

Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK
√ aziot-identity-service package is up-to-date - OK
√ host time is close to reference time - OK
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ tpmd is running - OK
√ identityd is running - OK
× read all preloaded certificates from the Certificates Service - Error
    could not load cert with ID "aziot-edged-trust-bundle"

    Caused by:
        parameter "id" has an invalid value
        caused by: not found
√ read all preloaded key pairs from the Keys Service - OK
√ check all EST server URLs utilize HTTPS - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)
--------------------------------------------
‼ host can connect to and perform TLS handshake with iothub AMQP port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    Since no hostname is provided, all hub connectivity tests will be skipped.
‼ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    Since no hostname is provided, all hub connectivity tests will be skipped.
‼ host can connect to and perform TLS handshake with iothub MQTT port - Warning
    Could not retrieve iothub_hostname from provisioning file.
    Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
    Since no hostname is provided, all hub connectivity tests will be skipped.
√ host can connect to and perform TLS handshake with DPS endpoint - OK

Configuration checks
--------------------
√ aziot-edged configuration is well-formed - OK
√ configuration up-to-date with config.toml - OK
√ container engine is installed and functional - OK
× configuration has correct URIs for daemon mgmt endpoint - Error
    SocketError - SocketErrorCode (TimedOut) : Operation timed out
    One or more errors occurred. (Got bad response: )
√ aziot-edge package is up-to-date - OK
√ container time is close to host time - OK
‼ DNS server - Warning
    Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
    Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
    You can ignore this warning if you are setting DNS server per module in the Edge deployment.
‼ production readiness: logs policy - Warning
    Container engine is not configured to rotate module logs which may cause it run out of disk space.
    Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
    You can ignore this warning if you are setting log policy per module in the Edge deployment.
× production readiness: Edge Agent's storage directory is persisted on the host filesystem - Error
    Could not check current state of edgeAgent container
× production readiness: Edge Hub's storage directory is persisted on the host filesystem - Error
    Could not check current state of edgeHub container
√ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK

Connectivity checks
-------------------
23 check(s) succeeded.
5 check(s) raised warnings. Re-run with --verbose for more details.
4 check(s) raised errors. Re-run with --verbose for more details.
7 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details.

Device Information

• Host OS Ubuntu server 20.04LTS
• Architecture amd64
• Container OS Linux containers

Runtime Versions

• aziot-edged: 1.4.0
• Edge Agent: Never started
• Edge Hub: Never started
• Docker/Moby:

Client:
 Version:           20.10.17+azure-1
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c70180fde3601def79a59cc3e996aa553c9b9
 Built:             Mon Jun  6 21:36:39 UTC 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.17+azure-3
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b84221c8560e7a3dee2a653353429e7628424
  Built:            Mon Jun  6 22:32:38 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.13+azure-1
  GitCommit:        a17ec496a95e55601607ca50828147e8ccaeebf1
 runc:
  Version:          1.0.3
  GitCommit:        f46b6ba2c9314cfc8caae24a32ec5fe9ef1059fe
 docker-init:
  Version:          0.19.0
  GitCommit: 

Logs

Sep 05 15:45:52 edgelinux aziot-identityd[4550]: 2022-09-05T15:45:52Z [INFO] - Starting service...
Sep 05 15:45:52 edgelinux aziot-identityd[4550]: 2022-09-05T15:45:52Z [INFO] - Version - 1.4.0
Sep 05 15:45:52 edgelinux aziot-identityd[4550]: 2022-09-05T15:45:52Z [INFO] - Provisioning starting. Reason: Startup
Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: 2022-09-05T15:45:52Z [INFO] - Starting service...
Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: 2022-09-05T15:45:52Z [INFO] - Version - 1.4.0
Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: ERROR:tcti:src/tss2-tcti/tcti-device.c:439:Tss2_Tcti_Device_Init() Failed to open device file /dev/tpm0: Permission denied
Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: WARNING:tcti:src/tss2-tcti/tctildr.c:62:tcti_from_init() TCTI init for function 0x7fdeeded3fb0 failed with a000a
Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: WARNING:tcti:src/tss2-tcti/tctildr.c:92:tcti_from_info() Could not initialize TCTI named: tcti-device
Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: ERROR:tcti:src/tss2-tcti/tctildr-dl.c:150:tcti_from_file() Could not initialize TCTI file: device
Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: ERROR:tcti:src/tss2-tcti/tctildr.c:418:Tss2_TctiLdr_Initialize_Ex() Failed to instantiate TCTI
Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: 2022-09-05T15:45:52Z [ERR!] - service encountered an error
Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: 2022-09-05T15:45:52Z [ERR!] - caused by: internal error
Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: 2022-09-05T15:45:52Z [ERR!] - caused by: could not initialize TPM
Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: 2022-09-05T15:45:52Z [ERR!] - caused by: tcti:IO failure
Sep 05 15:45:52 edgelinux aziot-tpmd[4557]: 2022-09-05T15:45:52Z [ERR!] -    0: <unknown>
Sep 05 15:45:52 edgelinux aziot-tpmd[4557]:    1: <unknown>
Sep 05 15:45:52 edgelinux systemd[1]: aziot-tpmd.service: Main process exited, code=exited, status=1/FAILURE
Sep 05 15:45:52 edgelinux systemd[1]: aziot-tpmd.service: Failed with result 'exit-code'.

Additional Information

The issue will happen on clean Ubuntu installation. If you try the next the TPM provisioning and iotedge 1.4 will work together.

1. On a clean ubuntu server 20.04LTS installation run the first script forcing installation of version 1.3 with TPM provisioning
2. uninstall iotedge following next steps (more or less taken by documentation)

sudo apt-get autoremove --purge aziot-edge

#get all container names, stop them all and delete them all
sudo docker stop $(sudo docker ps -a -q)
sudo docker rm $(sudo docker ps -a -q)

#get all docker images ids and delete them all
sudo docker rmi $(sudo docker images -a -q)

#delete volumes not connected to containers
sudo docker volume ls -f dangling=true
sudo docker volume prune

#purge docker
sudo apt-get autoremove --purge moby-engine

1. launch the script modified in order to install version 1.4 Ok the magic is there… The TPM provisioning is working and your containers start to be downloaded and installed…

• If on the same clean machine I’ll install iotedge 1.4 with symmetric key provisioning it works
• If I try to install TPM provisioning after uninstallation of iotedge 1.4 with Symmetric key provisioning the issue is still there