thin-edge.io: Restart operation stuck in executing

Describe the bug We’ve installed thin-edge on a custom board with a custom package-manager. We also had to relocate the tmp.path to a non-volatile path:

$ tedge config list
...
tmp.path=/mnt/data/tedge/download
...

When I trigger a restart from c8y the device restarts, but the agent does not set the operation to successful. I checked the log, but I could find anything pointing me to the error.

Log:

$ cat /var/log/tedge-agent.err
2023-08-11T08:13:24.09699871Z  INFO flockfile::unix: Lockfile created "/run/lock/tedge-agent.lock"
2023-08-11T08:13:24.097237419Z  INFO tedge_agent::agent: tedge-agent starting
2023-08-11T08:13:24.097401085Z  INFO sm-agent: tedge_agent::agent: Starting tedge agent
2023-08-11T08:13:24.104584794Z  INFO Runtime: Started
2023-08-11T08:13:24.10618296Z  INFO Runtime: Running Signal-Handler-0
2023-08-11T08:13:24.109716335Z  INFO Runtime: Running HttpFileTransferServer-1
2023-08-11T08:13:24.122435544Z  INFO Runtime: Running MQTT-2
2023-08-11T08:13:24.14085996Z  INFO Runtime: Running RestartManagerActor-3
2023-08-11T08:13:24.140973794Z  INFO Runtime: Running SoftwareManagerActor-4
2023-08-11T08:13:24.141049127Z  INFO Runtime: Running TedgeOperationConverter-5
2023-08-11T08:13:24.141117002Z  INFO Runtime: Running HealthMonitorActor-6
2023-08-11T08:13:24.141268585Z  INFO tedge-agent: send Message { topic: Topic { name: "tedge/health/tedge-agent" }, payload: "{"pid":2838,"status":"up","time":1691741604}", qos: AtLeastOnce, retain: true }
2023-08-11T08:13:24.621208585Z  INFO plugin_sm::plugin_manager: Plugin activated: /mnt/data/tedge/sm-plugins/cds
2023-08-11T08:13:33.55458817Z  INFO plugin_sm::plugin_manager: Plugin activated: /mnt/data/tedge/sm-plugins/ap
2023-08-11T08:13:33.554966295Z  INFO plugin_sm::plugin_manager: Default plugin type: ap
2023-08-11T08:13:33.570519378Z ERROR mqtt_channel::connection: MQTT connection error: Timeout
2023-08-11T08:13:33.614210211Z  INFO mqtt_channel::connection: MQTT connection established
2023-08-11T08:13:33.619202461Z  INFO MQTT: recv Ok(Some(Message { topic: Topic { name: "tedge/health/tedge-agent" }, payload: "{"pid":2838,"status":"up","time":1691741604}", qos: AtLeastOnce, retain: true }))
2023-08-11T08:13:33.714825795Z  INFO MQTT: send Message { topic: Topic { name: "tedge/commands/req/software/list" }, payload: "{"id":"ZpDTcMeq4_HqEQ3S5Xe4P"}", qos: AtLeastOnce, retain: false }
2023-08-11T08:13:33.71507942Z  INFO Mqtt-Request-Converter: recv Some(MqttMessage(Message { topic: Topic { name: "tedge/commands/req/software/list" }, payload: "{"id":"ZpDTcMeq4_HqEQ3S5Xe4P"}", qos: AtLeastOnce, retain: false }))
2023-08-11T08:13:33.715203628Z  INFO SoftwareSender: send SoftwareListRequest(SoftwareListRequest { id: "ZpDTcMeq4_HqEQ3S5Xe4P" })

2023-08-11T08:13:33.715303753Z  INFO SoftwareManager: recv Some(SoftwareListRequest(SoftwareListRequest { id: "ZpDTcMeq4_HqEQ3S5Xe4P" }))
2023-08-11T08:13:33.723645586Z  INFO SoftwareManager: send SoftwareListResponse(SoftwareListResponse { response: SoftwareRequestResponse { id: "ZpDTcMeq4_HqEQ3S5Xe4P", status: Executing, reason: None, current_software_list: None, failures: [] } })
2023-08-11T08:13:33.737589878Z  INFO Mqtt-Request-Converter: recv Some(SoftwareResponse(SoftwareListResponse(SoftwareListResponse { response: SoftwareRequestResponse { id: "ZpDTcMeq4_HqEQ3S5Xe4P", status: Executing, reason: None, current_software_list: None, failures: [] } })))
2023-08-11T08:13:33.737821003Z  INFO MqttPublisher: send Message { topic: Topic { name: "tedge/commands/res/software/list" }, payload: "{"id":"ZpDTcMeq4_HqEQ3S5Xe4P","status":"executing"}", qos: AtLeastOnce, retain: false }
2023-08-11T08:13:33.738023628Z  INFO MQTT: recv Ok(Some(Message { topic: Topic { name: "tedge/commands/res/software/list" }, payload: "{"id":"ZpDTcMeq4_HqEQ3S5Xe4P","status":"executing"}", qos: AtLeastOnce, retain: false }))
2023-08-11T08:13:44.478300131Z  INFO SoftwareManager: send SoftwareListResponse(SoftwareListResponse { response: SoftwareRequestResponse { id: "ZpDTcMeq4_HqEQ3S5Xe4P", status: Successful, reason: None, current_software_list: Some([SoftwareRequestResponseSoftwareList { plugin_type: "ap", modules: [SoftwareModuleItem { name: "thin-edge", version: Some("0.12.0_0.8.4"), url: None, action: None, reason: None }, SoftwareModuleItem { name: "CODESYS", version: Some("3.5.16.30-38"), url: None, action: None, reason: None }] }, SoftwareRequestResponseSoftwareList { plugin_type: "cds", modules: [] }]), failures: [] } })
2023-08-11T08:13:44.484829667Z  INFO Mqtt-Request-Converter: recv Some(SoftwareResponse(SoftwareListResponse(SoftwareListResponse { response: SoftwareRequestResponse { id: "ZpDTcMeq4_HqEQ3S5Xe4P", status: Successful, reason: None, current_software_list: Some([SoftwareRequestResponseSoftwareList { plugin_type: "ap", modules: [SoftwareModuleItem { name: "thin-edge", version: Some("0.12.0_0.8.4"), url: None, action: None, reason: None }, SoftwareModuleItem { name: "CODESYS", version: Some("3.5.16.30-38"), url: None, action: None, reason: None }] }, SoftwareRequestResponseSoftwareList { plugin_type: "cds", modules: [] }]), failures: [] } })))
2023-08-11T08:13:44.485580115Z  INFO MqttPublisher: send Message { topic: Topic { name: "tedge/commands/res/software/list" }, payload: "{"id":"ZpDTcMeq4_HqEQ3S5Xe4P","status":"successful","currentSoftwareList":[{"type":"ap","modules":[{"name":"thin-edge","version":"0.12.0_0.8.4"},{"name":"CODESYS","version":"3.5.16.30-38"}]},{"type":"cds","modules":[]}]}", qos: AtLeastOnce, retain: false }

2023-08-11T08:13:44.485750933Z  INFO MQTT: recv Ok(Some(Message { topic: Topic { name: "tedge/commands/res/software/list" }, payload: "{"id":"ZpDTcMeq4_HqEQ3S5Xe4P","status":"successful","currentSoftwareList":[{"type":"ap","modules":[{"name":"thin-edge","version":"0.12.0_0.8.4"},{"name":"CODESYS","version":"3.5.16.30-38"}]},{"type":"cds","modules":[]}]}", qos: AtLeastOnce, retain: false }))

To Reproduce

Expected behavior

Screenshots

Environment (please complete the following information):

OS [incl. version]
Hardware [incl. revision]
System-Architecture [e.g. result of “uname -a”]
thin-edge.io version [e.g. 0.1.0]

Additional context

About this issue

Original URL
State: closed
Created a year ago
Comments: 18 (18 by maintainers)

Most upvoted comments

I think the problem is caused by something different.

I write a small shell script:

Write File
Sync
Reboot This worked. The file content has survived the reboot.

I assume, that something restarts the tedge-agent when calling “init 6”, “shutdown -r now” or “reboot”. And he agent clears the file on startup. If I use “reboot -f” I do not have a problem. So I think it doesn’t has anything todo with thin-edge itself.

toewsar on Aug 22, 2023

I’m not a RUST specialist and also not fimilar with async/await. Can you confirm that the write operation is finished when calling sync?

I confirm that a call to file.sync_all().await? returns only after the system call to fsync returns. The await just means that another thread can make progress meantime.

However, the behavior of fsync is not guaranteed and depends on the file system.

didier-wenzek on Aug 18, 2023

$ git describe 
0.12.0-96-gf8aef0e7f4

toewsar on Aug 17, 2023