iotedge: Fetching logs crashes edge agent

Expected Behavior

Viewing logs via azure portal should show logs

Logs

<6> 2023-08-03 17:09:20.328 +00:00 [INF] - Received request ping with payload
<6> 2023-08-03 17:09:20.337 +00:00 [INF] - Successfully handled request ping
<6> 2023-08-03 17:09:20.656 +00:00 [INF] - Received direct method call - GetModuleLogs
<6> 2023-08-03 17:09:20.656 +00:00 [INF] - Received request GetModuleLogs with payload
<6> 2023-08-03 17:09:20.731 +00:00 [INF] - Processing request to get logs for {"schemaVersion":"1.0","items":{"id":"\\btcmservice\\b","filter":{"tail":1500,"since":"15m","until":null,"loglevel":null,"regex":""}},"encoding":1,"contentType":1}
<6> 2023-08-03 17:09:20.845 +00:00 [INF] - Initiating streaming logs for tcmservice
<4> 2023-08-03 17:09:21.406 +00:00 [WRN] - Error handling request GetModuleLogs
System.IO.IOException: Unexpected end of stream.
   at Microsoft.Azure.Devices.Edge.Util.Uds.HttpBufferedStream.ReadLine() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/uds/HttpBufferedStream.cs:line 91
   at Microsoft.Azure.Devices.Edge.Util.Uds.HttpChunkedStreamReader.Read(Byte[] buffer, Int32 offset, Int32 count) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/uds/HttpChunkedStreamReader.cs:line 87
   at System.IO.DelegatingStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at Akka.Streams.Implementation.IO.InputStreamPublisher.ReadAndEmit()
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Edge.Agent.Core.Logs.LogsProcessor.GetText(String id, Stream stream, ModuleLogFilter filter) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/logs/LogsProcessor.cs:line 93
   at Microsoft.Azure.Devices.Edge.Agent.Core.Logs.LogsProvider.ProcessByContentType(String id, Stream logsStream, ModuleLogOptions logOptions) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/logs/LogsProvider.cs:line 99
   at Microsoft.Azure.Devices.Edge.Agent.Core.Logs.LogsProvider.GetProcessedLogs(String id, Stream logsStream, ModuleLogOptions logOptions) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/logs/LogsProvider.cs:line 85
   at Microsoft.Azure.Devices.Edge.Agent.Core.Logs.LogsProvider.GetLogs(String id, ModuleLogOptions logOptions, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/logs/LogsProvider.cs:line 38
   at Microsoft.Azure.Devices.Edge.Agent.Core.Requests.ModuleLogsRequestHandler.<>c__DisplayClass8_0.<<HandleRequestInternal>b__1>d.MoveNext() in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/requests/ModuleLogsRequestHandler.cs:line 58
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Edge.Agent.Core.Requests.ModuleLogsRequestHandler.HandleRequestInternal(Option`1 payloadOption, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/requests/ModuleLogsRequestHandler.cs:line 76
   at Microsoft.Azure.Devices.Edge.Agent.Core.Requests.RequestHandlerBase`2.HandleRequest(Option`1 payloadJson, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/requests/RequestHandlerBase.cs:line 30
   at Microsoft.Azure.Devices.Edge.Agent.Core.Requests.RequestManager.ProcessRequest(String request, String payloadJson) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/requests/RequestManager.cs:line 54
<6> 2023-08-03 17:09:23.457 +00:00 [INF] - Termination requested, initiating shutdown.
<6> 2023-08-03 17:09:23.458 +00:00 [INF] - Main thread terminated
<6> 2023-08-03 17:09:23.466 +00:00 [INF] - Initiating shutdown cleanup.
<6> 2023-08-03 17:09:23.482 +00:00 [INF] - Stopping all modules...

Device Information

  • Host OS: Ubuntu 20.04
  • Architecture: amd64
  • Container OS: Linux Containers

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Comments: 22 (12 by maintainers)

Most upvoted comments

@nlcamp I have the same problem as WDoughty, but with access to the device. I noticed that this error occurs when i disconnect the device from the power supply and restart it. After that, i only can get rid off this “Log” error by redeploying all modules.

@nlcamp Actually had this issue with a separate device right now image image image

I have 6 custom modules and only the 1 is unable to fetch the logs.

@nlcamp We kept running into an issue running with the iotedge logs command and it would return this Error grabbing logs: error unmarshalling log entry (size=15370): proto: LogEntry: illegal tag 0 (wire type 6)

We ended up running docker logs edgeAgent and the version was this Version - 1.4.15.76247273 (9e713419235666f04ee5fa8823386890ef087193)

As for now the device does not have internet connection, if that helps at all.