diagnostics: (linux) dotnet-trace, dotnet-dump and dotnet-gcdump seem to leave the process in a state where it cannot be traced or dumped again.

AzureUser@machine-t:~$ sudo /home/AzureUser/.dotnet/tools/dotnet-trace collect --process-id 67295
No profile or providers specified, defaulting to trace profile 'cpu-sampling'

Provider Name                           Keywords            Level               Enabled By
Microsoft-DotNETCore-SampleProfiler     0x0000000000000000  Informational(4)    --profile
Microsoft-Windows-DotNETRuntime         0x00000014C14FCCBD  Informational(4)    --profile

Process        : /mnt/publish/process
Output File    : /home/AzureUser/trace.nettrace

[00:00:00:00]   Recording trace 102.00   (B)
Press <Enter> or <Ctrl+C> to exit...

Trace completed.

Run the same command again later…

AzureUser@machine:~$ sudo /home/AzureUser/.dotnet/tools/dotnet-trace collect --process-id 67295
No profile or providers specified, defaulting to trace profile 'cpu-sampling'

Provider Name                           Keywords            Level               Enabled By
Microsoft-DotNETCore-SampleProfiler     0x0000000000000000  Informational(4)    --profile
Microsoft-Windows-DotNETRuntime         0x00000014C14FCCBD  Informational(4)    --profile

Unable to start a tracing session: Microsoft.Diagnostics.NETCore.Client.ServerNotAvailableException: Process 67295 not running compatible .NET Core runtime.
   at Microsoft.Diagnostics.NETCore.Client.IpcClient.GetTransport(Int32 processId) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 63
   at Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessage(Int32 processId, IpcMessage message, IpcMessage& response) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 104
   at Microsoft.Diagnostics.NETCore.Client.EventPipeSession..ctor(Int32 processId, IEnumerable`1 providers, Boolean requestRundown, Int32 circularBufferMB) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClient/EventPipeSession.cs:line 30
   at Microsoft.Diagnostics.Tools.Trace.CollectCommandHandler.Collect(CancellationToken ct, IConsole console, Int32 processId, FileInfo output, UInt32 buffersize, String providers, String profile, TraceFileFormat format, TimeSpan duration, String clrevents, String clreventlevel) in /_/src/Tools/dotnet-trace/CommandLine/Commands/CollectCommand.cs:line 130
Unable to create session.

This occurs with both dotnet-trace and dotnet-dump running on:

Ubuntu 18.04.4 LTS \n \l

.NET Core SDK (reflecting any global.json):
 Version:   3.1.301
 Commit:    7feb845744

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  18.04
 OS Platform: Linux
 RID:         ubuntu.18.04-x64
 Base Path:   /usr/share/dotnet/sdk/3.1.301/

Host (useful for support):
  Version: 3.1.5
  Commit:  65cd789777

.NET Core SDKs installed:
  3.1.301 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.App 3.1.5 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.1.5 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 56 (26 by maintainers)

Most upvoted comments

@juliusfriedman createdump is an external command-line app that lives right next to libcoreclr. Also, do you have a mount of the publish directory with the linux files from a Windows machine?

hoyosjs on Jul 6, 2020

See https://github.com/dotnet/diagnostics/blob/2e2e6775d46a31e44218d62657b4169fa13ac74d/documentation/debugging-coredump.md#L6

Given that you ran sudo ulimit -c unlimited before starting the failing process. You may be able to get a usable core dump by:

Set the coredump_filter for the process at least 0x3f. See http://man7.org/linux/man-pages/man5/core.5.html
running kill -9 67295.

If we had a core dump it might be useful to @josalem

sdmaclea on Jul 6, 2020

It’s not critical now. I’ll try using an 18.04 VM then.

hoyosjs on Jul 16, 2020

Stuck in ProcessLogQueue? Normal?

Yes. This code has a dedicated thread that sits in a loop accessing a blocking collection. It waits for the next item, takes it, writes it out, and goes back to waiting for the next item. It will spend most of its life blocked waiting for the next piece of data to be logged. When data is logged, the logger adds it to the blocking collection.

stephentoub on Jul 8, 2020

@josalem, Yes.

lldb installed.

What should I do next?

lldb -p 67295 process save-core ?

Architecture set to: x86_64-pc-linux.
(lldb) process save-core
error: 'process save-core' takes one arguments:
Usage: process save-core FILE
(lldb) process save-core core.dump
error: Failed to save core file for process: no ObjectFile plugins were able to save a core for this process
(lldb)

AzureUser@machine-t:~$ ls /tmp |grep dotnet-diagnostic-
dotnet-diagnostic-21971-15572610-socket
dotnet-diagnostic-36314-16903719-socket
AzureUser@machine-t:~$ ls -lahust /tmp |grep dotnet-diagnostic-
   0 srw-------  1 root      root         0 Jul  1 15:01 dotnet-diagnostic-36314-16903719-socket
   0 srw-------  1 root      root         0 Jul  1 11:19 dotnet-diagnostic-21971-15572610-socket

@josalem, @sdmaclea, @tommcdon anything else I can do?

juliusfriedman on Jul 6, 2020

@josalem

tommcdon on Jul 6, 2020