apm-agent-dotnet: When adding a label, the Span does not get sent

I think so @gregkalapos . Thank you very much for your help. We’ll be in touch if there are any related issues that we discover.

Regards,

Steve

@fenneh thanks for the update - sounds good.

I can’t give you any explanation on how it’s come about, but your explanation sounds reasonable.

Let me know if you managed to fix the issue.

From the agent’s point of view I’d say there is nothing to do, but if you need help we can transfer the issue to the APM Server team.

To shed a bit of light on this… I work with @SteveDunn.

I think I may know what’s going on but I’m not entirely sure how it’s come about.

We have multiple index templates for apm within elastic, only two of which are setting mappings. To allow others to follow I’ll use screenshots as opposed to json.

The index that the events are being saved to is apm-7.2.0-span so there’s two possible templates it could have used when it was created.

Here are the two index configurations (Naturally different)

apm-server : https://gist.github.com/fenneh/9d503554b919a44fba67023070d05a87#file-apm-server-json apm-7.2.0 : https://gist.github.com/fenneh/9d503554b919a44fba67023070d05a87#file-apm-7-2-0-json

I’m assuming this is falling apart due to the omission of

          "labels": {
            "type": "object"

Within apm-server - I’m not sure how elastic would chose what template to use if they both have the same order.

If I had to hazard a guess at what’s happened, the template apm-server is from an older version of apm server, an upgrade of apm server has happened and the template names have been changed.

I think the resolution for us would be to drop the apm-server template and force a new index to be created by altering the output index names on APM server itself (To allow the index to pick up the correct template).

I have experienced the same issue when adding Labels to a transaction - the transaction does not appear to end up in Elastic … the spans that were associated with that transaction (which I had not put any Labels into) do end up in Elastic.

Strangely, when I first experimented with APM transactions with labels were getting into Elastic. I am not very familiar with the APMServer (someone else in my company installed and maintains it) … not sure which version its at … but am wondering if it is due to a mismatch between the APMServer and this agent.

Question … I had not noticed the LogLevel configuration parameter! Where does this logging end up (ie is it in a file)?

I’m unable to reproduce this with the following small sample:

internal class Program
{
	public class LogPerformance
	{
		public T CaptureSpan<T>(Func<T> func)
		{
			return Elastic.Apm.Agent.Tracer.CurrentTransaction.CaptureSpan(
				"spanName",
				"spanType",
				span =>
				{
					span.Labels["foo"] = "bar";

					return func();
				},
				"spanSubType");
		}
	}

	private static void Main(string[] args)
	{

		var transaction = Agent.Tracer.StartTransaction("aa", "bb");
		var logPerf = new LogPerformance();
		logPerf.CaptureSpan<int>(() => 1+1);

		transaction.End();

		Thread.Sleep(150000);
	}
}

Here is the corresponding span with the label:

Can you maybe give us agent logs on trace level? Any additional info - like what type of app it is, is the transaction that should contain span there (so only the span is missing)?