opentelemetry-python: Error while exporting span batch: end time is None

Describe your environment

Python version: 3.8
Library versions:

opentelemetry-api==1.3.0
opentelemetry-exporter-datadog==0.22b0
opentelemetry-exporter-jaeger==1.3.0
opentelemetry-exporter-jaeger-proto-grpc==1.3.0
opentelemetry-exporter-jaeger-thrift==1.3.0
opentelemetry-exporter-otlp==1.3.0
opentelemetry-exporter-otlp-proto-grpc==1.3.0
opentelemetry-instrumentation==0.22b0
opentelemetry-instrumentation-tornado==0.22b0
opentelemetry-proto==1.3.0
opentelemetry-sdk==1.3.0
opentelemetry-semantic-conventions==0.22b0
opentelemetry-util-http==0.22b0

Run using Kubernetes in AWS

Steps to reproduce

Instrument Python code with Opentelemetry
Add Datadog (or Jaeger) exporter
Deploy everything and run the tracing instrumented system
Sometimes error in exporter appears

What is the expected behavior? I expect to see no errors when span is exported in exporter.

What is the actual behavior? In production, the following error happens sometimes

Exception while exporting Span batch.
Traceback (most recent call last):
  File "/app/.venv/lib/python3.8/site-packages/opentelemetry/exporter/datadog/spanprocessor.py", line 173, in export
    self.span_exporter.export(self.traces[trace_id])  # type: ignore
  File "/app/.venv/lib/python3.8/site-packages/opentelemetry/exporter/datadog/exporter.py", line 110, in export
    datadog_spans = self._translate_to_datadog(spans)
  File "/app/.venv/lib/python3.8/site-packages/opentelemetry/exporter/datadog/exporter.py", line 151, in _translate_to_datadog
    datadog_span.duration_ns = span.end_time - span.start_time
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

Additional context I’m not sure why end time is somehow not timestamped and comes out as None. Do you have any ideas why end timestamp is missed somehow?

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 21 (10 by maintainers)

Most upvoted comments

Also, I assume the reason we don’t see the entire stack trace is that the export process is happening in another thread so the stack trace shifts, we don’t see exactly where the errant span comes from.

Correct.

It seems like though at a minimum we would want to gate against this case, right? Apparently there is a case where end_time will be None inside of these exporters, in the case where a span is not ended.

I’m concerned that we’ll end up silencing such issues and never find the real root cause. The underlying problem could be a symptom of a bigger issue so it is better we try to identify the source instead of silencing the error.

Maybe we still raise an exception but we could probably raise a more useful one to help debugging where the span came from.

This makes sense. If possible, I’d suggest to do this in your copy of otel DD exporter and see if you can log additional helpful info.

owais on Jan 12, 2022