istio: Istio instanced traces don't appear to be valid
Consider the following application flow:
service-a -> service-b -> service-c -> service-d
When we just rely on istio to start traces, we only ever see single traces (a->b, b->c, or c->d), we never see a->b->c->d in Jaeger. B is correctly forwarding the trace headers to C, and C to D.
Digging into this a little more, it looks like envoy is not correctly setting up the trace and the span. If I dump out the headers on service-b, you can see the only these tracing related headers that were added by envoy on service A.
"x-b3-traceid": "fcb5121ea0e8d5fc",
"x-b3-spanid": "fcb5121ea0e8d5fc",
"x-request-id": "740751d64d2d1ef9e328cc43e60e97fd"
I find it very odd that the traceid and spanid here are the same.
If I don’t rely on istio to instantiate the trace, and a zipkin library in service-a to correctly instantiate a trace and span, and dump the headers in service b, I see:
"x-b3-traceid": "b00fe06a1b060c67",
"x-b3-spanid": "3f0783fb8b035aa4",
"x-b3-parentspanid": "564c4c8ee127f144",
"x-b3-flags": "0",
"x-request-id": "fbf97ddb-4728-9639-ad46-fe6ed86029d2"
Having service A instantiate the trace in this manner correctly gives me an a->b->c->d trace in Jaeger.
Obviously we don’t want to have to do this, the whole idea is to have istio instantiate traces when they’re not present rather than rely on client libraries in the applications.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 1
- Comments: 67 (52 by maintainers)
I’ve tried doing distributed tracing a few times (different languages, rest+grpc, rest-only, grpc-only) - each resulting in a similar issue - all requests show up in the trace, but the inheritance isn’t right.
Of course it works right for the bookinfo app, and I’m parroting the same headers to child requests that the sample app is, so I’m at a loss as to what I’m doing wrong.
What makes it weirder is that it tends to be “more correct” if you don’t go through the ingress gateway to start the trace.
I’m using istio 1.0.5, no mtls enabled, but obviously tracing is enabled. Running k8s 1.13.3 At this point I’ve created some dummy services, with a deployment yaml so I can easily share them, if you have any time on slack (although I would need an invite seemingly)
Mike
😃