istio: Lightstep tracing with caCertificates defined report missing spans
Bug description
On Istio 1.8.2, following this doc: https://istio.io/latest/docs/tasks/observability/distributed-tracing/lightstep/#deploy-istio-with-public-or-developer-mode-satellites
… leads to broken parent spans that worked correctly prior to the upgrade:
Our config before the upgrade was:
values:
global:
enableTracing: true
tracer:
lightstep:
address: [internal IP to satellite]
secure: true
cacertPath: /etc/lightstep/cacert.pem
accessToken: [redacted]
proxy:
tracer: lightstep
Our config after the upgrade is:
meshConfig:
defaultConfig:
tracing:
tlsSettings:
mode: "SIMPLE"
caCertificates: "/etc/lightstep/cacert.pem"
values:
global:
tracer:
lightstep:
address: [internal IP of satellite]
accessToken: [redacted]
proxy:
tracer: lightstep
The CA cert is correctly mounted to the application pods in the mesh:
Mounts:
/etc/lightstep/ from lightstep-certs (ro)
And exec’ing into the istio-proxy container shows that /etc/lightstep indeed contains the correct file. Nothing shows up in the istio-proxy logs for the app pods.
However, this shows up in the ingressgateway pods:
2021-03-02T21:47:05.959141Z error cache resource:file-root:/etc/lightstep/cacert.pem failed to generate secret for proxy from file: open /etc/lightstep/cacert.pem: no such file or directory
2021-03-02T21:47:05.959194Z error sds resource:file-root:/etc/lightstep/cacert.pem Close connection. Failed to get secret for proxy "router~10.1.10.164~istio-ingressgateway-5b7475bbb6-547xf.istio-system~istio-system.svc.cluster.local" from secret cache: open /etc/lightstep/cacert.pem: no such file or directory
2021-03-02T21:47:05.959299Z info sds resource:file-root:/etc/lightstep/cacert.pem connection is terminated: rpc error: code = Canceled desc = context canceled
2021-03-02T21:47:05.959527Z warning envoy config StreamSecrets gRPC config stream closed: 2, open /etc/lightstep/cacert.pem: no such file or directory
[ ] Docs [ ] Installation [ ] Networking [ ] Performance and Scalability [x] Extensions and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure [ ] Upgrade
Expected behavior
All spans that correctly generated before the upgrade persist after the upgrade.
Version (include the output of istioctl version --remote
and kubectl version --short
and helm version --short
if you used Helm)
1.8.2
How was Istio installed?
Profile generated from provided yaml
Environment where the bug was observed (cloud vendor, OS, etc)
AKS
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 16 (8 by maintainers)
Thanks! I will add the doc.
Yep, all spans are reported correctly after manually mounting the Lightstep cacert to the gateway