istio: Lightstep tracing with caCertificates defined report missing spans

Bug description

On Istio 1.8.2, following this doc: https://istio.io/latest/docs/tasks/observability/distributed-tracing/lightstep/#deploy-istio-with-public-or-developer-mode-satellites

… leads to broken parent spans that worked correctly prior to the upgrade:

image

Our config before the upgrade was:

  values:
    global:
      enableTracing: true
      tracer:
        lightstep:
          address: [internal IP to satellite]
          secure: true
          cacertPath: /etc/lightstep/cacert.pem
          accessToken: [redacted]
      proxy:
        tracer: lightstep

Our config after the upgrade is:

  meshConfig:
    defaultConfig:
      tracing:
        tlsSettings:
          mode: "SIMPLE"
          caCertificates: "/etc/lightstep/cacert.pem"

  values:
    global:
      tracer:
        lightstep:
          address: [internal IP of satellite]
          accessToken: [redacted]
      proxy:
        tracer: lightstep

The CA cert is correctly mounted to the application pods in the mesh:

    Mounts:
      /etc/lightstep/ from lightstep-certs (ro)

And exec’ing into the istio-proxy container shows that /etc/lightstep indeed contains the correct file. Nothing shows up in the istio-proxy logs for the app pods.

However, this shows up in the ingressgateway pods:

2021-03-02T21:47:05.959141Z	error	cache	resource:file-root:/etc/lightstep/cacert.pem failed to generate secret for proxy from file: open /etc/lightstep/cacert.pem: no such file or directory
2021-03-02T21:47:05.959194Z	error	sds	resource:file-root:/etc/lightstep/cacert.pem Close connection. Failed to get secret for proxy "router~10.1.10.164~istio-ingressgateway-5b7475bbb6-547xf.istio-system~istio-system.svc.cluster.local" from secret cache: open /etc/lightstep/cacert.pem: no such file or directory
2021-03-02T21:47:05.959299Z	info	sds	resource:file-root:/etc/lightstep/cacert.pem connection is terminated: rpc error: code = Canceled desc = context canceled
2021-03-02T21:47:05.959527Z	warning	envoy config	StreamSecrets gRPC config stream closed: 2, open /etc/lightstep/cacert.pem: no such file or directory

[ ] Docs [ ] Installation [ ] Networking [ ] Performance and Scalability [x] Extensions and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure [ ] Upgrade

Expected behavior

All spans that correctly generated before the upgrade persist after the upgrade.

Version (include the output of istioctl version --remote and kubectl version --short and helm version --short if you used Helm)

1.8.2

How was Istio installed?

Profile generated from provided yaml

Environment where the bug was observed (cloud vendor, OS, etc)

AKS

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 16 (8 by maintainers)

Most upvoted comments

Thanks! I will add the doc.

I was able to mount the cert manually to the ingressgateway deployment and this time it indeed fixed the missing span issue.

Does the sidecar also report spans correctly now?

Yep, all spans are reported correctly after manually mounting the Lightstep cacert to the gateway