istio: opentelemetry tracing breaks sidecar injection

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
  • This is not a question about how to use Istio

Bug Description

When deploying istio with helm and the following values for tracing:

global:
  proxy:
    tracer: opentelemetry

meshConfig:
  enableTracing: true
  defaultConfig:
    tracing: {}
  defaultProviders:
      tracing:
      - "opentelemetry"
  extensionProviders:
  - name: "opentelemetry"
    opentelemetry:
      service: otel-node-collector.kube-system.svc.cluster.local
      port: 4317

proxy:
  enableTracing: true

results in the following meshConfig that seem to track with the required settings for opentelemetry tracing:

data:
  mesh: |-
    defaultConfig:
      discoveryAddress: istiod.istio-system.svc:15012
      tracing: {}
    defaultProviders:
      tracing:
      - opentelemetry
    enablePrometheusMerge: true
    enableTracing: true
    extensionProviders:
    - name: opentelemetry
      opentelemetry:
        port: 4317
        service: otel-node-collector.kube-system.svc.cluster.local
    rootNamespace: istio-system
    trustDomain: cluster.local
  meshNetworks: 'networks: {}'

With this config in place, pods cannot be created. The logs for istiod show the follow

2024-01-09T08:40:35.239386Z     error   Pod injection failed: failed to run injection template: failed applying injection overlay: unmarshal patched pod: json: cannot unmarshal string into Go struct field Probe.spec.containers.readinessProbe.failureThreshold of type int32

And an event with the same text appears in the relevant replicatset or statefulset. Removing the global.proxy.tracer setting allows pods to start, but the resulting mesh.defaultConfig (combined with a Telemetry object), no longer results in traces being sent to the collector.

Version

istio:
client version: 1.20.0
control plane version: 1.20.1
data plane version: 1.20.1 (6 proxies)

kubernetes:
Client Version: v1.27.5-dispatcher
Kustomize Version: v5.0.1
Server Version: v1.27.3

Additional Information

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 18 (9 by maintainers)

Most upvoted comments

the following configuration work as expected:

helm install istiod istio/istiod -n istio-system --wait -f - <<EOF
global:
  proxy:
    tracer: none

meshConfig:
  enableTracing: true
  defaultConfig:
    tracing: {}
  defaultProviders:
    tracing:
      - "opentelemetry"
  extensionProviders:
    - name: "opentelemetry"
      opentelemetry:
        service: otel-node-collector.kube-system.svc.cluster.local
        port: 4317
EOF

I don’t think we can reasonably partially fail. We could fail and either not start Istiod at all, or fail with a more explicit injection message (“failed to read config” not “<obscure parsing error”). I’d lean towards the latter.