istio: Misconfigured meshConfig fails silently

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
  • This is not a question about how to use Istio

Bug Description

A misconfigured meshConfig will fail (mostly) silently, making it hard to notice that the config won’t be configured as it will fall back to the default config.

To reproduce:

  1. Install Istio with meshConfig.defaultConfig.proxyMetadata.ISTIO_META_DNS_CAPTURE: true (note true is boolean, not string)
  2. Deploy workloads
  3. Send traffic
  4. Notice that the flag above is not set, istio.* metrics are missing, etc.

This won’t raise any warning/error in the workload’s istio-proxy, but you can see this warn level log in istiod:

...
2024-01-25T11:14:27.754775Z	info	mesh networks configuration updated to: {

}
2024-01-25T11:14:27.755719Z	warn	failed to read mesh config from ConfigMap: failed reading MeshConfig config: failed to convert to proto. json: cannot unmarshal bool into Go value of type string. YAML:
accessLogFile: /dev/stdout
accessLogFormat: |
  [%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% "%UPSTREAM_TRANSPORT_FAILURE_REASON%" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME% traceID=%REQ(x-b3-traceid)% "%REQ(END-USER)%"
defaultConfig:
  discoveryAddress: istiod.istio-system.svc:15012
  proxyMetadata:
    ISTIO_META_DNS_CAPTURE: true
...

It would be nice to

  • surface this better, maybe at workload proxy level as well?
  • log it as error instead or warning

What do you think?

Version

Istio 1.20.2

$ kubectl version
Client Version: v1.25.3+k3s1
Kustomize Version: v4.5.7
Server Version: v1.28.5+k3s1

$ helm version --short
v3.10.3+gv3.10.3 via helmfile

Additional Information

No response

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 15 (15 by maintainers)

Commits related to this issue

Most upvoted comments

I think https://github.com/istio/istio/pull/53708 is just injection config, not mesh config?

Here we can do two things:

  1. in Bootstrap stop loading the meshConfiguration if it is invalid, and using the default meshConfig.
  2. Adding we validating webhook for meshConfig, and prevent updating invalid config.

With these two hurdles, istio cannot flip from a valid meshconfig to an invalid mesh config