istio: Misconfigured meshConfig fails silently
Is this the right place to submit this?
- This is not a security vulnerability or a crashing bug
- This is not a question about how to use Istio
Bug Description
A misconfigured meshConfig will fail (mostly) silently, making it hard to notice that the config won’t be configured as it will fall back to the default config.
To reproduce:
- Install Istio with
meshConfig.defaultConfig.proxyMetadata.ISTIO_META_DNS_CAPTURE: true(notetrueis boolean, not string) - Deploy workloads
- Send traffic
- Notice that the flag above is not set,
istio.*metrics are missing, etc.
This won’t raise any warning/error in the workload’s istio-proxy, but you can see this warn level log in istiod:
...
2024-01-25T11:14:27.754775Z info mesh networks configuration updated to: {
}
2024-01-25T11:14:27.755719Z warn failed to read mesh config from ConfigMap: failed reading MeshConfig config: failed to convert to proto. json: cannot unmarshal bool into Go value of type string. YAML:
accessLogFile: /dev/stdout
accessLogFormat: |
[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% "%UPSTREAM_TRANSPORT_FAILURE_REASON%" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME% traceID=%REQ(x-b3-traceid)% "%REQ(END-USER)%"
defaultConfig:
discoveryAddress: istiod.istio-system.svc:15012
proxyMetadata:
ISTIO_META_DNS_CAPTURE: true
...
It would be nice to
- surface this better, maybe at workload proxy level as well?
- log it as error instead or warning
What do you think?
Version
Istio 1.20.2
$ kubectl version
Client Version: v1.25.3+k3s1
Kustomize Version: v4.5.7
Server Version: v1.28.5+k3s1
$ helm version --short
v3.10.3+gv3.10.3 via helmfile
Additional Information
No response
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 15 (15 by maintainers)
Commits related to this issue
- istiod: improve error message when values.yaml is invalid Fixes https://github.com/istio/istio/issues/51071 Not ideal fix, longer term is in https://github.com/istio/istio/issues/48991 — committed to howardjohn/istio by howardjohn 2 years ago
- istiod: improve error message when values.yaml is invalid Fixes https://github.com/istio/istio/issues/51071 Not ideal fix, longer term is in https://github.com/istio/istio/issues/48991 — committed to howardjohn/istio by howardjohn 2 years ago
- istiod: improve error message when values.yaml is invalid (#51228) * istiod: improve error message when values.yaml is invalid Fixes https://github.com/istio/istio/issues/51071 Not ideal fix, longer... — committed to istio/istio by howardjohn 2 years ago
I think https://github.com/istio/istio/pull/53708 is just injection config, not mesh config?
Here we can do two things:
With these two hurdles, istio cannot flip from a valid meshconfig to an invalid mesh config