istio: Istio-Proxy returning UH / no healthy upstream at startup for STRICT_DNS clusters

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
  • This is not a question about how to use Istio

Bug Description

We have around 3-4k STRICT_DNS clusters (via ServiceEntries), and observed during startup istio-proxy returns a bunch of UH / no healthy upstream, which subsequently subside.

Our assumption is - Given the large amount of STRICT_DNS clusters, istio-proxy is taking time to complete the DNS resolution, and until then it doesn’t show the endpoints for such clusters (checked with istioctl proxy-config endpoints command), and hence returns a UH / no healthy upstream.

Istio-proxy should not call itself ready if it is not able to handle calls to STRICT_DNS clusters, as it has not completed DNS resolution for them.

Questions:

  1. Can istio-proxy ensure all DNS resolutions are complete before taking any traffic?
  2. Can it do runtime DNS resolution if it was not already done, and avoid UH / no healthy upstream errors?

Version

$ istioctl version
client version: 1.17.4
control plane version: 1.17-dca7c
data plane version: 1.17-dca7c (11 proxies)
╰─$ k version                                                                                                                                                                       11:03AM
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:47:38Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"25+", GitVersion:"v1.25.16-eks-8cb36c9", GitCommit:"3a3ea80e673d7867f47bdfbccd4ece7cb5f4a83a", GitTreeState:"clean", BuildDate:"2023-11-22T21:53:22Z", GoVersion:"go1.20.10", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.27) and server (1.25) exceeds the supported minor version skew of +/-1

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 41 (40 by maintainers)

Most upvoted comments

No, I think its just based on the interval not “on demand” like this.