istio: istio-agent drops every xDS client connection after the first, blocking grpc-go upgrade beyond 1.65.0
Is this the right place to submit this?
- This is not a security vulnerability or a crashing bug
- This is not a question about how to use Istio
Bug Description
grpc-go post 1.65.0 introduces separate xDS client connections to istiod for each named grpc service connected to that were previously handled by a single xDS client. In our usage scenario, we follow the instructions in https://istio.io/latest/blog/2021/proxyless-grpc/#in-your-kubernetes-deployment having the grpc-agent enabled and provide settings in a bootstrap.json file.
Once we upgrade to the grpc-go client that uses multiple clients, all gRPC requests start failing due to the istio-agent side car dropping all xDS connections after the first (e.g. clients that use gRPC service a, b and c, only the first connection gets created successfully (see log below). We expect the istio-agent to proxy the additional xDS client connections instead of being dropped. The end result is that we’re blocked on upgrading the grpc-go client beyond 1.65.0 where the multiple xDS client connections are introduced.
{"level":"info","time":"2024-10-11T17:57:26.160505Z","scope":"xdsproxy","msg":"Initializing with upstream address \"istiod-1-22-4.istio-system.svc:15012\" and cluster \"Kubernetes\""}
{"level":"info","time":"2024-10-11T17:57:26.427983Z","scope":"cache","msg":"generated new workload certificate","resourceName":"default","latency":267051261,"ttl":86399572022083}
{"level":"info","time":"2024-10-11T17:57:26.428066Z","scope":"cache","msg":"Root cert has changed, start rotating root cert"}
{"level":"info","time":"2024-10-11T17:57:26.428100Z","scope":"cache","msg":"returned workload certificate from cache","ttl":86399571901173}
{"level":"info","time":"2024-10-11T17:57:26.428366Z","scope":"cache","msg":"returned workload trust anchor from cache","ttl":86399571634504}
{"level":"info","time":"2024-10-11T17:57:26.428802Z","scope":"cache","msg":"returned workload trust anchor from cache","ttl":86399571198514}
{"level":"info","time":"2024-10-11T17:57:33.536595Z","scope":"xdsproxy","msg":"connected to upstream XDS server: istiod-1-22-4.istio-system.svc:15012","id":1}
{"level":"warn","time":"2024-10-11T17:57:33.808855Z","scope":"xdsproxy","msg":"registered overlapping stream; closing previous"}
{"level":"warn","time":"2024-10-11T17:57:33.809498Z","scope":"xdsproxy","msg":"registered overlapping stream; closing previous"}
{"level":"info","time":"2024-10-11T17:57:33.815859Z","scope":"xdsproxy","msg":"connected to upstream XDS server: istiod-1-22-4.istio-system.svc:15012","id":2}
{"level":"info","time":"2024-10-11T17:57:33.816460Z","scope":"xdsproxy","msg":"connected to upstream XDS server: istiod-1-22-4.istio-system.svc:15012","id":3}
{"level":"warn","time":"2024-10-11T17:57:34.817348Z","scope":"xdsproxy","msg":"registered overlapping stream; closing previous"}
{"level":"warn","time":"2024-10-11T17:57:34.818000Z","scope":"xdsproxy","msg":"registered overlapping stream; closing previous"}
Version
$ istioctl version
client version: 1.22.2
istiod version: 1.22.4
data plane version: 1.22.4 (247 proxies)
$ kubectl version
Client Version: v1.29.7
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.8-gke.1096000
Additional Information
Google GKE
Original grpc-go ticket when upgrade broke gRPC calls : https://github.com/grpc/grpc-go/issues/7691
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 1
- Comments: 33 (14 by maintainers)
This issue is blocking folks from upgrading to the latest grpc versions. Is there an update worth sharing here? Or maybe a workaround that people can use?
Thanks.
Would something like https://github.com/grpc/proposal/pull/492 work for Istio?
Actually, I suspect if you can figure out how to make gRPC C++ connect to istiod xDS server, we can just take that solution verbatim into Istio as a way to skip xDS proxying to save resources and also benefit from advanced xDS features you are adding. It’s either that or using Envoy’s gRPC client which also probably needs some features.
We can’t revert the changes on the gRPC side, since they’re an inherent part of supporting xDS fallback.
I don’t actually see why making Istio’s proxy behave correctly is so complicated or risky. And I think that this decision is essentially the same as saying that Istio will not support proxyless gRPC. That’s obviously your call to make, but it seems fairly short-sighted.
Its not really a bug in either project per-se but the integration between them. Istio has an XDS proxy that supports a single XDS connection, because it was designed to be used with clients which only opens a single connection (which was everyone at the time of writing).
gRPC support was then added to Istio as a super experimental thing primarily driven by one person without much adoption or maintenance around it. As, essentially, a hack around some issues that I have since forgotten the details, one of the setups with gRPC is to use the XDS proxy. Its not the only one - there is
grpc-simplethat does not, itsgrpc-agentthat does use it.gRPC then changed to open multiple connections which then breaks things.
Probably the best path forward is to get gRPC working without utilizing the xDS proxy at all, rather than making the xDS proxy substantially more complex to support a niche case that most users are not impacted by.
Note that this is not just an issue for grpc-go; it will be an issue for all gRPC languages as well. This stems from the fact that as part of the xDS fallback design (see gRFC A71), we’ve switched to using a separate ADS stream for each data plane channel target, and that’s a cross-language design.
(At the moment, gRPC C-core will not have this problem, because our subchannel sharing mechanism means that even though we still have separate ADS streams, we’ll still share the underlying connection. But we may need to disable subchannel sharing in this case if we run into MAX_CONCURRENT_STREAMS limits.)