istio: Listeners missing for service entries with TCP ports and multiple addresses [1.22]
Is this the right place to submit this?
- This is not a security vulnerability or a crashing bug
- This is not a question about how to use Istio
Bug Description
After upgrading from 1.21 to 1.22 listeners are missing for second and consecutive addresses listed in addresses block in service entry definition when resolution is NONE and protocol is TCP.
If I change protocol from TCP to TLS additional listeners get added to match the number of hosts specified in the spec but only the first address entry is used.
Version
$ istioctl version
client version: 1.22.1
control plane version: 1.22.1
data plane version: 1.22.1 (18 proxies)
$ kubectl version
Client Version: v1.29.0-eks-5e0fdde
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0-eks-036c24b
$ helm version --short
v3.14.3+gf03cc04
Additional Information
Istio 1.21 and Kubernetes 1.29
Added this for comparison. Configuration is taken from an identical cluster that runs Istio 1.21 and Kubernetes 1.29. As you can see three listeners have been created - one for each address entry. The sample-app can communicate with the external service (all three AWS MSK brokers).
$ kubectl --context=cluster-02 get serviceentry -n sample-ns msk-serviceentry -o yaml | yq e '.spec'
addresses:
- 10.123.8.192/28
- 10.123.8.208/28
- 10.123.8.224/28
exportTo:
- .
hosts:
- b-1.msk02.fedcba.b12.kafka.us-east-1.amazonaws.com
- b-2.msk02.fedcba.b12.kafka.us-east-1.amazonaws.com
- b-3.msk02.fedcba.b12.kafka.us-east-1.amazonaws.com
location: MESH_EXTERNAL
ports:
- name: tcp
number: 9092
protocol: TCP
- name: tcp-2
number: 9098
protocol: TCP
resolution: NONE
$ istioctl --context=cluster-02 proxy-config listener sample-app-6f45cbb67d-2xt62.sample-ns --port 9092
ADDRESSES PORT MATCH DESTINATION
0.0.0.0 9092 Addr: 10.123.8.192/28 Cluster: outbound|9092||b-1.msk02.fedcba.b12.kafka.us-east-1.amazonaws.com
0.0.0.0 9092 Addr: 10.123.8.208/28 Cluster: outbound|9092||b-1.msk02.fedcba.b12.kafka.us-east-1.amazonaws.com
0.0.0.0 9092 Addr: 10.123.8.224/28 Cluster: outbound|9092||b-1.msk02.fedcba.b12.kafka.us-east-1.amazonaws.com
0.0.0.0 9092 ALL Cluster: outbound|9092||b-1.msk02.fedcba.b12.kafka.us-east-1.amazonaws.com
$ istioctl --context=cluster-02 proxy-config listener sample-app-6f45cbb67d-2xt62.sample-ns --port 9098
ADDRESSES PORT MATCH DESTINATION
0.0.0.0 9098 Addr: 10.123.8.192/28 Cluster: outbound|9098||b-1.msk02.fedcba.b12.kafka.us-east-1.amazonaws.com
0.0.0.0 9098 Addr: 10.123.8.208/28 Cluster: outbound|9098||b-1.msk02.fedcba.b12.kafka.us-east-1.amazonaws.com
0.0.0.0 9098 Addr: 10.123.8.224/28 Cluster: outbound|9098||b-1.msk02.fedcba.b12.kafka.us-east-1.amazonaws.com
0.0.0.0 9098 ALL Cluster: outbound|9098||b-1.msk02.fedcba.b12.kafka.us-east-1.amazonaws.com
Istio 1.22 and Kubernetes 1.30 - TCP protocol
Only one listener gets created in this case. The sample-app can talk to only one MSK broker, other requests end up being BlackHole’d.
$ kubectl get serviceentry -n sample-ns msk-serviceentry -o yaml | yq e '.spec'
addresses:
- 10.123.16.192/28
- 10.123.16.208/28
- 10.123.16.224/28
exportTo:
- .
hosts:
- b-1.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
- b-2.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
- b-3.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
location: MESH_EXTERNAL
ports:
- name: msk-mtls-auth
number: 9092
protocol: TCP
- name: msk-iam-auth
number: 9098
protocol: TCP
resolution: NONE
$ istioctl proxy-config listener sample-app-6fb986c8d5-7mm67.sample-ns --port 9092
ADDRESSES PORT MATCH DESTINATION
0.0.0.0 9092 ALL Non-HTTP/Non-TCP
0.0.0.0 9092 Addr: 10.123.16.192/28 Cluster: outbound|9092||b-1.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
$ istioctl proxy-config listener sample-app-6fb986c8d5-7mm67.sample-ns --port 9098
ADDRESSES PORT MATCH DESTINATION
0.0.0.0 9098 ALL Non-HTTP/Non-TCP
0.0.0.0 9098 Addr: 10.123.16.192/28 Cluster: outbound|9098||b-1.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
Istio 1.22 and Kubernetes 1.30 - TLS protocol
Not sure if this is relevant, since Istio seems to be using SNI only and probably ignores addresses block. The sample-app can communicate with all MSK brokers.
$ kubectl get serviceentry -n sample-ns msk-serviceentry -o yaml | yq e '.spec'
addresses:
- 10.123.16.192/28
- 10.123.16.208/28
- 10.123.16.224/28
exportTo:
- .
hosts:
- b-1.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
- b-2.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
- b-3.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
location: MESH_EXTERNAL
ports:
- name: msk-mtls-auth
number: 9092
protocol: TLS
- name: msk-iam-auth
number: 9098
protocol: TLS
resolution: NONE
$ istioctl proxy-config listener sample-app-6fb986c8d5-7mm67.sample-ns --port 9092
ADDRESSES PORT MATCH DESTINATION
0.0.0.0 9092 ALL Non-HTTP/Non-TCP
0.0.0.0 9092 SNI: b-3.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com; Addr: 10.123.16.192/28 Cluster: outbound|9092||b-3.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
0.0.0.0 9092 SNI: b-2.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com; Addr: 10.123.16.192/28 Cluster: outbound|9092||b-2.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
0.0.0.0 9092 SNI: b-1.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com; Addr: 10.123.16.192/28 Cluster: outbound|9092||b-1.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
$ istioctl proxy-config listener sample-app-6fb986c8d5-7mm67.sample-ns --port 9098
ADDRESSES PORT MATCH DESTINATION
0.0.0.0 9098 ALL Non-HTTP/Non-TCP
0.0.0.0 9098 SNI: b-3.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com; Addr: 10.123.16.192/28 Cluster: outbound|9098||b-3.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
0.0.0.0 9098 SNI: b-2.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com; Addr: 10.123.16.192/28 Cluster: outbound|9098||b-2.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
0.0.0.0 9098 SNI: b-1.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com; Addr: 10.123.16.192/28 Cluster: outbound|9098||b-1.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 20 (16 by maintainers)
Commits related to this issue
- pilot: unified sidecar scope creation For https://github.com/istio/istio/issues/51747 — committed to howardjohn/istio by howardjohn 2 years ago
- pilot: unified sidecar scope creation For https://github.com/istio/istio/issues/51747 — committed to howardjohn/istio by howardjohn 2 years ago
- pilot: unified sidecar scope creation For https://github.com/istio/istio/issues/51747 — committed to jewertow/upstream-istio by howardjohn 2 years ago
- pilot: unified sidecar scope creation For https://github.com/istio/istio/issues/51747 — committed to howardjohn/istio by howardjohn 2 years ago
- pilot: unified sidecar scope creation For https://github.com/istio/istio/issues/51747 — committed to howardjohn/istio by howardjohn 2 years ago
Wanted to followup on this. The 1.22 and 1.23 changes for this have been reverted.
There is a simple workaround:
(don’t create this if you already have a Sidecar).
We will forward fix this with https://github.com/istio/istio/pull/51776 once its ready in the near future
1.24 essentially made the default behavior that way, you can see the upgrade notes that explain the minor behavioral differences: https://istio.io/latest/news/releases/1.24.x/announcing-1.24/upgrade-notes/#sidecar-scoping-changes
@ashevtsov-wawa the bug fix will be released in 1.22.4, so I am closing the issue.
I put a PR up to fully unify these two codepaths: https://github.com/istio/istio/pull/51776. It fixes this case, though I am sure there are some weird edge cases I didn’t think of, will see how it goes through tests.
cc @hzxuzhonghu @ramaraochavali