istio: Listeners missing for service entries with TCP ports and multiple addresses [1.22]

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
  • This is not a question about how to use Istio

Bug Description

After upgrading from 1.21 to 1.22 listeners are missing for second and consecutive addresses listed in addresses block in service entry definition when resolution is NONE and protocol is TCP.

If I change protocol from TCP to TLS additional listeners get added to match the number of hosts specified in the spec but only the first address entry is used.

Version

$ istioctl version
client version: 1.22.1
control plane version: 1.22.1
data plane version: 1.22.1 (18 proxies)

$ kubectl version
Client Version: v1.29.0-eks-5e0fdde
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0-eks-036c24b

$ helm version --short
v3.14.3+gf03cc04

Additional Information

Istio 1.21 and Kubernetes 1.29

Added this for comparison. Configuration is taken from an identical cluster that runs Istio 1.21 and Kubernetes 1.29. As you can see three listeners have been created - one for each address entry. The sample-app can communicate with the external service (all three AWS MSK brokers).

$ kubectl --context=cluster-02 get serviceentry -n sample-ns msk-serviceentry -o yaml | yq e '.spec'
addresses:
  - 10.123.8.192/28
  - 10.123.8.208/28
  - 10.123.8.224/28
exportTo:
  - .
hosts:
  - b-1.msk02.fedcba.b12.kafka.us-east-1.amazonaws.com
  - b-2.msk02.fedcba.b12.kafka.us-east-1.amazonaws.com
  - b-3.msk02.fedcba.b12.kafka.us-east-1.amazonaws.com
location: MESH_EXTERNAL
ports:
  - name: tcp
    number: 9092
    protocol: TCP
  - name: tcp-2
    number: 9098
    protocol: TCP
resolution: NONE

$ istioctl --context=cluster-02 proxy-config listener sample-app-6f45cbb67d-2xt62.sample-ns --port 9092
ADDRESSES PORT MATCH                 DESTINATION
0.0.0.0   9092 Addr: 10.123.8.192/28 Cluster: outbound|9092||b-1.msk02.fedcba.b12.kafka.us-east-1.amazonaws.com
0.0.0.0   9092 Addr: 10.123.8.208/28 Cluster: outbound|9092||b-1.msk02.fedcba.b12.kafka.us-east-1.amazonaws.com
0.0.0.0   9092 Addr: 10.123.8.224/28 Cluster: outbound|9092||b-1.msk02.fedcba.b12.kafka.us-east-1.amazonaws.com
0.0.0.0   9092 ALL                   Cluster: outbound|9092||b-1.msk02.fedcba.b12.kafka.us-east-1.amazonaws.com

$ istioctl --context=cluster-02 proxy-config listener sample-app-6f45cbb67d-2xt62.sample-ns --port 9098
ADDRESSES PORT MATCH                 DESTINATION
0.0.0.0   9098 Addr: 10.123.8.192/28 Cluster: outbound|9098||b-1.msk02.fedcba.b12.kafka.us-east-1.amazonaws.com
0.0.0.0   9098 Addr: 10.123.8.208/28 Cluster: outbound|9098||b-1.msk02.fedcba.b12.kafka.us-east-1.amazonaws.com
0.0.0.0   9098 Addr: 10.123.8.224/28 Cluster: outbound|9098||b-1.msk02.fedcba.b12.kafka.us-east-1.amazonaws.com
0.0.0.0   9098 ALL                   Cluster: outbound|9098||b-1.msk02.fedcba.b12.kafka.us-east-1.amazonaws.com

Istio 1.22 and Kubernetes 1.30 - TCP protocol

Only one listener gets created in this case. The sample-app can talk to only one MSK broker, other requests end up being BlackHole’d.

$ kubectl get serviceentry -n sample-ns msk-serviceentry -o yaml | yq e '.spec'
addresses:
  - 10.123.16.192/28
  - 10.123.16.208/28
  - 10.123.16.224/28
exportTo:
  - .
hosts:
  - b-1.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
  - b-2.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
  - b-3.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
location: MESH_EXTERNAL
ports:
  - name: msk-mtls-auth
    number: 9092
    protocol: TCP
  - name: msk-iam-auth
    number: 9098
    protocol: TCP
resolution: NONE

$ istioctl proxy-config listener sample-app-6fb986c8d5-7mm67.sample-ns --port 9092
ADDRESSES PORT MATCH                  DESTINATION
0.0.0.0   9092 ALL                    Non-HTTP/Non-TCP
0.0.0.0   9092 Addr: 10.123.16.192/28 Cluster: outbound|9092||b-1.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com

$ istioctl proxy-config listener sample-app-6fb986c8d5-7mm67.sample-ns --port 9098
ADDRESSES PORT MATCH                  DESTINATION
0.0.0.0   9098 ALL                    Non-HTTP/Non-TCP
0.0.0.0   9098 Addr: 10.123.16.192/28 Cluster: outbound|9098||b-1.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com

Istio 1.22 and Kubernetes 1.30 - TLS protocol

Not sure if this is relevant, since Istio seems to be using SNI only and probably ignores addresses block. The sample-app can communicate with all MSK brokers.

$ kubectl get serviceentry -n sample-ns msk-serviceentry -o yaml | yq e '.spec'
addresses:
  - 10.123.16.192/28
  - 10.123.16.208/28
  - 10.123.16.224/28
exportTo:
  - .
hosts:
  - b-1.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
  - b-2.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
  - b-3.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
location: MESH_EXTERNAL
ports:
  - name: msk-mtls-auth
    number: 9092
    protocol: TLS
  - name: msk-iam-auth
    number: 9098
    protocol: TLS
resolution: NONE

$ istioctl proxy-config listener sample-app-6fb986c8d5-7mm67.sample-ns --port 9092
ADDRESSES PORT MATCH                                                                                           DESTINATION
0.0.0.0   9092 ALL                                                                                             Non-HTTP/Non-TCP
0.0.0.0   9092 SNI: b-3.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com; Addr: 10.123.16.192/28 Cluster: outbound|9092||b-3.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
0.0.0.0   9092 SNI: b-2.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com; Addr: 10.123.16.192/28 Cluster: outbound|9092||b-2.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
0.0.0.0   9092 SNI: b-1.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com; Addr: 10.123.16.192/28 Cluster: outbound|9092||b-1.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com

$ istioctl proxy-config listener sample-app-6fb986c8d5-7mm67.sample-ns --port 9098
ADDRESSES PORT MATCH                                                                                           DESTINATION
0.0.0.0   9098 ALL                                                                                             Non-HTTP/Non-TCP
0.0.0.0   9098 SNI: b-3.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com; Addr: 10.123.16.192/28 Cluster: outbound|9098||b-3.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
0.0.0.0   9098 SNI: b-2.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com; Addr: 10.123.16.192/28 Cluster: outbound|9098||b-2.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com
0.0.0.0   9098 SNI: b-1.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com; Addr: 10.123.16.192/28 Cluster: outbound|9098||b-1.msk03.abcdef.a01.kafka.us-east-1.amazonaws.com

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 20 (16 by maintainers)

Commits related to this issue

Most upvoted comments

Wanted to followup on this. The 1.22 and 1.23 changes for this have been reverted.

There is a simple workaround:

apiVersion: networking.istio.io/v1
kind: Sidecar
metadata:
  name: sidecar
  namespace: default
spec:
  egress:
  - hosts:
    - '*/*'

(don’t create this if you already have a Sidecar).

We will forward fix this with https://github.com/istio/istio/pull/51776 once its ready in the near future

1.24 essentially made the default behavior that way, you can see the upgrade notes that explain the minor behavioral differences: https://istio.io/latest/news/releases/1.24.x/announcing-1.24/upgrade-notes/#sidecar-scoping-changes

@ashevtsov-wawa the bug fix will be released in 1.22.4, so I am closing the issue.

I put a PR up to fully unify these two codepaths: https://github.com/istio/istio/pull/51776. It fixes this case, though I am sure there are some weird edge cases I didn’t think of, will see how it goes through tests.

cc @hzxuzhonghu @ramaraochavali