serving: Istio ingress: duplicate listener 0.0.0.0_8081 found

/area networking

What version of Knative?

0.19.x

Expected Behavior

Istio ingress gateway with provided knative-istio-controller configuration exposes Knative services consistently.

Actual Behavior

After a Knative service is added to the knative-local-gateway, the istio-ingressgateway pod reports the following error, and the ingress never comes online:

warning	envoy config	gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) 0.0.0.0_8081: duplicate listener 0.0.0.0_8081 found
2

Istio listener config:

$ istioctl proxy-config listeners istio-ingressgateway-xxx.istio-system
ADDRESS PORT  MATCH DESTINATION
0.0.0.0 8081  ALL   Route: http.80
0.0.0.0 15021 ALL   Non-HTTP/Non-TCP
0.0.0.0 15090 ALL   Non-HTTP/Non-TCP

This is running on a “toy” dev cluster which gets spun up and torn down each day and has all state managed using GitOps via Flux. Twice now, after tweaking a bunch of random stuff (ports, deleting/recreating gateways, etc), I’ve gotten the ingress into a working state, but when the cluster comes back up the following day with the same config, it’s broken again.

When it works, the listener config looks like this:

$ istioctl proxy-config listeners istio-ingressgateway-86f88b6f6-plp4p.istio-system
ADDRESS PORT  MATCH DESTINATION
0.0.0.0 8080  ALL   Route: http.80
0.0.0.0 8081  ALL   Route: http.8081
0.0.0.0 15021 ALL   Non-HTTP/Non-TCP
0.0.0.0 15090 ALL   Non-HTTP/Non-TCP

As best I can tell, Istio is incorrectly merging the two gateways (default istio-ingressgateway and knative-local-gateway), but I’m not familiar enough with Istio/Knative Serving or their intersection to understand whether this is a problem with Knative’s use of Istio or an Istio bug.

Steps to Reproduce the Problem

k8s 1.17 Istio 1.7.4

knative-istio-controller.yaml (relevant sections only, should be all default):

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: knative-ingress-gateway
  namespace: knative-serving
  labels:
    serving.knative.dev/release: "v0.19.0"
    networking.knative.dev/ingress-provider: istio
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: cluster-local-gateway
  namespace: knative-serving
  labels:
    serving.knative.dev/release: "v0.19.0"
    networking.knative.dev/ingress-provider: istio
spec:
  selector:
    istio: cluster-local-gateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: knative-local-gateway
  namespace: knative-serving
  labels:
    serving.knative.dev/release: "v0.19.0"
    networking.knative.dev/ingress-provider: istio
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 8081
        name: http
        protocol: HTTP
      hosts:
        - "*"

---
apiVersion: v1
kind: Service
metadata:
  name: knative-local-gateway
  namespace: istio-system
  labels:
    serving.knative.dev/release: "v0.19.0"
    networking.knative.dev/ingress-provider: istio
spec:
  type: ClusterIP
  selector:
    istio: ingressgateway
  ports:
    - name: http2
      port: 80
      targetPort: 8081

Knative service:

---
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: kinesis-seldon-adapter
  namespace: polyaxon
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/minScale: "1"
    spec:
      containers:
        - image: ko://my-module
          imagePullPolicy: Always
          ports:
            - containerPort: 8080

Gist with partial Istio proxy config dump showing the duplicate listeners in the failing state and the correct listeners in the working state (again, both states stemming from the same config shown above): https://gist.github.com/jasonnance/885e9772370ac6e46924844646597943

This is part of a more complex workflow involving Knative Eventing, so let me know if I’ve left out anything relevant.

Thanks!