istio: ingress gateway is not updated when secret is removed

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

Hi,

I think I found an issue that Ingress Gateway is not updated when the secret is removed from kubernetes secrets. This issue seems to me as a bug.

The issue can be described as follows: For Simple TLS traffic through Ingress Gateway a secret with certificates is created and it is mounted to Ingress Gateway using Gateway resource. When a request is sent through the Ingress Gateway traffic is passing as expected. When the secret is deleted from Kubernetes secret, the traffic is expected to NOT PASS through the Ingress Gateway. However, in the tests I have observed that even though the secret is removed, the traffic is continuing to pass through the Ingress Gateway. The reason is that Istiod is not updating Ingress Gateway with the empty/not found secret.

The reproduction steps are as follows:

1. Create secret kubectl create -n istio-system secret tls httpbin-credential --key=httpbin.example.com.key --cert=httpbin.example.com.crt
2. Apply httpbin

apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
      annotations:
        sidecar.istio.io/inject: "true"
    spec:
      serviceAccountName: httpbin
      imagePullSecrets:
      - name: armdocker
      containers:
      - image: docker.io/kong/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80

1. Apply the following Gateway and VirtualService Resources

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin-gateway
spec:
  selector:
    istio: ingressgateway # use istio default ingress gateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: httpbin-credential
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
spec:
  hosts:
  - "*"
  gateways:
  - httpbin-gateway
  http:
  - route:
    - destination:
        port:
          number: 8000
        host: httpbin

1. The “httpbin-credential” secret is in active status in Ingress Gateway

istioctl pc secrets istio-ingressgateway-75547ff498-jgh5g
RESOURCE NAME                       TYPE           STATUS     VALID CERT     SERIAL NUMBER                               NOT AFTER                NOT BEFORE
default                             Cert Chain     ACTIVE     true           33324975111525841046938231364134988081      2023-06-14T09:15:54Z     2023-06-13T09:13:54Z
kubernetes://httpbin-credential     Cert Chain     ACTIVE     true           0                                           2024-06-12T09:27:13Z     2023-06-13T09:27:13Z
ROOTCA                              CA             ACTIVE     true           278969411461750071063118506127200108662     2123-04-09T09:38:18Z     2023-05-03T09:38:18Z

1. Send a curl request through the Ingress Gateway curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" --cacert certs/example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418" See that the request is successful.
2. Delete secret kubectl delete secret -n istio-system httpbin-credential
3. The log in Istiod shows

2023-06-13T09:31:50.559584Z     info    ads     Push debounce stable[12] 1 for config Secret/istio-system/httpbin-credential: 100.232554ms since last change, 100.232347ms since last push, full=false
2023-06-13T09:31:50.559699Z     info    ads     XDS: Incremental Pushing:2023-06-13T09:16:07Z/5 ConnectedEndpoints:2 Version:2023-06-13T09:16:07Z/5
2023-06-13T09:31:50.564708Z     warn    ads     failed to fetch key and certificate for kubernetes://httpbin-credential: secret istio-system/httpbin-credential not found

1. The “httpbin-credential” secret is STILL in active status in Ingress Gateway.

istioctl pc secrets istio-ingressgateway-75547ff498-jgh5g
RESOURCE NAME                       TYPE           STATUS     VALID CERT     SERIAL NUMBER                               NOT AFTER                NOT BEFORE
default                             Cert Chain     ACTIVE     true           33324975111525841046938231364134988081      2023-06-14T09:15:54Z     2023-06-13T09:13:54Z
kubernetes://httpbin-credential     Cert Chain     ACTIVE     true           0                                           2024-06-12T09:27:13Z     2023-06-13T09:27:13Z
ROOTCA                              CA             ACTIVE     true           278969411461750071063118506127200108662     2123-04-09T09:38:18Z     2023-05-03T09:38:18Z

1. Send curl request as in the step 5, it will be seen that it is still successful. `

Moreover, I see that problem when the content of secret is removed. In that case the log gives me the following: warn ads failed to fetch key and certificate for kubernetes://httpbin-credential: found keys "tls.crt" and "tls.key", but they were empty

I expected that when the secret is removed the mounted secret from Ingress Gateway would be removed as well and Ingress Gateway does not letting the traffic through the Ingress Gateway. I am wondering if that is the expected behavior of Istio or is it a bug that needs to be resolved.

Best regards

Version

$ istioctl version
client version: 1.17.2
control plane version: 1.17.2
data plane version: 1.17.2 (2 proxies)

Additional Information

No response

Affected product area

• Ambient
• Docs
• Installation
• Networking
• Performance and Scalability
• Extensions and Telemetry
• Security
• Test and Release
• User Experience
• Developer Infrastructure
• Upgrade
• Multi Cluster
• Virtual Machine
• Control Plane Revisions