istio: Istio ingressgateway MEM has linear growth and isn't auto-released when to create and delete 500 knative svc (consumes about 500k ~ 1M for 1 ksvc)

Bug description Istio ingressgateway MEM isn’t auto-released after create and delete 500 ksvc (consumes about 500k ~ 1M for 1 ksvc)

[ ] Configuration Infrastructure [ ] Docs [ ] Installation [ ] Networking [X] Performance and Scalability [ ] Policies and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure

Expected behavior Istio ingressgateway MEM is auto-released after delete 500 Knative services.

Steps to reproduce the bug

• created 500 knative services
• observed Istio ingressgateway MEM increases about 500M+
• deleted 500 knative services
• after restart ingressgateway pod, the mem is released

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

 istioctl version --remote
client version: 1.5.4
cluster-local-gateway version:
cluster-local-gateway version:
cluster-local-gateway version:
cluster-local-gateway version:
cluster-local-gateway version:
ingressgateway version: 1.5.4
ingressgateway version: 1.5.4
ingressgateway version: 1.5.4
pilot version: 1.5.4
pilot version: 1.5.4
pilot version: 1.5.4
pilot version: 1.5.4
pilot version: 1.5.4
pilot version: 1.5.4
data plane version: 1.5.4 (8 proxies)

How was Istio installed?

cat << EOF > ./istio-minimal-operator.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      proxy:
        autoInject: disabled
      useMCP: false
      # The third-party-jwt is not enabled on all k8s.
      # See: https://istio.io/docs/ops/best-practices/security/#configure-third-party-service-account-tokens
      jwtPolicy: first-party-jwt

  addonComponents:
    pilot:
      enabled: true
    prometheus:
      enabled: false

  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
      - name: cluster-local-gateway
        enabled: true
        label:
          istio: cluster-local-gateway
          app: cluster-local-gateway
        k8s:
          service:
            type: ClusterIP
            ports:
            - port: 15020
              name: status-port
            - port: 80
              name: http2
            - port: 443
              name: https
EOF

./istioctl manifest generate -f istio-minimal-operator.yaml \
--set values.gateways.istio-egressgateway.enabled=false \
--set values.gateways.istio-ingressgateway.sds.enabled=true \
--set values.gateways.istio-ingressgateway.autoscaleMin=3 \
--set values.gateways.istio-ingressgateway.autoscaleMax=6 \
--set values.pilot.autoscaleMin=3 \
--set values.pilot.autoscaleMax=6 \
--set hub=icr.io/ext/istio  > istio.yaml

  kubectl apply -f istio.yaml    // more visibility than istioctl manifest apply

Environment where bug was observed (cloud vendor, OS, etc) IBM Kubernetes Cluster

{
 "allocated": "173674696",
 "heap_size": "1449394176",
 "pageheap_unmapped": "143187968",
 "pageheap_free": "790110208",
 "total_thread_cache": "25870296"
}