kserve: prev revision is not removed when using inference logger

/kind bug

What steps did you take and what happened: add annotation on isvc. when patching isvc, new revision is created and i expect prev revision is not removed [A clear and concise description of what the bug is.]

What did you expect to happen: prev revision should be removed.

What’s the InferenceService yaml: [To help us debug please run kubectl get isvc $name -n $namespace -oyaml and paste the output]

apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: inferece-service-78
  namespace: default
  annotations:
    prometheus.io/scrape: "true"
    queue.sidecar.serving.knative.dev/resourcePercentage: "10"
spec:
  predictor:
    minReplicas: 1
    scaleTarget: 4
    canaryTrafficPercent: 100
    logger:
        mode: all
        url: http://message-dumper.default/
    containers:
    - name: 'kserve-container'
      image: IMAGE
      resources:
          requests:
            cpu: 400m

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment: tested on quick start version of 0.10 and 0.11 on homepage

• Istio Version:
• Knative Version:
• KServe Version:
• Kubeflow version:
• Cloud Environment:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
• Minikube/Kind version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):