istio: Istio Operator does not uninstall components

Bug Description

In short: Istio Operator does not uninstal Istio resources when I delete an IstioOperator CR. It constantly reporting Reconciling IstioOperator and Deleting IstioOperator istio-control-plane-1-11-2.

Setup:

• Created cluster with kind utility

kind create cluster --name istio-test --config ./kind_istio-test-1.20.yaml

where kind_istio-test-1.20.yaml containes the following

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  image: kindest/node:v1.20.2@sha256:8f7ea6e7642c0da54f04a7ee10431549c0257315b3a634f6ef2fecaaedb19bab
- role: worker
  image: kindest/node:v1.20.2@sha256:8f7ea6e7642c0da54f04a7ee10431549c0257315b3a634f6ef2fecaaedb19bab

• Installed Istio Operator

./istio-1.11.2/bin/istioctl operator init --watchedNamespaces=istio-system --revision=1-11-2

• Installed Control Plane and Gateways IstioOperator CRs

kubectl apply -f - <<EOF
---
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: istio-control-plane-1-11-2
spec:
  revision: "1-11-2"
  profile: minimal
EOF


kubectl apply -f - <<EOF
---
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: gateways
spec:
  profile: empty
  revision: "1-11-2"
  components:
    egressGateways:
      - name: istio-egressgateway
        enabled: true
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
EOF

• As a result I saw istiod and gateways pods in istio-system namespace

$ kubectl get pods -n istio-system
NAME                                    READY   STATUS    RESTARTS   AGE
istio-egressgateway-698f9dc949-5lxbg    1/1     Running   0          37m
istio-ingressgateway-55d9556456-xnpdh   1/1     Running   0          37m
istiod-1-11-2-64d6c964b9-27w8l          1/1     Running   0          37m

** IstioOperator CRs deletion ** When I try to remove one of this CRs then kubectl hangs

$ kubectl delete istiooperators.install.istio.io -n istio-system istio-control-plane-1-11-2 
istiooperator.install.istio.io "istio-control-plane-1-11-2" deleted

In fact the CR and all related Kubernetes resources are not deleted. During this time I see constantly repeating Reconciling IstioOperator and Deleting IstioOperator istio-control-plane-1-11-2 messages in istio-operator pod logs. For example:

...
2021-09-03T15:34:41.765435Z	info	installer	Deleting IstioOperator istio-control-plane-1-11-2
2021-09-03T15:34:42.098423Z	info	installer	Reconciling IstioOperator
2021-09-03T15:34:42.098804Z	info	installer	Deleting IstioOperator istio-control-plane-1-11-2
2021-09-03T15:34:42.753390Z	info	installer	Reconciling IstioOperator
2021-09-03T15:34:42.753862Z	info	installer	Deleting IstioOperator istio-control-plane-1-11-2
2021-09-03T15:34:44.047766Z	info	installer	Reconciling IstioOperator
2021-09-03T15:34:44.048132Z	info	installer	Deleting IstioOperator istio-control-plane-1-11-2
2021-09-03T15:34:46.632213Z	info	installer	Reconciling IstioOperator
2021-09-03T15:34:46.632669Z	info	installer	Deleting IstioOperator istio-control-plane-1-11-2
2021-09-03T15:34:51.768007Z	info	installer	Reconciling IstioOperator
2021-09-03T15:34:51.768424Z	info	installer	Deleting IstioOperator istio-control-plane-1-11-2
...

With Istio 1.10.4 I also can not uninstall components. kubectl can successfully delete IstioOperator CRs and it does not hang, but related Kubernetes resources still exist and the following messages appear in istio-operator pod logs

2021-09-03T16:08:21.051523Z	info	installer	Reconciling IstioOperator
2021-09-03T16:08:21.051771Z	info	installer	Deleting IstioOperator istio-control-plane-1-10-4
2021-09-03T16:08:21.070348Z	info	installer	API server conflict during finalizer removal, retrying.
2021-09-03T16:08:21.078207Z	info	installer	Reconciling IstioOperator

Version

$ istioctl version
client version: 1.11.2
control plane version: 1.11.2
data plane version: 1.11.2 (2 proxies)

$ kubectl version --short
Client Version: v1.19.7
Server Version: v1.20.2

The same behavior is when server version is v1.19.7

$ kind version
kind v0.10.0 go1.15.7 linux/amd64

Additional Information

bug-report.zip