kubernetes: Controller-manager fails to successfully start when converting a cluster from dual-stack to single-stack

What happened:

Tried to downgrade a running dual stack cluster to single stack by modifying service-cluster-ip-range and cluster-cidr parameters to have only one CIDR (both parameters had ipv4,ipv6 CIDR ,changed it to have only ipv4 CIDR). After making this change controller-manager fails to start successfully. The below error is seen in the controller manager logs:

controllermanager.go:226] error starting controllers: node:ccd-focal-clus2-1 has an allocated cidr: 2001:db8:1234:5678:8:3::/120 at index:1 that does not exist in cluster cidrs configuration

node.Spec.PodCIDRs :

spec:
    podCIDR: 10.244.2.0/28
    podCIDRs:
    - 10.244.2.0/28
    - 2001:db8:1234:5678:8:3::/120

What you expected to happen:

Successful downgrade from dual to single stack cluster

How to reproduce it (as minimally and precisely as possible):

In a running dual stack cluster edit the parameters “service-cluster-ip-range” and “cluster-cidr” in kube-controller-manager.yaml and kube-apiserver.yaml to have only one CIDR(keep the primary cidr, delete the secondary). Also, delete parameters “node-cidr-mask-size-ipv4” and “node-cidr-mask-size-ipv6” if present.

Anything else we need to know?:

kube-controller-manager logs: 4_downgrade_dualstack.txt

Environment:

• Kubernetes version (use kubectl version): v1.22.0-alpha.2
• Cloud provider or hardware configuration: local
• OS (e.g: cat /etc/os-release): “Ubuntu 20.04.2 LTS”
• Kernel (e.g. uname -a): Linux ccd-focal-clus2-1 5.4.0-66-generic
• Install tools:
• Network plugin and version (if this is a network-related bug): calico
• Others: