cluster-api: Fail to move cluster with associated IPAddresses
What steps did you take and what happened?
I was trying to move a cluster from a bootstrap cluster to the new mgmt-cluster, which failed, because of the webhook.
logs:
Creating IPAddress="mgmt-cluster-control-plane-9p4r4" Namespace="default"
Object already exists, updating IPAddress="mgmt-cluster-control-plane-9p4r4" Namespace="default"
Retrying with backoff Cause="error updating \"ipam.cluster.x-k8s.io/v1alpha1, Kind=IPAddress\" default/mgmt-cluster-control-plane-9p4r4: admission webhook \"validation.ipaddress.ipam.cluster.x-k8s.io\" denied the request: spec: Forbidden: the spec of IPAddress is immutable
What did you expect to happen?
I expect the clusterctl move succeed
Cluster API version
v1.5.1
Kubernetes version
v1.26.7
Anything else you would like to add?
Error: [action failed after 10 attempts: error updating "ipam.cluster.x-k8s.io/v1alpha1, Kind=IPAddress" default/mgmt-cluster-control-plane-pwhfv: admission webhook "validation.ipaddress.ipam.cluster.x-k8s.io" denied the request: spec: Forbidden: the spec of IPAddress is immutable, action failed after 10 attempts: error updating "ipam.cluster.x-k8s.io/v1alpha1, Kind=IPAddress" default/mgmt-cluster-control-plane-6xk8h: admission webhook "validation.ipaddress.ipam.cluster.x-k8s.io" denied the request: spec: Forbidden: the spec of IPAddress is immutable, action failed after 10 attempts: error updating "ipam.cluster.x-k8s.io/v1alpha1, Kind=IPAddress" default/mgmt-cluster-control-plane-9p4r4: admission webhook "validation.ipaddress.ipam.cluster.x-k8s.io" denied the request: spec: Forbidden: the spec of IPAddress is immutable]
Label(s) to be applied
/kind bug One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.
About this issue
- Original URL
- State: open
- Created 9 months ago
- Comments: 22 (19 by maintainers)
Hi @mcbenjemaa!
Sorry for the delay getting back on this. I mentioned the annotation but it looks like I was wrong. This got changed to Labels at some point. I was thinking of the old behavior. See: https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/commit/ba50ef2698df8583cd46c4170d79c2f374afc74d
Reading through the claim reconciliation code it looks like it is relying on the label for cluster name: https://github.com/kubernetes-sigs/cluster-api-ipam-provider-in-cluster/blob/b8ccdf8c362cd51ee120e50f8ec2b637f02e3755/internal/controllers/ipaddressclaim.go#L180
We added a change in the cluster-api-ipam-provider to respect the paused status of the parent cluster, but it needs the cluster annotation on the claim object to find the owning cluster. https://github.com/kubernetes-sigs/cluster-api-ipam-provider-in-cluster/pull/105
That work was done for CAPV but not any other infrastructure providers, as far as I know. https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/pull/1857
What infrastructure provider are you using? Do you have an example of one of your claims? Does it have the cluster annotation?
It looks to me that we may have this issue in CAPV aswell - need to dive into it.We don’t have this issue in CAPV. Pausing on the IPAddressClaim isn’t explicitly taken care of, but the claims are only reconciled as part of the flow for VSphereVM. As long as those are paused the reconciliation for IPAddressClaims is paused. This is the same pattern for other dependent objects in reconcilers.
I’m building a provider for Proxmox, and I’m injecting static IPs with IPAM incluster.