cloud-provider-azure: IPv6 service throw ServiceIPAddressCannotBeChangedOnUpdate error when ensure load balancer.
Describe the bug If the static IP for the loadbalancer service has the property servicePublicIPAddress, it will fail to ensure load balancer when creating the service with an existing static IP. The issue does not exist in kubernetes 1.18. But it exists in kubernetes 1.20.0-rc.0
Normal EnsuringLoadBalancer 4m54s (x80 over 6h15m) service-controller Ensuring load balancer
Warning CreateOrUpdatePublicIPAddress 4m53s (x80 over 6h15m) azure-cloud-provider Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: {
"error": {
"code": "ServiceIPAddressCannotBeChangedOnUpdate",
"message": "Service IP Address property is set at creation time and cannot be changed.",
"details": []
}
}
Steps To Reproduce
- using aks-engine to deploy cluster with dual stack enabled.
{
"apiVersion": "vlabs",
"location": "southcentralus",
"properties": {
"featureFlags": {
"enableIPv6DualStack": true
},
"orchestratorProfile": {
"orchestratorType": "Kubernetes",
"orchestratorRelease": "1.20",
"kubernetesConfig": {
"networkPlugin": "kubenet",
"kubeProxyMode":"ipvs",
"useManagedIdentity": true,
"cloudProviderDisableOutboundSNAT": true,
"apiServerConfig": {
"--feature-gates": "IPv6DualStack=true"
},
"kubeletConfig": {
"--feature-gates": "IPv6DualStack=true"
},
"controllerManagerConfig": {
"--feature-gates": "IPv6DualStack=true"
}
}
},
"masterProfile": {
"count": 1,
"dnsPrefix": "***",
"vmSize": "Standard_D2_v3"
},
"agentPoolProfiles": [
{
"name": "linuxpool1",
"count": 2,
"vmSize": "Standard_D2_v3",
"availabilityProfile": "VirtualMachineScaleSets"
}
],
"linuxProfile": {
"adminUsername": "***",
"ssh": {
"publicKeys": [
{
"keyData": "****"
}
]
}
}
}
}
- Create an IPv6 address with servicePublicIPAddress property. To create IPv6 address with this property, you also have to create an IPv4 address.
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {},
"resources": [
{
"name": "test-ipv4",
"type": "Microsoft.Network/publicIPAddresses",
"apiVersion": "2020-05-01",
"location": "[resourceGroup().location]",
"sku": {
"name": "Standard"
},
"properties": {
"publicIPAllocationMethod": "Static",
"publicIPAddressVersion": "IPv4",
"IpTags": ["***IpTags****" ]
}
},
{
"name": "test-ipv6",
"type": "Microsoft.Network/publicIPAddresses",
"apiVersion": "2020-05-01",
"location": "[resourceGroup().location]",
"sku": {
"name": "Standard"
},
"properties": {
"publicIPAllocationMethod": "Static",
"publicIPAddressVersion": "IPv6",
"servicePublicIPAddress": {
"id": "[resourceid('Microsoft.Network/publicIPAddresses/', 'test-ipv4')]"
}
},
"dependsOn": [
"[resourceid('Microsoft.Network/publicIPAddresses/', 'test-ipv4')]"
]
}
],
"outputs": {
"IPs": {
"type": "array",
"value": [
"[reference(resourceid('Microsoft.Network/publicIPAddresses/', 'test-ipv4')).ipAddress]",
"[reference(resourceid('Microsoft.Network/publicIPAddresses/', 'test-ipv6')).ipAddress]"
]
}
}
}
- Create an IPv6 loadbalancer service to reference the IPv6 address created in step 2.
apiVersion: v1
kind: Service
metadata:
name: nginx-se-ipv6
spec:
ipFamilies:
- IPv6
externalTrafficPolicy: Local
selector:
app: nginx
ports:
- protocol: TCP
port: 80
targetPort: 80
type: LoadBalancer
loadBalancerIP: <***the-ipv6-address-created-in-step-2***>
It will pending on ensuring load balancer, and never succeed. When describe the loadbalancer service, it shows following error message. From the error message, looks like it’s trying to update the IPv6 address without the servicePublicIPAddress property while ensuring load balancer.
"error": {
"code": "ServiceIPAddressCannotBeChangedOnUpdate",
"message": "Service IP Address property is set at creation time and cannot be changed.",
"details": []
}
Expected behavior It should successfully ensure the load balancer with a specified static IPv6 address with property servicePublicIPAddress.
AKS Engine version v0.58.0
Kubernetes version v1.20.0-rc.0
Additional context
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 21 (12 by maintainers)
Workaround: build the ccm image from master. If you provision the cluster by aks-engine, please turn off the out-of-tree node ipam controller by setting
--allocate-node-cidrs=falsein/etc/kubernetes/manifests/cloud-provider-azure.yamlafter logging into the master.