cloud-provider-azure: IPv6 service throw ServiceIPAddressCannotBeChangedOnUpdate error when ensure load balancer.

Describe the bug If the static IP for the loadbalancer service has the property servicePublicIPAddress, it will fail to ensure load balancer when creating the service with an existing static IP. The issue does not exist in kubernetes 1.18. But it exists in kubernetes 1.20.0-rc.0

  Normal   EnsuringLoadBalancer           4m54s (x80 over 6h15m)  service-controller    Ensuring load balancer
  Warning  CreateOrUpdatePublicIPAddress  4m53s (x80 over 6h15m)  azure-cloud-provider  Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: {
  "error": {
    "code": "ServiceIPAddressCannotBeChangedOnUpdate",
    "message": "Service IP Address property is set at creation time and cannot be changed.",
    "details": []
  }
}

Steps To Reproduce

  1. using aks-engine to deploy cluster with dual stack enabled.
{
    "apiVersion": "vlabs",
    "location": "southcentralus",
    "properties": {
        "featureFlags": {
            "enableIPv6DualStack": true
        },
        "orchestratorProfile": {
            "orchestratorType": "Kubernetes",
            "orchestratorRelease": "1.20",
            "kubernetesConfig": {
                "networkPlugin": "kubenet",
                "kubeProxyMode":"ipvs",
                "useManagedIdentity": true,
                "cloudProviderDisableOutboundSNAT": true,
                "apiServerConfig": {
                    "--feature-gates": "IPv6DualStack=true"
                },
                "kubeletConfig": {
                    "--feature-gates": "IPv6DualStack=true"
                },
                "controllerManagerConfig": {
                    "--feature-gates": "IPv6DualStack=true"
                }
            }
        },
        "masterProfile": {
            "count": 1,
            "dnsPrefix": "***",
            "vmSize": "Standard_D2_v3"
        },
        "agentPoolProfiles": [
            {
                "name": "linuxpool1",
                "count": 2,
                "vmSize": "Standard_D2_v3",
                "availabilityProfile": "VirtualMachineScaleSets"
            }
        ],
        "linuxProfile": {
            "adminUsername": "***",
            "ssh": {
                "publicKeys": [
                    {
                        "keyData": "****"
                    }
                ]
            }
        }
    }
}
  1. Create an IPv6 address with servicePublicIPAddress property. To create IPv6 address with this property, you also have to create an IPv4 address.
{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {},
    "resources": [
        {
            "name": "test-ipv4",
            "type": "Microsoft.Network/publicIPAddresses",
            "apiVersion": "2020-05-01",
            "location": "[resourceGroup().location]",
            "sku": {
                "name": "Standard"
            },
            "properties": {
                "publicIPAllocationMethod": "Static",
                "publicIPAddressVersion": "IPv4",
                "IpTags": ["***IpTags****" ]
            }
        },
        {
            "name": "test-ipv6",
            "type": "Microsoft.Network/publicIPAddresses",
            "apiVersion": "2020-05-01",
            "location": "[resourceGroup().location]",
            "sku": {
                "name": "Standard"
            },
            "properties": {
                "publicIPAllocationMethod": "Static",
                "publicIPAddressVersion": "IPv6",
                "servicePublicIPAddress": {
                    "id": "[resourceid('Microsoft.Network/publicIPAddresses/', 'test-ipv4')]"
                }
            },
            "dependsOn": [
                "[resourceid('Microsoft.Network/publicIPAddresses/', 'test-ipv4')]"
            ]
        }
    ],
    "outputs": {
        "IPs": {
            "type": "array",
            "value": [
                "[reference(resourceid('Microsoft.Network/publicIPAddresses/', 'test-ipv4')).ipAddress]",
                "[reference(resourceid('Microsoft.Network/publicIPAddresses/', 'test-ipv6')).ipAddress]"
            ]
        }
    }
}
  1. Create an IPv6 loadbalancer service to reference the IPv6 address created in step 2.
apiVersion: v1
kind: Service
metadata:
  name: nginx-se-ipv6
spec:
  ipFamilies:
    - IPv6
  externalTrafficPolicy: Local
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer
  loadBalancerIP: <***the-ipv6-address-created-in-step-2***>

It will pending on ensuring load balancer, and never succeed. When describe the loadbalancer service, it shows following error message. From the error message, looks like it’s trying to update the IPv6 address without the servicePublicIPAddress property while ensuring load balancer.

  "error": {
    "code": "ServiceIPAddressCannotBeChangedOnUpdate",
    "message": "Service IP Address property is set at creation time and cannot be changed.",
    "details": []
  }

Expected behavior It should successfully ensure the load balancer with a specified static IPv6 address with property servicePublicIPAddress.

AKS Engine version v0.58.0

Kubernetes version v1.20.0-rc.0

Additional context

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 21 (12 by maintainers)

Most upvoted comments

Workaround: build the ccm image from master. If you provision the cluster by aks-engine, please turn off the out-of-tree node ipam controller by setting --allocate-node-cidrs=false in /etc/kubernetes/manifests/cloud-provider-azure.yaml after logging into the master.