application-gateway-kubernetes-ingress: Cannot use private IP address as it is either allocated to a resource or being cleaned up.

Describe the bug Whenever we define a second ingress, or remove all of the ingresses and try to create a new ingress, we get the following error:

Log:

application-gateway-ingress-controller-ingress-azure-9bf4b95b8r ingress-azure E0430 17:37:43.416766 1 controller.go:132] Error mutating App Gateway config from k8s event. network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 – Original Error: Code=“ApplicationGatewayFrontendIPStaticPrivateIPAddressNotAvailable” Message=“FrontendIpConfiguration /subscriptions/SUBSCRIPTION_ID/resourceGroups/k8s/providers/Microsoft.Network/applicationGateways/k8s-appgateway/frontendIPConfigurations/AzureVnet-feip2 cannot use private IP address 10.161.134.5 as it is either allocated to a resource or being cleaned up.” Details=[]

To Reproduce Steps to reproduce the behavior:

1. Install controller
2. Create a AGW
3. Create the following ingress (this will result in a nice ingress):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/appgw-ssl-certificate: "mycert"
    appgw.ingress.kubernetes.io/use-private-ip: "true"
spec:
  rules:
  - host: guestbook.test-k8s.mydomain.com
    http:
      paths:
      - backend:
          serviceName: frontend
          servicePort: 80

1. Repeat step #3. The ingress will look normal, but the error (ApplicationGatewayFrontendIPStaticPrivateIPAddressNotAvailable) will appear.

Ingress Controller details Version: 1.2.0-rc1 pod describe:

kdp application-gateway-ingress-controller-ingress-azure-9bf4b95b8r
Name:           application-gateway-ingress-controller-ingress-azure-9bf4b95b8r
Namespace:      cpt
Priority:       0
Node:           aks-cpt-34239724-vmss000000/10.161.132.4
Start Time:     Thu, 30 Apr 2020 17:17:01 +0200
Labels:         aadpodidbinding=application-gateway-ingress-controller-ingress-azure
                app=ingress-azure
                pod-template-hash=9bf4b78c7
                release=application-gateway-ingress-controller
Annotations:    prometheus.io/port: 8123
                prometheus.io/scrape: true
Status:         Running
IP:             10.161.132.7
IPs:            <none>
Controlled By:  ReplicaSet/application-gateway-ingress-controller-ingress-azure-9bf4b78c7
Containers:
  ingress-azure:
    Container ID:   docker://a6697e2d36a2a92f2c26a8c29444606fa20e69c77d49f1540910854ef89a39b6
    Image:          azure-application-gateway/kubernetes-ingress:1.2.0-rc1
    Image ID:       docker-pullable://azure-application-gateway/kubernetes-ingress@sha256:dd95b2feaf24e7ba6773452fb842d0eba5a6ea8a5d19bf22035fdcad78b18941
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Thu, 30 Apr 2020 17:17:03 +0200
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:      http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      application-gateway-ingress-controller-cm-ingress-azure  ConfigMap  Optional: false
    Environment:
      AZURE_CONTEXT_LOCATION:  /etc/appgw/azure.json
      AGIC_POD_NAME:           application-gateway-ingress-controller-ingress-azure-9bf4b95b8r (v1:metadata.name)
      AGIC_POD_NAMESPACE:      cpt (v1:metadata.namespace)
    Mounts:
      /etc/appgw/azure.json from azure (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from application-gateway-ingress-controller-sa-ingress-azure-tobqzbq (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  azure:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/azure.json
    HostPathType:  File
  application-gateway-ingress-controller-sa-ingress-azure-tobqzbq:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  application-gateway-ingress-controller-sa-ingress-azure-tobqzbq
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>