application-gateway-kubernetes-ingress: FailedApplyingAppGwConfig: Either all or no rule should have the priority specified.

Describe the bug AGIC stop updating AppGateway after a while and complains with error “Either all or no rule should have the priority specified.”

To Reproduce I created a bunch of ingress with several hosts (all of them pointing to the same frontend-ip in the AppGateway). When starting from an empty set of rules, they work. Every listener, rule, backend pool, etc… is created successfully. After a while, when some change is made in an ingress in Kubernetes, AGIC stop doing updates and complains with this error log:

I0504 10:45:40.120520 1 mutate_app_gateway.go:177] BEGIN AppGateway deployment I0504 10:45:40.426904 1 mutate_app_gateway.go:183] END AppGateway deployment E0504 10:45:40.427098 1 controller.go:141] network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="ApplicationGatewayRequestRoutingRulePartialPriorityDefi ned" Message="Either all or no rule should have the priority specified." Details=[] E0504 10:45:40.427154 1 worker.go:62] Error processing event.network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="ApplicationGatewayRequestRoutingRuleP artialPriorityDefined" Message="Either all or no rule should have the priority specified." Details=[] I0504 10:45:40.432780 1 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"hub", Name:"agic-ingress-azure-99d8fc565-s8g4w", UID:"61131005-4cdc-4be4-9fc3-60acb2487e23", APIVersion:"v1", ResourceVersion :"21111199", FieldPath:""}): type: 'Warning' reason: 'FailedApplyingAppGwConfig' network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="ApplicationGatewayReque stRoutingRulePartialPriorityDefined" Message="Either all or no rule should have the priority specified." Details=[]

Ingress Controller details

• Output of kubectl describe pod <ingress controller> . The <ingress controller> pod name can be obtained by running helm list.

Name: agic-ingress-azure-99d8fc565-s8g4w Namespace: hub Priority: 0 Node: aks-agentpool-42282736-vmss000004/18.242.2.8 Start Time: Wed, 04 May 2022 12:43:19 +0200 Labels: aadpodidbinding=agic-ingress-azure app=ingress-azure pod-template-hash=99d8fc565 release=agic Annotations: checksum/config: 3ae39f08526de407f7a48e9a94fab871961bbc45019141cd61533c27147a2894 kubectl.kubernetes.io/restartedAt: 2022-05-04T12:24:43+02:00 prometheus.io/port: 8123 prometheus.io/scrape: true Status: Running IP: 10.244.0.62 IPs: IP: 10.244.0.62 Controlled By: ReplicaSet/agic-ingress-azure-99d8fc565 Containers: ingress-azure: Container ID: containerd://d6654a6d6e3036c4fce4a63d98c82cc153a5bea7fd11c42b00e085be8bf97aad Image: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.1 Image ID: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:cc131292df265926942e23ca5601a3de66e8feabcb81f705d8f7d84b740f81b6 Port: <none> Host Port: <none> State: Running Started: Wed, 04 May 2022 12:43:29 +0200 Ready: True Restart Count: 0 Liveness: http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3 Readiness: http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3 Environment Variables from: agic-cm-ingress-azure ConfigMap Optional: false Environment: AZURE_CLOUD_PROVIDER_LOCATION: /etc/appgw/azure.json AGIC_POD_NAME: agic-ingress-azure-99d8fc565-s8g4w (v1:metadata.name) AGIC_POD_NAMESPACE: hub (v1:metadata.namespace) Mounts: /etc/appgw/ from azure (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-m2m5r (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: azure: Type: HostPath (bare host directory volume) Path: /etc/kubernetes/ HostPathType: Directory kube-api-access-m2m5r: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message

---

Normal Scheduled 9m42s default-scheduler Successfully assigned hub/agic-ingress-azure-99d8fc565-s8g4w to aks-agentpool-42282736-vmss000004 Normal Pulling 9m42s kubelet Pulling image “mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.1” Normal Pulled 9m35s kubelet Successfully pulled image “mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.1” in 6.134151482s Normal Created 9m33s kubelet Created container ingress-azure Normal Started 9m33s kubelet Started container ingress-azure Warning FailedApplyingAppGwConfig 2m14s (x8 over 9m29s) azure/application-gateway network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 – Original Error: Code=“ApplicationGatewayRequestRoutingRulePartialPriorityDefined” Message=“Either all or no rule should have the priority specified.” Details=[]

• Output of `kubectl logs <ingress controller>. I0504 10:50:48.172706 1 mutate_app_gateway.go:177] BEGIN AppGateway deployment I0504 10:50:48.450022 1 mutate_app_gateway.go:183] END AppGateway deployment E0504 10:50:48.450489 1 controller.go:141] network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 – Original Error: Code=“ApplicationGatewayRequestRoutingRulePartialPriorityDefined” Message=“Either all or no rule should have the priority specified.” Details=[] E0504 10:50:48.450543 1 worker.go:62] Error processing event.network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 – Original Error: Code=“ApplicationGatewayRequestRoutingRulePartialPriorityDefined” Message=“Either all or no rule should have the priority specified.” Details=[] I0504 10:50:48.451785 1 event.go:282] Event(v1.ObjectReference{Kind:“Pod”, Namespace:“hub”, Name:“agic-ingress-azure-99d8fc565-s8g4w”, UID:“61131005-4cdc-4be4-9fc3-60acb2487e23”, APIVersion:“v1”, ResourceVersion:“21111199”, FieldPath:“”}): type: ‘Warning’ reason: ‘FailedApplyingAppGwConfig’ network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 – Original Error: Code=“ApplicationGatewayRequestRoutingRulePartialPriorityDefined” Message=“Either all or no rule should have the priority specified.” Details=[] I0504 10:50:50.496012 1 reflector.go:530] pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167: Watch close - *v1.IngressClass total 0 items received I0504 10:51:00.488228 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167: forcing resync I0504 10:51:00.560594 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167: forcing resync I0504 10:51:00.577887 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167: forcing resync I0504 10:51:00.595082 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.21.2/tools/cache/reflector.go:167: forcing resync