application-gateway-kubernetes-ingress: TLS End to End Issue with Backend Health - Unhealthy, Resulting in 502 Errors

Describe the bug I am trying to configure TLS E2E. The frontend listener, backend pool, health probes all create successfully. When trying to access the URL I receive a 502 from the AppGW The HTTPS health probe if I edit it and click Test - is successful If I go to Backend Health, it shows - Cannot connect to server. Check whether any NSG/UDR/Firewall is blocking access to server. Connection Troubleshoot, manually specify IP of backend on port tcp/443 is succesful

To Reproduce I am using the exported root as the trusted cert on the gateway and it is selected Ok in the front end

I can forward a port via kubectl and access the pod just perfect over https by manipulating the domain to localhost

The cert is a wildcard

The 502 returned in the browser is receiving the correct cert and is valid

It seems the issue is with AGIC -> pod ?

Ingress Controller details

• Output of `kubectl describe pod:

`PS C:\repos\aks-lab-001> kubectl get pod NAME READY STATUS RESTARTS AGE ingress-azure-1590078545-7bf89c5df5-lhh84 2/2 Running 16 3h29m mic-5bf56d5658-6jqbs 1/1 Running 0 9h mic-5bf56d5658-kcgt7 1/1 Running 0 9h nmi-c26ph 1/1 Running 2 85d nmi-wbm7c 1/1 Running 5 85d PS C:\repos\aks-lab-001> kubectl describe pod ingress-azure-1590078545-7bf89c5df5-lhh84 Name: ingress-azure-1590078545-7bf89c5df5-lhh84 Namespace: default Priority: 0 Node: aks-linux-35064155-vmss000000/15.0.0.4 Start Time: Thu, 21 May 2020 17:29:23 +0100 Labels: aadpodidbinding=ingress-azure-1590078545 app=ingress-azure pod-template-hash=7bf89c5df5 release=ingress-azure-1590078545 security.istio.io/tlsMode=istio service.istio.io/canonical-name=ingress-azure service.istio.io/canonical-revision=latest Annotations: prometheus.io/port: 8123 prometheus.io/scrape: true sidecar.istio.io/status: {“version”:“fca84600f9d5ec316cf1cf577da902f38bac258ab0fd595ee208ec0203dc0c6d”,“initContainers”:[“istio-init”],“containers”:[“istio-proxy”]… Status: Running IP: 15.0.0.7 IPs: <none> Controlled By: ReplicaSet/ingress-azure-1590078545-7bf89c5df5 Init Containers: istio-init: Container ID: docker://caf5dc3a17278ede6d4ecf28af00b02a7ca632610c7b3ad37926de4fa8b4c4fa Image: docker.io/istio/proxyv2:1.5.4 Image ID: docker-pullable://istio/proxyv2@sha256:e16e2801b7fd93154e8fcb5f4e2fb1240d73349d425b8be90691d48e8b9bb944 Port: <none> Host Port: <none> Command: istio-iptables -p 15001 -z 15006 -u 1337 -m REDIRECT -i * -x

  -b
  *
  -d
  15090,15020
State:          Terminated
  Reason:       Completed
  Exit Code:    0
  Started:      Thu, 21 May 2020 17:29:25 +0100
  Finished:     Thu, 21 May 2020 17:29:25 +0100
Ready:          True
Restart Count:  0
Limits:
  cpu:     100m
  memory:  50Mi
Requests:
  cpu:        10m
  memory:     10Mi
Environment:  <none>
Mounts:       <none>

Containers: ingress-azure: Container ID: docker://85776a692fef00987f2f2ec4b3c5977cd6ee976e6700c2e3ef1aabc2329141c8 Image: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.0.0 Image ID: docker-pullable://mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:c295f99ae66443c5a392fd894620fcd1fc313b9efdec96d13f166fefb29780a9 Port: <none> Host Port: <none> State: Running Started: Thu, 21 May 2020 18:47:08 +0100 Last State: Terminated Reason: Error Exit Code: 2 Started: Thu, 21 May 2020 18:41:37 +0100 Finished: Thu, 21 May 2020 18:41:58 +0100 Ready: True Restart Count: 16 Liveness: http-get http://:15020/app-health/ingress-azure/livez delay=15s timeout=1s period=20s #success=1 #failure=3 Readiness: http-get http://:15020/app-health/ingress-azure/readyz delay=5s timeout=1s period=10s #success=1 #failure=3 Environment Variables from: ingress-azure-1590078545 ConfigMap Optional: false Environment: AZURE_CONTEXT_LOCATION: /etc/appgw/azure.json AGIC_POD_NAME: ingress-azure-1590078545-7bf89c5df5-lhh84 (v1:metadata.name) AGIC_POD_NAMESPACE: default (v1:metadata.namespace) Mounts: /etc/appgw/azure.json from azure (rw) /var/run/secrets/kubernetes.io/serviceaccount from ingress-azure-1590078545-token-v44t6 (ro) istio-proxy: Container ID: docker://66e8eb4b7e86fc7418c75d8deb910adce4671cd9b68eebce6ea615c2b2598f97 Image: docker.io/istio/proxyv2:1.5.4 Image ID: docker-pullable://istio/proxyv2@sha256:e16e2801b7fd93154e8fcb5f4e2fb1240d73349d425b8be90691d48e8b9bb944 Port: 15090/TCP Host Port: 0/TCP Args: proxy sidecar –domain $(POD_NAMESPACE).svc.cluster.local –configPath /etc/istio/proxy –binaryPath /usr/local/bin/envoy –serviceCluster ingress-azure.$(POD_NAMESPACE) –drainDuration 45s –parentShutdownDuration 1m0s –discoveryAddress istiod.istio-system.svc:15012 –zipkinAddress zipkin.istio-system:9411 –proxyLogLevel=warning –proxyComponentLogLevel=misc:error –connectTimeout 10s –proxyAdminPort 15000 –concurrency 2 –controlPlaneAuthPolicy NONE –dnsRefreshRate 300s –statusPort 15020 –trust-domain=cluster.local –controlPlaneBootstrap=false State: Running Started: Thu, 21 May 2020 17:29:27 +0100 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 1Gi Requests: cpu: 100m memory: 128Mi Readiness: http-get http://:15020/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30 Environment: JWT_POLICY: third-party-jwt PILOT_CERT_PROVIDER: istiod CA_ADDR: istio-pilot.istio-system.svc:15012 POD_NAME: ingress-azure-1590078545-7bf89c5df5-lhh84 (v1:metadata.name) POD_NAMESPACE: default (v1:metadata.namespace) INSTANCE_IP: (v1:status.podIP) SERVICE_ACCOUNT: (v1:spec.serviceAccountName) HOST_IP: (v1:status.hostIP) ISTIO_META_POD_PORTS: [ ] ISTIO_META_APP_CONTAINERS: [ ingress-azure ] ISTIO_META_CLUSTER_ID: Kubernetes ISTIO_META_POD_NAME: ingress-azure-1590078545-7bf89c5df5-lhh84 (v1:metadata.name) ISTIO_META_CONFIG_NAMESPACE: default (v1:metadata.namespace) ISTIO_META_INTERCEPTION_MODE: REDIRECT ISTIO_METAJSON_ANNOTATIONS: {“prometheus.io/port”:“8123”,“prometheus.io/scrape”:“true”}

  ISTIO_META_WORKLOAD_NAME:      ingress-azure-1590078545
  ISTIO_META_OWNER:              kubernetes://apis/apps/v1/namespaces/default/deployments/ingress-azure-1590078545
  ISTIO_META_MESH_ID:            cluster.local
  ISTIO_KUBE_APP_PROBERS:        {"/app-health/ingress-azure/livez":{"httpGet":{"path":"/health/alive","port":8123,"scheme":"HTTP"},"timeoutSeconds":1},"/app-health/ingress-azure/readyz":{"httpGet":{"path":"/health/ready","port":8123,"scheme":"HTTP"},"timeoutSeconds":1}}
Mounts:
  /etc/istio/pod from podinfo (rw)
  /etc/istio/proxy from istio-envoy (rw)
  /var/run/secrets/istio from istiod-ca-cert (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from ingress-azure-1590078545-token-v44t6 (ro)
  /var/run/secrets/tokens from istio-token (rw)

Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: azure: Type: HostPath (bare host directory volume) Path: /etc/kubernetes/azure.json HostPathType: File ingress-azure-1590078545-token-v44t6: Type: Secret (a volume populated by a Secret) SecretName: ingress-azure-1590078545-token-v44t6 Optional: false istio-envoy: Type: EmptyDir (a temporary directory that shares a pod’s lifetime) Medium: Memory SizeLimit: <unset> podinfo: Type: DownwardAPI (a volume populated by information about the pod) Items: metadata.labels -> labels metadata.annotations -> annotations istio-token: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 43200 istiod-ca-cert: Type: ConfigMap (a volume populated by a ConfigMap) Name: istio-ca-root-cert Optional: false QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: <none>`

• Output of kubectl logs <ingress controller>: <snip> – App Gwy config --} I0521 19:40:50.369045 1 mutate_app_gateway.go:182] Applied App Gateway config in 20.446695951s I0521 19:40:50.369062 1 mutate_app_gateway.go:198] cache: Updated with latest applied config. I0521 19:40:50.370154 1 mutate_app_gateway.go:203] END AppGateway deployment`

• Any Azure support tickets associated with this issue. N/A