ingress-nginx: Terminating pod causes network timeouts with GCP L4 LB and externalTrafficPolicy: Local

NGINX Ingress controller version: 0.34.1

Kubernetes version (use kubectl version): v1.16.13-gke.401

Environment:

• Cloud provider or hardware configuration: GKE
• Load Balancer: GCP TCP Load Balancer
• Extra Options: service.type: LoadBalancer service.externalTrafficPolicy: Local lifecycle.preStop.exec.command: ["sh", "-c", "sleep 60 && /wait-shutdown"] kind: Deployment

What happened:

Incoming connections to the HTTP/HTTPS ports from the Load Balancer start timing out immediately upon the start of the termination process if the pod is the last copy on a Node. This results in downscaling using the HPA to periodically cause almost 30 seconds of service disruption, as the Load Balancer continues to send traffic during that time until it is removed as a backend to the LB due to failing health checks.

What you expected to happen:

Regardless of whether externalTrafficPolicy is Local or Cluster, the preStop hook should be honored so that the Load Balancer has time to remove the empty Node.

This appears to be a result of the same issue causing https://github.com/kubernetes/kubernetes/issues/85643. Inside the Node, the HTTP(S) NodePort continues to work correctly during the termination process until the app actually stops, at which point the NGINX endpoint has been safely removed as an Endpoint. However, the moment the termination process begins, that port is closed to external traffic, meaning that there is no grace period for the Load Balancer to remove the Node from its backend pool and any traffic sent to the Node is silently lost.

How to reproduce it:

1. Deploy ingress-nginx in GKE using externalTrafficPolicy: Local and a delaying preStop hook.
2. Run no more than one NGINX pod per Node.
3. Send traffic directly to the HTTP NodePort on each Node and observe that it reaches the Default Backend.
4. Remove one pod from the Deployment.
5. Observe that, immediately, the HTTP NodePort is closed to outside traffic despite the pod itself continuing to run NGINX.

Anything else we need to know: I’m trying to find any solution around this problem (however hacky) that preserves the remote IP and, ideally, allows me to continue using ingress-nginx.

Already attempted:

• Using service.type: Both to ensure at least one NGINX pod on every Node through the DaemonSet – this helps a bit, but, in addition to overprovisioning the service, has its own problems around the Node itself being terminated.
• Disabling HPA with a higher baseline – this eliminates the risk of scaling, and thus prevents the issue from occurring, but leaves us at risk of both overprovisioning and not being able to handle an unexpected spike properly.
• Running with externalTrafficPolicy: Cluster – I could find no configuration in GKE where this preserved the remote IP, which is a hard requirement.
• Switching to a different Ingress Controller – so far, it looks like the only option which would avoid this problem would be the GCE Ingress and an L7 GCP LB. However, I currently rely on NGINX features that the GCE Ingress does not support.

/kind bug