ingress-nginx: Nginx ingress causing pod's nginx timeout while connecting to fastcgi upstream

I just had to choose nginx over traefik as my k8s ingress controller because I need source ip preservation using ProxyProtocol (cluster is behind an AWS ELB doing TCP load balancing).

I’m experiencing a very strange issue I can’t easily reproduce and this makes debugging really hard. Traffic flows this way:

AWS ELB > INGRESS > POD (2 containers: nginx and php-fpm)

When using traefik as ingress controller, everything is OK.

As soon as I replace traefik with nginx (using nginx-ingress-controller v0.9.0-beta.7), first requests are served correctly, then php-fpm child processes start getting stuck after a few minutes and several connections get timed out (both my pod’s nginx and the ingress one show a 499 as HTTP response code for those requests). As a result, the nginx used as ingress controller starts banning “unresponsive” pods until all pods are banned and incoming requests can’t be served.

This happens only under a production workload, while I have no issues in the staging environment. Apparently, HTTP headers are the same - I can’t see any relevant difference between headers passed by traefik and the ones forwarded by nginx.

I don’t want to raise pm.max_children value on php-fpm containers because, as I said, current setup works really smooth if I stick with traefik.

Any idea on what may be causing such a weird behaviour? Thanks.

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 2
Comments: 20 (7 by maintainers)

Most upvoted comments

Hi @zuc, this issue really bothers my team and me, but eventually we have found a way to bypass it, after experimenting many solutions.

Firstly, let me rule out a few potential causes:

Various timeouts, including proxy-read-timeout, proxy-send-timeout, proxy-connect-timeout (neither setting them in the annotations of the controller or individual ingress helps) and idle timeout on the classic load balancer of the ingress controller.
Type of Kubernetes, regardless if it is an EKS or one provisioned by kops
Load balancer type. We changed it to a Nginx controller on NLB, but the same error keeps happening.
Nginx itself. We tried a k8s service of type LoadBalancer to bypass Nginx, but it did not work.
Any client issue, which is quite obvious, because the traffic has not reached the client side yet.

Theoretical root cause

The flow in reality is

AWS ELB / CLB > NodePort > Nginx ingress controller pod

Through several conversation with AWS support, I believe that it is because unexpected intermittent TCP resets happen on the NodePort. A NodePort is essentially a NAT component and TCP resets can happen on such a component. When such resets take place, Nginx thinks that its client, the NodePort, has aborted its connection thus throws a 499 error.

When we tried a Nginx ingress controller on NLB, encountered the same problem and contacted AWS support, they told me that they could find TCP resets in NLB’s logs. I believe that if packet sniffing is done at each k8s node, TCP resets can be found in the packet dumps. But I did not do it.

Solution

This solution has been applied to 2 applications and proven effective. You need:

EKS (the prerequisite of using ALB ingress controller)
ALB ingress controller (follow here to set it up)
ALB ingress with target-type as ip. Check out this doc for more details.

When the TargetType of ALB is IP, traffic is routed directly from LB instances to pod IP by AWS networking infra, without going through any NodePort, so the erroneous component is bypassed and no more errors.

Another big advantage of using ALB is that it attaches many HTTP headers, such as source IP header, just like how Nginx does, while traefik ingress controller does not.

wanwenli on Jul 26, 2020