ingress-nginx: Random / Sporadic 502 gateway timeouts
Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report
NGINX Ingress controller version: 0.25.0
Kubernetes version (use kubectl version):
v1.12.10
Environment: aws / eks
- Cloud provider or hardware configuration:
- OS (e.g. from /etc/os-release): Amazon Linux
- Kernel (e.g.
uname -a): 4.14.106-97.85.amzn2.x86_64
What happened: We’re seeing random and sporadic 502’s being returned and unable to reliably reproduce.
What you expected to happen: ingress should respond with a 200
How to reproduce it (as minimally and precisely as possible): Unsure as it happens very sporadically
Anything else we need to know:
messages from ingress controllers: “*2169 upstream prematurely closed connection while reading response header from upstream” “*1360038 connect() failed (113: No route to host) while connecting to upstream” “*1655177 upstream timed out (110: Connection timed out) while connecting to upstream”
This was working a week ago; now we’re receiving these 502’s from multiple deployments (some of which have not changed in over a month). We’ve checked the load on the upstream pods and they are handing traffic well and we can port-forward to them directly and not have any 502’s or connection issues.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 21 (6 by maintainers)
Wanted to leave this here in case it helps anyone else. I was getting intermittent 502’s (browser confusingly reported a CORS error), and eventually realized that my Kubernetes manifests were using a shared metadata selector label. Be sure these are unique for each service!
@miclefebvre if you can’t do that, you could use the annotation
nginx.ingress.kubernetes.io/proxy-next-upstream: error timeout http_502to activate retries in that case