kubernetes-ingress: A rollout restart, or scale down of haproxy causes 503 connection timeout errors

I’m using helm to run HaProxy ingress with autoscaling enabled (chart version 1.21.1). Whenever an HaProxy pod terminates (because of a scale down event, or a rollout restart), I start seeing 503 backend connection timeout errors for a few seconds.

hapoxy-termination-errors

I tried adding the following example config for graceful shutdown, but that did not resolve the issue:

## Example preStop for graceful shutdown
lifecycle: {}
    preStop:
      exec:
        command: ["/bin/sh", "-c", "kill -USR1 $(pidof haproxy); while killall -0 haproxy; do sleep 1; done"]

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (4 by maintainers)

Most upvoted comments

I can confirm. We have identified the culprit and the fix is in the queue, being reviewed.

@evandam Fix has been commited in https://github.com/haproxytech/kubernetes-ingress/commit/6afd804b0410154daf601fcf3ca5969623aeef89 and the release is incoming, I’ll check the exact time frame we expect it to be published.

I have same issue with haproxy ingress controller.