ingress-nginx: Readiness and Liveness probe failed: HTTP probe failed with statuscode: 500

NGINX Ingress controller version: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.11.0 installed with helm using stable chart. Kubernetes version (use kubectl version): 1.8.4 Environment:

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): kops version 1.8.1 (Debian I think)

What happened: nginx-ingress-controller pod Readiness and Liveness probe failed: HTTP probe failed with statuscode: 500. The pod is terminated and restarted. This happens 2-5 times until it starts successfully.

What you expected to happen: Pod to start successfully without failing Readiness and Liveness probe.

How to reproduce it (as minimally and precisely as possible): We are running the nginx-ingress-controller as a daemonset so whenever a new node is created we see this problem.

Anything else we need to know: This issue has been opened before:

• https://github.com/kubernetes/ingress-nginx/issues/2058
• https://github.com/kubernetes/ingress-nginx/issues/1989

Here are the events from the nginx-ingress-controller pod:

Events:
  Type     Reason                 Age                From                                                  Message
  ----     ------                 ----               ----                                                  -------
  Normal   SuccessfulMountVolume  2m                 kubelet, ip-10-0-19-85.eu-central-1.compute.internal  MountVolume.SetUp succeeded for volume "ingress1-nginx-ingress-token-jm48x"
  Warning  FailedSync             1m (x3 over 2m)    kubelet, ip-10-0-19-85.eu-central-1.compute.internal  Error syncing pod
  Normal   Pulling                1m                 kubelet, ip-10-0-19-85.eu-central-1.compute.internal  pulling image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.11.0"
  Normal   Pulled                 48s                kubelet, ip-10-0-19-85.eu-central-1.compute.internal  Successfully pulled image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.11.0"
  Warning  Unhealthy              13s (x3 over 33s)  kubelet, ip-10-0-19-85.eu-central-1.compute.internal  Liveness probe failed: HTTP probe failed with statuscode: 500
  Warning  Unhealthy              4s (x4 over 34s)   kubelet, ip-10-0-19-85.eu-central-1.compute.internal  Readiness probe failed: HTTP probe failed with statuscode: 500
  Normal   Created                0s (x2 over 48s)   kubelet, ip-10-0-19-85.eu-central-1.compute.internal  Created container
  Normal   Started                0s (x2 over 48s)   kubelet, ip-10-0-19-85.eu-central-1.compute.internal  Started container
  Normal   Killing                0s                 kubelet, ip-10-0-19-85.eu-central-1.compute.internal  Killing container with id docker://nginx-ingress-controller:Container failed liveness probe.. Container will be killed and recreated.
  Normal   Pulled                 0s                 kubelet, ip-10-0-19-85.eu-central-1.compute.internal  Container image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.11.0" already present on machine

Here is the default probe config:

       livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1

Here is the helm chart values we use: https://gist.github.com/max-rocket-internet/ba6b368502f58bc7061d3062939b5dca

I have logs from pod with --v=10 argument set but there is a lot of output and some of it is sensitive. Here is an excerpt but let me know if need more:

I0305 10:57:12.548693       7 main.go:47] annotation kubernetes.io/ingress.class is not present in ingress default/env1-app1-part1
I0305 10:57:15.587793       7 round_trippers.go:417] curl -k -v -XGET  -H "Accept: application/vnd.kubernetes.protobuf, */*" -H "User-Agent: nginx-ingress-controller/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer xxxxxxxx" https://100.64.0.1:443/api/v1/namespaces/default/configmaps/ingress-controller-leader-nginx
I0305 10:57:15.590327       7 round_trippers.go:436] GET https://100.64.0.1:443/api/v1/namespaces/default/configmaps/ingress-controller-leader-nginx 200 OK in 2 milliseconds
I0305 10:57:15.590344       7 round_trippers.go:442] Response Headers:
I0305 10:57:15.590350       7 round_trippers.go:445]     Content-Type: application/vnd.kubernetes.protobuf
I0305 10:57:15.590355       7 round_trippers.go:445]     Content-Length: 437
I0305 10:57:15.590362       7 round_trippers.go:445]     Date: Mon, 05 Mar 2018 10:57:15 GMT
I0305 10:57:15.590397       7 request.go:871] Response Body:
00000000  6b 38 73 00 0a 0f 0a 02  76 31 12 09 43 6f 6e 66  |k8s.....v1..Conf|
...
I0305 10:57:15.590459       7 leaderelection.go:243] lock is held by ingress1-nginx-ingress-controller-9jsqp and has not yet expired
I0305 10:57:15.590467       7 leaderelection.go:180] failed to acquire lease default/ingress-controller-leader-nginx
I0305 10:57:22.549142       7 main.go:47] annotation kubernetes.io/ingress.class is not present in ingress default/env1-app2-admin
I0305 10:57:26.091336       7 main.go:152] Received SIGTERM, shutting down
I0305 10:57:26.091359       7 nginx.go:359] shutting down controller queues
I0305 10:57:26.091376       7 nginx.go:367] stopping NGINX process...
2018/03/05 10:57:26 [notice] 29#29: signal process started
I0305 10:57:29.097347       7 nginx.go:380] NGINX process has stopped
I0305 10:57:29.097372       7 main.go:160] Handled quit, awaiting pod deletion
I0305 10:57:30.992643       7 round_trippers.go:417] curl -k -v -XGET  -H "User-Agent: nginx-ingress-controller/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer xxxxxx" -H "Accept: application/vnd.kubernetes.protobuf, */*" https://100.64.0.1:443/api/v1/namespaces/default/configmaps/ingress-controller-leader-nginx
I0305 10:57:30.994766       7 round_trippers.go:436] GET https://100.64.0.1:443/api/v1/namespaces/default/configmaps/ingress-controller-leader-nginx 200 OK in 2 milliseconds
I0305 10:57:30.994786       7 round_trippers.go:442] Response Headers:
I0305 10:57:30.994792       7 round_trippers.go:445]     Content-Length: 437
I0305 10:57:30.994818       7 round_trippers.go:445]     Date: Mon, 05 Mar 2018 10:57:30 GMT
I0305 10:57:30.994832       7 round_trippers.go:445]     Content-Type: application/vnd.kubernetes.protobuf
I0305 10:57:30.994891       7 request.go:871] Response Body:
00000000  6b 38 73 00 0a 0f 0a 02  76 31 12 09 43 6f 6e 66  |k8s.....v1..Conf|
....
000001b0  00 1a 00 22 00                                    |...".|
I0305 10:57:30.995001       7 leaderelection.go:243] lock is held by ingress1-nginx-ingress-controller-9jsqp and has not yet expired
I0305 10:57:30.995029       7 leaderelection.go:180] failed to acquire lease default/ingress-controller-leader-nginx
I0305 10:57:39.097529       7 main.go:163] Exiting with 0