kubernetes: Readiness probe is failing, but the service is accessible when execd

What happened: Readiness probe failed

What you expected to happen: Succesfull container startup

How to reproduce it (as minimally and precisely as possible): I created a NodePort service, and then the issue seemed to start. I deleted the NodePort service and reverted back to the previous LoadBalancer service but the issue remains.

If I remove the liveness probe from my deployment config, the containers start up successfully, and I’m able to access them via the LoadBalancer service external IP

Cloud provider or hardware configuration: Google Cloud Platform

The container in question is a Nginx Container listening on port 80.

The event from kubectl describe is: Unhealthy Readiness probe failed: Get http://10.24.1.33:80/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

I can exec into the container and do: wget 10.24.1.33 which works correctly (HTTP 200 response).

Readiness probe config:

          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 5 
            periodSeconds: 5

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 8
Comments: 35 (3 by maintainers)

Most upvoted comments

@saurabhdevops You can increase the probe timeout if you don’t expect the container to respond within 1 second (livenessProbe.timeoutSeconds).

+31

mgalgs on Jan 17, 2018

Did anyone find a solution to this?

+26

cattermo on Sep 26, 2019

sorry to bump this, but I’m on EKS (if it matters) and my nginx ingress readiness probes are taking more than 1 second to respond too. Sometimes they hang upwards of 5-8 seconds and then respond. I increased livenessProbe.timeoutSeconds but I’m trying to figure out why this delay occurs. Does anyone here have any suggestions?

+17

Macmee on Sep 29, 2018

The issue was - the container was taking longer then a second to respond.

+15

chrissound on Aug 29, 2017

Here facing the same issue Readiness probe failed: Get http://127.0.0.1:8085/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) even though service return 200 response

vatsramkesh on Feb 11, 2020

Any Updates ?

Biga-incorta on Dec 3, 2019

@chrissound Would you mind suggesting what you did to resolve this? I am facing a similar issue.

saurabhdevops on Oct 30, 2017

@Macmee Having the same strange issue here. Basically we started using NodePort, and it sporadically times out the readiness / liveness probe checks.

When I inspect our monitoring metrics, the endpoints used for this has a maximum response time of 4.5ms, in other words, there is something stalling the connection into the container/pod somehow.

The way I run it is that two services attach to the container, one headless and one http. I’m curious if that could be part of the problem?

Have you learned anything more about the issue?

JorgenG on Nov 19, 2018

hi everyone. I wonder how can we know the root cause is the resource? This issue is happening to our software currently. I check CPU usage is only 40%, memory is 36%. is this the situation of thread starvation

vanhaxl on Jan 17, 2020