kubernetes: Exec liveness probe does not honor timeoutSeconds on 1.20.7

What happened?

We have a liveness probe with timeoutSeconds set to 5 but the probe doesn’t seem to be failing after 5s when the monitored endpoint takes minutes to respond.

What did you expect to happen?

We expect the probe to fail after 5seconds if the monitored endpoint takes longer than 5 seconds to respond.

How can we reproduce it (as minimally and precisely as possible)?

I can’t share my application but hopefully, the data included here will be enough to get you started. The only important thing about /api/serverstatus/hosted/internal/XXX is that this endpoint will block for XXX seconds before returning 200 code. We added it to our app to prove our theory.

This problem seems to be similar to Exec-type liveness or readiness probes ignore timeout;

Timeout not enforced

image

Timeout enforced at the curl level

image

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"c2b5237ccd9c0f1d600d3072634ca66cefdf272f", GitTreeState:"clean", BuildDate:"2021-08-04T18:03:20Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.7", GitCommit:"4966fd0dded6419fd23233a38c8905c67a3ea78e", GitTreeState:"clean", BuildDate:"2021-08-17T02:06:07Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.22) and server (1.20) exceeds the supported minor version skew of +/-1

Cloud provider

Azure - AKS 1.20.7

OS version

# On Linux:
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
$ uname -a
Linux aks-primary-34443167-vmss000000 5.4.0-1047-azure #49~18.04.1-Ubuntu SMP Thu Apr 22 21:28:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Install tools

Container runtime (CRI) and and version (if applicable)

Related plugins (CNI, CSI, …) and versions (if applicable)

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 31 (20 by maintainers)

Most upvoted comments

Note that the startup probe is also failing. @SergeyKanzhelev can you confirm that the liveness probe isn’t engaged until startup probe is all clear?

this is true, as documented.