cert-manager: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": context deadline exceeded

Im trying to install cert-certificates in an fresh k8s cluster machine Steps:

helm repo add jetstack https://charts.jetstack.io
helm repo update
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.crds.yaml
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.8.0 \
  --set installCRDs=true

and then I got this issue

Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": context deadline exceeded

The pods.service and endpoint seems to look ok

# kubectl get svc,pods,endpoints -n cert-manager
NAME                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/cert-manager           ClusterIP   <...>    <none>        9402/TCP   12m
service/cert-manager-webhook   ClusterIP   <...>   <none>        443/TCP    12m

NAME                                          READY   STATUS    RESTARTS   AGE
pod/cert-manager-6bbf595697-4z855             1/1     Running   0          12m
pod/cert-manager-cainjector-6bc9d758b-gg48g   1/1     Running   0          12m
pod/cert-manager-webhook-d98678bf5-wp7t9      1/1     Running   0          12m

NAME                             ENDPOINTS             AGE
endpoints/cert-manager           <...>     12m
endpoints/cert-manager-webhook   <...>   12m

can someone help to understand why am I getting this issue? Im running those command on Linux

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 7
Comments: 18 (2 by maintainers)

Most upvoted comments

kind of, I run this as workaround

kubectl delete validatingwebhookconfiguration validating-webhook-configuration
kubectl delete mutatingwebhookconfiguration mutating-webhook-configuration

klepiz on Jul 7, 2022

(cert-manager v1.9.0 via Helm Chart)

I had the same problem and it solved it by following, hope it helps any future readers

1. Below to fully delete all objects created by cert-manager otherwise it crashed while re-installing

helm uninstall cert-manager -n cert-manager
kubectl delete roles cert-manager-startupapicheck:create-cert -n cert-manager;
kubectl delete serviceaccount cert-manager-startupapicheck -n cert-manager;
kubectl delete serviceaccount default -n cert-manager;
kubectl delete jobs cert-manager-startupapicheck -n cert-manager;
kubectl delete rolebindings cert-manager-startupapicheck:create-cert -n cert-manager;

2. Also deleted all cert-manager related CRDs (Not sure if this was necessary but Certificates come up by itself once cert-manager works properly anyway)

kubectl delete -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.0/cert-manager.crds.yaml

3. Update Chart

helm repo add jetstack https://charts.jetstack.io
helm repo update

4. Install Chart with below config (Set host network true to make webhook pod run in the host’s network namespace & Set securePort to 10260 to prevent a conflict between the webhook and the kubelet)

helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.9.0 --set startupapicheck.timeout=5m --set installCRDs=true --set webhook.hostNetwork=true --set webhook.securePort=10260

Reference: https://cert-manager.io/docs/troubleshooting/webhook/#io-timeout

imageschool on Nov 16, 2022

(cert-manager v1.9.0 via Helm Chart)

I had the same problem and it solved it by following, hope it helps any future readers

1. Below to fully delete all objects created by cert-manager otherwise it crashed while re-installing
helm uninstall cert-manager -n cert-manager
kubectl delete roles cert-manager-startupapicheck:create-cert -n cert-manager;
kubectl delete serviceaccount cert-manager-startupapicheck -n cert-manager;
kubectl delete serviceaccount default -n cert-manager;
kubectl delete jobs cert-manager-startupapicheck -n cert-manager;
kubectl delete rolebindings cert-manager-startupapicheck:create-cert -n cert-manager;
2. Also deleted all cert-manager related CRDs (Not sure if this was necessary but Certificates come up by itself once cert-manager works properly anyway)
kubectl delete -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.0/cert-manager.crds.yaml
3. Update Chart
helm repo add jetstack https://charts.jetstack.io
helm repo update
4. Install Chart with below config (Set host network true to make webhook pod run in the host’s network namespace & Set securePort to 10260 to prevent a conflict between the webhook and the kubelet)
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.9.0 --set startupapicheck.timeout=5m --set installCRDs=true --set webhook.hostNetwork=true --set webhook.securePort=10260
Reference: https://cert-manager.io/docs/troubleshooting/webhook/#io-timeout

Thanks for the guidelines, @imageschool 's solution works to me

nduc4nh on Nov 18, 2022