kubeflow: Kubeflow installation: "Internal error occurred: failed calling webhook "webhook.cert-manager.io""

/kind bug

What steps did you take and what happened: Installation of Kubeflow v0.7.1 via kfctl and the kfctl_existing_arrikto.0.7.0.yaml file (this also happens with kfctl_k8s_istio.v1.0.0.yaml ) fails with the following error:

failed to apply: (kubeflow.error): Code 500 with message: kfApp Apply failed for kustomize: (kubeflow.error): Code 500 with message: Apply.Run Error error when creating "/tmp/kout354727301": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post https://kubernetes.default.svc:443/apis/webhook.cert-manager.io/v1beta1/mutations?timeout=30s: Service Unavailable

What did you expect to happen: Kubeflow installation to be a success

Anything else you would like to add:

$ kubectl get apiservices | grep webhook

v1beta1.webhook.cert-manager.io        cert-manager/cert-manager-webhook   False (FailedDiscoveryCheck)   89m

$ kubectl get all -n cert-manager

NAME                                           READY   STATUS    RESTARTS   AGE
pod/cert-manager-cainjector-74966df6bf-m5sbg   1/1     Running   0          90m
pod/cert-manager-f9ddbb-z6wtt                  1/1     Running   0          90m
pod/cert-manager-webhook-68d879df6-l88xh       1/1     Running   1          90m


NAME                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/cert-manager           ClusterIP   10.233.59.240   <none>        9402/TCP   90m
service/cert-manager-webhook   ClusterIP   10.233.57.170   <none>        443/TCP    90m


NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cert-manager              1/1     1            1           90m
deployment.apps/cert-manager-cainjector   1/1     1            1           90m
deployment.apps/cert-manager-webhook      1/1     1            1           90m

NAME                                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/cert-manager-cainjector-74966df6bf   1         1         1       90m
replicaset.apps/cert-manager-f9ddbb                  1         1         1       90m
replicaset.apps/cert-manager-webhook-68d879df6       1         1         1       90m

$ kubectl get apiservice v1beta1.webhook.cert-manager.io -o yaml

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  annotations:
    cert-manager.io/inject-ca-from-secret: cert-manager/cert-manager-webhook-tls
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apiregistration.k8s.io/v1beta1","kind":"APIService","metadata":{"annotations":{"cert-manager.io/inject-ca-from-secret":"cert-manager/cert-manager-webhook-tls"},"labels":{"app":"webhook","app.kubernetes.io/component":"cert-manager","app.kubernetes.io/instance":"cert-manager-v0.7.1","app.kubernetes.io/managed-by":"kfctl","app.kubernetes.io/name":"cert-manager","app.kubernetes.io/part-of":"kubeflow","app.kubernetes.io/version":"v0.7.1","kustomize.component":"cert-manager"},"name":"v1beta1.webhook.cert-manager.io"},"spec":{"group":"webhook.cert-manager.io","groupPriorityMinimum":1000,"service":{"name":"cert-manager-webhook","namespace":"cert-manager"},"version":"v1beta1","versionPriority":15}}
  creationTimestamp: "2020-02-28T13:13:08Z"
  labels:
    app: webhook
    app.kubernetes.io/component: cert-manager
    app.kubernetes.io/instance: cert-manager-v0.7.1
    app.kubernetes.io/managed-by: kfctl
    app.kubernetes.io/name: cert-manager
    app.kubernetes.io/part-of: kubeflow
    app.kubernetes.io/version: v0.7.1
    kustomize.component: cert-manager
  name: v1beta1.webhook.cert-manager.io
  resourceVersion: "53407"
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.webhook.cert-manager.io
  uid: b0ce5f61-367f-4078-9d10-ca6c20d3acec
spec:
  caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURMVENDQWhXZ0F3SUJBZ0lRVDZheFV1OHBVYnlDWnNDWHQ1eStrekFOQmdrcWhraUc5dzBCQVFzRkFEQkEKTVJ3d0dnWURWUVFLRXhOalpYSjBMVzFoYm1GblpYSXVjM2x6ZEdWdE1TQXdIZ1lEVlFRREV4ZGpaWEowTFcxaApibUZuWlhJdWQyVmlhRzl2YXk1allUQWVGdzB5TURBeU1qZ3hNekV6TXpaYUZ3MHlOVEF5TWpZeE16RXpNelphCk1FQXhIREFhQmdOVkJBb1RFMk5sY25RdGJXRnVZV2RsY2k1emVYTjBaVzB4SURBZUJnTlZCQU1URjJObGNuUXQKYldGdVlXZGxjaTUzWldKb2IyOXJMbU5oTUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQwpBUUVBdCtiZ1RRblYxbFdYR3pnUFl1STNJcnhKWHJPbFNzQUFidE80KytCR1VzYVkvTGdUanZSTEdxcVFCendXCk1Dc0s4ZzdNazZia0pXY2hVREZwMmgwYmN1R0hXWFN5bSt1ZzFSSHlzVHI0WExWY04veStPRGNBTHR4MmFSLzgKMGhhWnJkbWxFK0tURjI1emp5NjRDbXRaYUV3Ylo5cFBrVlJ4V0RCMDltQXBEMS9uTWxnWVVxbEtGNWwydWUvWgpnS2ZMTTZ2SnphRFZDZEt3b3JDa2NoakZqc2hvNDViM1BiMFk5RWFtYk1WUnl0eU14Ri9tVXpwUmFEREl6UTJwCkFHWXhhd3hDaER1M1NSdjVWdjNEbkRMQnpPTFNyWG1uNWJ6M1pRS04yNmszWjZjNmZkeDVhZVZ1V2o0bDNPcXgKTUpxa3c3VHIvdXBjVEt2ZlliQ1dlU2t4blFJREFRQUJveU13SVRBT0JnTlZIUThCQWY4RUJBTUNBcVF3RHdZRApWUjBUQVFIL0JBVXdBd0VCL3pBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQUNRYWV4dWZ0dE9Kd1Qzays4ZFZjCmFhaTlyeDVDMENIV3VvUENFOExQQ05SclI0ZzJzdUx5VldwTUJMWFdGLzV5aGcwdzRPMSsxOWE4Y0ZYLzdvRFAKMWpIdGplSWtXT0dSK2lMSXRVMjhwdG1JNFp1SHVrTnNCZDlZKzlERjFBV21ZVzR3VzdDNnRpbVJLenlvU0hqVgo4dFRMTzhMMGJKQWVLanBibHhpSzQ3aitKR2phUlBHZC8wbHg0VWMwYWcvTlczZUJqYktES0k5d3pTc01rdVl3CjVtcWtmekZpbGFrV2NKSHV0YmFKNEZlU1MzV01hakhYVWcrbW1rcVBsOWk0V2hUZlVDd1o5L2x0bzdZaDBMeU0KUU9HclZ6c3g0WFhsVVh0aHN5cFJpc3VySXRBdndPYzkwaDhHK0ZVZ3k2dzd5amcrZW5HMzVJUzZGWFhrb2RCbwo0UT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  group: webhook.cert-manager.io
  groupPriorityMinimum: 1000
  service:
    name: cert-manager-webhook
    namespace: cert-manager
    port: 443
  version: v1beta1
  versionPriority: 15
status:
  conditions:
  - lastTransitionTime: "2020-02-28T13:13:08Z"
    message: 'failing or missing response from https://10.233.57.170:443/apis/webhook.cert-manager.io/v1beta1:
      Get https://10.233.57.170:443/apis/webhook.cert-manager.io/v1beta1: Service
      Unavailable'
    reason: FailedDiscoveryCheck
    status: "False"
    type: Available

Environment:

behind enterprise proxy
Kubeflow version: 0.7.1
kfctl version: (use kfctl version): kfctl v1.0-rc.1-0-g963c787
Kubernetes platform: vanilla k8s
Kubernetes version: (use kubectl version): v1.15.3
OS (e.g. from /etc/os-release): 18.04.4 LTS (Bionic Beaver)"

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 39 (1 by maintainers)

Most upvoted comments

Issue-Label Bot is automatically applying the labels:

Label	Probability
area/jupyter	0.52

Please mark this comment with 👍 or 👎 to give our bot feedback! Links: app homepage, dashboard and code for this bot.

issue-label-bot[bot] on Aug 12, 2020

@Kloempi and myself was able to solve this problem following the steps required by this: https://github.com/jetstack/cert-manager/issues/2640

Do describe API Service on the webhook.cert-manager and get the IP address of the pod. It should say somthing like Error 503 service unavailable. Note that IP Address.
Edit /etc/kubernetes/manifests/kube-apiserver.yaml and find the “no_proxy” settings. Add the pod IP address for certmanager-webhook pod.
systemctl restart kubelet
verify the no_proxy setup kubectl exec -n kube-system kube-apiserver-{your master} – env | grep -i proxy
verify the API service is ready.
redeploy kflow

Not the best workaround, but i worked for us.

twittidai on Apr 21, 2020

After installing cert-manager using the manifest with no errors, I don’t see any services with webhook in their name:

kubectl get apiservices | grep webhook
<nothing>

medined on Mar 26, 2020

my situation is v0.7.1 uninstallation left cert-manager namespace. after delete the cert-manager namespace by

# https://cert-manager.io/docs/installation/uninstall/kubernetes/
kubectl delete apiservice v1beta1.webhook.cert-manager.io
kubectl delete namespace cert-manager

kfctl reapply everything works.

dynofu on Mar 12, 2020