cert-manager: Cannot create http01 ClusterIssuer with DigitalOcean provider using new static manifests

Describe the bug:

Following documentation to install with static manifests, then attempting to create an Issuer or ClusterIssuer on a fresh DO k8s cluster results in the following error:

Error from server (InternalError): error when creating "30-staging-clusterissuer.yml": Internal error occurred: failed calling admission webhook "clusterissuers.admission.certmanager.k8s.io": the server is currently unable to handle the request

Issue seems quite different from #1103.

Expected behaviour:

An Issuer or ClusterIssuer should be able to be created after following documentation instructions for static manifests.

Steps to reproduce the bug:

create a k8s cluster on Digital Ocean
follow cert-manager static manifest installation instructions
attempt to create an Issuer or ClusterIssuer

Anything else we need to know?:

I first tried to setup cert-manager 0.5.2 with static manifests but the documentation is severely lacking, CRDs are absent, and there are a number of other missing things, like pods failing to start with log output missing secret "webhook-ca".

While looking for a way to solve it I noticed that there seems to be quite a refactoring with much better documentation and manifests on master. With CRDs set up and namespace created, everything seemed to be in order except that I had to apply -f with --validate=false due to #1143.

I then proceeded to create a ClusterIssuer following this part of the documentation:

$ kubectl apply -f 30-staging-clusterissuer.yml 
Error from server (InternalError): error when creating "30-staging-clusterissuer.yml": Internal error occurred: failed calling admission webhook "clusterissuers.admission.certmanager.k8s.io": the server is currently unable to handle the request

Nothing of significance appears in the pod logs.

Since there were no match in cert-manager issues I looked for similar errors in other kubernetes projects involving admission webhooks and found this.

Check that your cluster has aggregate api server enabled. Test that the configmap extension-apiserver-authentication-reader in kube-system namespace has key requestheader-client-ca-file

by running kubectl describe configmap -n kube-system extension-apiserver-authentication, which does contain requestheader-client-ca-file.

So I ran kubectl get apiservice clusterissuers.admission.certmanager.k8s.io -o yaml which I suppose is expected to return something along the lines of:

status:
  conditions:
  - lastTransitionTime: 2018-02-27T07:59:50Z
    message: all checks passed
    reason: Passed
    status: "True"
    type: Available

But returns this error instead:

Error from server (NotFound): apiservices.apiregistration.k8s.io "clusterissuers.admission.certmanager.k8s.io" not found

So I tried a more general kubectl get apiservice for which the output contains a single reference to anything related to certmanager.k8s.io:

v1beta1.admission.certmanager.k8s.io   cert-manager/cert-manager-webhook   False (FailedDiscoveryCheck)   26m

with no trace of issuers.admission.certmanager.k8s.io, clusterissuers.admission.certmanager.k8s.io, or certificates.admission.certmanager.k8s.io.

The output of kubectl describe apiservice v1beta1.admission.certmanager.k8s.io contains:

Status:
  Conditions:
    Last Transition Time:  2018-12-14T14:34:51Z
    Message:               no response from https://10.245.20.156:443: Get https://10.245.20.156:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
    Reason:                FailedDiscoveryCheck
    Status:                False
    Type:                  Available

Environment details::

Kubernetes version (e.g. v1.10.2): v1.12.3
Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): DigitalOcean
cert-manager version (e.g. v0.4.0): v0.6.0 (master)
Install method (e.g. helm or static manifests): static manifests

/kind bug

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 2
Comments: 18 (4 by maintainers)

Most upvoted comments

Is there a resolution here? It looks very similar to what I’m seeing:

Error from server (InternalError): error when creating "cluster-issuer-prod.yaml": Internal error occurred: failed calling admission webhook "clusterissuers.admission.certmanager.k8s.io": the server could not find the requested resource

+11

adammartin on Jan 25, 2019

Seems like you didn’t quite catch it, or maybe I wasn’t clear… I admit it’s quite a convoluted story. As I mentioned earlier (see the links of my original message):

Attempt to install 0.5.2

I first tried to install 0.5.2 following the matching “latest” documentation (not master), the only instructions being:

With static manifests

As some users may not be able to run Tiller in their own environment, static Kubernetes deployment manifests are provided which can be used to install cert-manager.

You can get a copy of the static manifests from the deploy directory.

Commands used:

kubectl apply -f with-rbac-webhook.yaml # fails because of missing CRD
kubectl apply -f 00-crds.yml (from master)
kubectl apply -f with-rbac-webhook.yaml # success

… except now pods won’t start due to missing webhook-ca secret on one pod and another error (I can’t recall) on the other pod.

Attempt to install 0.6 from master (i.e with webhook)

I followed the exact instructions here except with --validate=false due to #1143.

kubectl apply -f 00-crds.yml
kubectl create namespace cert-manager
kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true
kubectl apply -f cert-manager.yml --validate=false

Attempt to install 0.6 without webhook

Same commands, except cert-manager.yml comes from a few commits earlier, and without --validate (since that’s not an issue).

kubectl apply -f 00-crds.yml
kubectl create namespace cert-manager
kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true
kubectl apply -f cert-manager.yml

lloeki on Dec 17, 2018

As you’ve noticed, we’ve removed the separation between different manifest types in favour of an all-in-one bundle for the upcoming v0.6 release.

RBAC-only is the way to go for sure but how are you supposed to install the no-webhook variant with a single static manifest? Or is there no no-webhook option anymore by design?

You should have a much smoother experience if you stick to the latest release branch of the project

That’s what I thought at first but to be honest as a static manifest user, master was a much more pleasant experience 😉

lloeki on Dec 17, 2018