cert-manager: cannot issue certificated with cert manager, not sure why.

REQUEST FOR HELP?

/kind bug

/kind feature

I have installed the cert-manager, and it works. I have setup an issuer to use the http service per the documentation: apiVersion: certmanager.k8s.io/v1alpha1

kind: Issuer
metadata:
  name: letsencrypt-staging
  namespace: default 
spec:
  acme:
    # The ACME server URL
    server: https://acme-staging.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: security@mydomain.ai
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging
    # Enable the HTTP-01 challenge provider
    http01: {}

Then I tried to create a certificate:

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: staging-mydomain-ai
  namespace: default
spec:
  secretName: staging-mydomain-ai-tls
  issuerRef:
    name: letsencrypt-staging
  commonName: staging.mydomain.ai
  dnsNames:
  - staging.mydoamin.ai
  acme:
    config:
    - http01:
        ingress: mydomain-staging 
      domains:
      - staging.mydomain.ai

I have such an ingress working, but when I describe it seems to get stuck on getting the cert issued. (I’m using google cloud platform) and the issuer is a namespace issuer (not cluster). The kubectl is running in kube-system of course with the cert-manager, and my services are running in the same namespace (default)

rather than issue the certificate I get this:

ame:         staging-mydomain-ai
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"Certificate","metadata":{"annotations":{},"name":"staging-mydomain-ai","namespace":"default"},"spec":...
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Certificate
Metadata:
  Cluster Name:                   
  Creation Timestamp:             2018-01-27T20:01:10Z
  Deletion Grace Period Seconds:  <nil>
  Deletion Timestamp:             <nil>
  Generation:                     0
  Resource Version:               16691689
  Self Link:                      /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificates/staging-mydomain-ai
  UID:                            d1ef0cc4-039c-11e8-b8a7-42010a9c0017
Spec:
  Acme:
    Config:
      Domains:
        staging.mydomain.ai
      Http 01:
        Ingress:  mydomain-staging
  Common Name:    staging.mydomain.ai
  Dns Names:
    staging.mydomain.ai
  Issuer Ref:
    Name:       letsencrypt-prod
  Secret Name:  staging-mydomain-ai-tls
Events:
  Type     Reason                 Age   From                     Message
  ----     ------                 ----  ----                     -------
  Warning  ErrorCheckCertificate  3m    cert-manager-controller  Error checking existing TLS certificate: secret "staging-scoutr-ai-tls" not found
  Normal   PrepareCertificate     3m    cert-manager-controller  Preparing certificate with issuer
  Normal   PresentChallenge       3m    cert-manager-controller  Presenting http-01 challenge for domain staging.mydomain.ai
  Normal   SelfCheck              3m    cert-manager-controller  Performing self-check for domain staging.mydomain.ai

When I tried to re-run it generate a certificate, and when I moved to production again it got stuck like this. I don’t know what’s going on or why it cannot connect? or what it’s stuck on the self perform self check? I don’t have a clue what’s going on. Also how can I re-confgure the shim after to use ingress? Can I use a local issuer or only clusterissuer?

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 4
  • Comments: 23 (10 by maintainers)

Most upvoted comments

I’m currently struggling through deploying this to GCP and will send a full list of gotchas when I’m done (hopefully in a PR), but basically there is no actual “you need these N things to get it working.” The “quick start” is “helm install stable/cert-manager” but in reality, it’s “install cert-manager, install an issuer, and maybe install a certificate but we don’t provide any clear path to doing that.”

@bbriggs I am struggling to configure cert-manager on my GCP kubernetes cluster too. A guide or improved documentation would be awesome!

It’s really sad when this is the best documentation I can find on how to actually get cert-manager working.

Okay. I think it makes sense now, but just to be exhaustively clear…

First I will create this Ingress resource:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    kubernetes.io/ingress.class: "gce"
spec:
  tls:
  - hosts:
    - www.mysite.com
    secretName: my-site-tls
  rules:
  - host: www.mysite.com
    http:
     paths:
     - backend:
         serviceName: my-site-service
         servicePort: 80

Once the IP address is allocated I will point the A record for www.mysite.com to the newly allocated address. Once I’ve confirmed www.mysite.com resolves to the Ingress I will update the ingress to match the following…

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.class: "gce"
spec:
  tls:
  - hosts:
    - www.mysite.com
    secretName: my-site-tls
  rules:
  - host: www.mysite.com
    http:
     paths:
     - backend:
         serviceName: my-site-service
         servicePort: 80

At this point cert-manager will begin the process of verifying the endpoint and creating the tls secret. Does this sound like the correct way to go if I don’t want to risk the failed verification timeout?

@DoryZi performing self-check means cert-manager is trying to make an HTTP request for http://staging.mydomain.ai/.wellknown/acme-challenge/... to make sure the challenge file is there before it asks the ACME server to check.

It will be stuck if it can’t make that request because, e.g.

  • the domain name staging.mydomain.ai doesn’t exist
  • the domain name staging.mydomain.ai doesn’t resolve to the IP of your ingress controller
  • your Ingress isn’t working for some reason (e.g. target path annotation or forced ssl redirect)
  • your network permission don’t let cert-manager make the request

If it is something to do with your Ingress settings, like it uses rewrite-target see #286 for a way to deal with that.

I found the solution after carefully reading the docs. By using the annotation certmanager.k8s.io/acme-http01-edit-in-place: "true" the old behavior can still be used. I wasn’t aware that the default behavior had changed, so I assumed this was specific to GCE.

I’ll see if I add some documentation on this, since there’s nothing here ATM: https://cert-manager.readthedocs.io/en/master/tutorials/acme/http-validation.html

When creating an ingress are we supposed to leave out the initial tls options?

For instance:

  1. Create an ingress
  2. Point mysite.com to ingress IP
  3. Create cert-manager Certificate resource
  4. Cert manager automagically adds tls to the targeted ingress !!! <---- not sure about this

Is that how the workflow should be setup?

@ericuldall you either create a Certificate and Ingress, or an Ingress with the acme annotation. In both cases the Ingress should include the ‘tls’ reference to the certifiate/TLS Secret (whether that Secret exists yet or not).

The clean method it to create the Certificate resource first, using ‘ingressClass’ in the Certificate to identify your ingress controller. The cert-manager will create a temporary Ingress, issue the certificate and create the Secret. Then, whenever you do deploy the Ingress for your application, you don’t need any Ingress annotation at all, and the TLS Secret will be there ready to go.

The alternative, convenience method is you deploy an Ingress with the acme annotation, and that refers to the TLS Secret (that doesn’t exist yet). The ingress controller will just sit on the Ingress waiting for the named Secret to appear. cert-manager will see the Ingress annotation and create a Certificate resource for you, which will trigger the certificate to be issued and the certificate Secret created, When the Secret appears, the ingress controller will pick it up.