cert-manager: cert-manager does not use ingress.class from Ingress annotated with cert-manager.io/cluster-issuer

Describe the bug:

When using the cert-manager.io/cluster-issuer annotation, for example like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myexample
  annotations:
    kubernetes.io/ingress.class: mycustomingressclass
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
    - hosts:
        - myexample.example.com
      secretName: myexample
  rules:
    - host: myexample.example.com
      http:
        paths:
          - backend:
              serviceName: myexampleservice
              servicePort: http

And with ClusterIssuer like this:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: myemail@example.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - http01:
          ingress: {}

The Ingress generated to handle the http01 challenge have no kubernetes.io/ingress.class annotation set.

Alternatively, if the ClusterIssuer would have ingress class set, for example to someotherclass, the generated Ingress would have the same class set, in this case to someotherclass.

Expected behaviour:

The Ingress generated to handle the http01 challenge should be annotated with the same ingress class as the Ingress annotated with cert-manager.io/cluster-issuer. In above case this should be mycustomingressclass.

As a workaround, the acme.cert-manager.io/http01-ingress-class attribute can be set on Ingress, but this just duplicates value of kubernetes.io/ingress.class:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myexample
  annotations:
    kubernetes.io/ingress.class: mycustomingressclass
    cert-manager.io/cluster-issuer: letsencrypt-prod
    # Note that below line sets the same ingress class like in above kubernetes.io/ingress.class
    acme.cert-manager.io/http01-ingress-class: mycustomingressclass
spec:
  tls:
    - hosts:
        - myexample.example.com
      secretName: myexample
  rules:
    - host: myexample.example.com
      http:
        paths:
          - backend:
              serviceName: myexampleservice
              servicePort: http

Anything else we need to know?:

I understand why ClusterIssuer have ability to set ingress class. This is required when manually creating Certificate resources. But in case of using the cert-manager.io/cluster-issuer annotation it works in surprising way as just adding cert-manager.io/cluster-issuer annotation does not work if using multiple ingress classes. One must also add acme.cert-manager.io/http01-ingress-class with just use the same value as kubernetes.io/ingress.class. I believe there should be option on ClusterIssuer/Issuer to allow for using ingress class from annotated ingress instead of the one set up on ClusterIssuer/Issuer. I would even consider enabling it by default, to avoid the surprise I had:)

/kind bug

About this issue

Original URL
State: open
Created 4 years ago
Reactions: 24
Comments: 43 (6 by maintainers)

Commits related to this issue

Ingress override in certificate. https://github.com/jetstack/cert-manager/issues/2538 — committed to wunderio/drupal-project-k8s by Jancis 3 years ago
Ingress override in certificate. https://github.com/jetstack/cert-manager/issues/2538 — committed to wunderio/drupal-project-k8s by Jancis 3 years ago

Most upvoted comments

@sagikazarmark is completely right!

The documentation says if no default ingress class is set it should stick to the ingress class (annotation) of the resource.

I have configured a cluster issuer:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
...
    - selector: {}
      http01:
        ingress: {}

And i have two ingress classes:

nginx (for internal use / private IPs only)
nginx-public (for external use / public IPs)

If i create a new ingress ressource with the annotation

kind: Ingress
apiVersion: extensions/v1beta1
...
  annotations:
    kubernetes.io/ingress.class: nginx-public

the acme-solver ingress still uses the default “nginx” ingress and the challenge could not be resolved! In older releases this definitely worked for me!

Cert-Manager: v0.15.1 k8s: 1.15.10

+10

discostur on Jun 19, 2020

As I mentioned in https://github.com/jetstack/cert-manager/issues/2314#issuecomment-575899971 I either consider this a bug in cert-manager itself or in the documentation.

From https://cert-manager.io/docs/usage/ingress/#supported-annotations

If not specified (edit: acme.cert-manager.io/http01-ingress-class annotation) and the acme-http01-edit-in-place annotation is not set, this defaults to the ingress class of the ingress resource.

The defaults to the ingress class of the ingress resource is clearly not true, and in my opinion it would be the logical thing to do, so for me, this is a bug, not a feature.

It’s worth keeping in mind that ingress class will become a part of the ingress spec soon.

sagikazarmark on Apr 25, 2020

My team and I are facing this exact same situation : we are migrating our cluster from Kubernetes 1.14 to 1.18 and wanted to upgrade to the last cert-manager version too (from 0.9.1 to 1.1.0)

Unfortunately without adding an extra annotation to all our existing Ingress objects, or hardcoding the class name of one of our Ingress Controllers in the ClusterIssuer specs, it’s impossible to generate a certificate : the issued Ingress cm-acme-http-solver is never attached to one of our existing Ingress Controllers and the challenge is never achieved.

We reverted in order to use old cert-manager 0.9.1 version and hopefully the certificate generation is still working in our new Kubernetes 1.18 cluster.

I strongly agree with all the remarks made in this ticket : it will be a good thing that a next cert-manager version brings this behaviour back (especially because the current documentation does send a big mixed signal about it, letting us thinking that’s it’s still here : https://cert-manager.io/docs/configuration/acme/http01/#class)

To sum up please considering offering in a next cert-manager version with this ClusterIssuer setup:

spec:
  acme:
      - http01:
          ingress: {}

The same behaviour as in the old spec (aka the issued Ingress cm-acme-http-solver have the kubernetes.io/ingress.class annotation with the same value of the source Ingress we have trying to generate a certificate for):

spec:
  acme:
    http01: {}

Thank you for your time, and all the hard work done with this project!

lboix on Feb 4, 2021

Is there any progress on this? I am stumbling upon this 2 year old abandoned issue on what appears to be a completely necessary fix. This completely breaks the ability to use HTTP01 on top of nginx if it only surfaces specific ingress classes. My only solution is to rely on DNS01 which thankfully wasn’t a huge issue since I use Cloudflare

khumps on Sep 27, 2022

Defining acme.cert-manager.io/http01-override-ingress-name (where ingress-name is an actual resource name) in certificate resource annotations seems to have solved our issue. But this is just a workaround for the issue that cert-manager can’t find existing ingress with the same domain name and edit-in-place defined, instead creates a new one that creates a new ip (on gcs for example) and does not resolve to be able to validate.

Jancis on Sep 20, 2021

Actually I’d like to come back to what @discostur mentioned. There are a lot of issues out there with engineers using networking.k8s.io/v1beta1 on >NGINX-controller 1.9.0, however the described behaviour from above seems to have changed.

I have a public and a private ingress controller and tried to challenge ACME obviously over the public one. But it always keeps choosing the private one by the kubernetes.io/ingress.class annotation although this annotation is deprecated and not in use by my system, as it shouldn’t anymore by best-practice. It still seems to register to the private ingress controller even after deleting it and also it keeps setting the same annotation to ACME ingresses. This is crazy… I will consider a fallback.

Possible solution: https://web.archive.org/web/20200911002539/http://ukhomeoffice.github.io/application-container-platform/how-to-docs/cert-manager-upgrade-from-v0.8.html

Cheers guys

goddamit-io on Sep 15, 2021

Further to this, I have found that adding the annotation acme.cert-manager.io/http01-ingress-class to the Ingress does not work if there is an existing order or cert, it will keep creating challenge ingresses with no kubernetes.io/ingress.class annotation at all. Sometimes deleting the pending order is enough, but sometimes we have found we have to delete the certificate resource so that it gets recreated, in order to flush the bad behavior and pick up the duplicate class annotation.

The documentation pages still documents this the way it is supposed to work, and the way it used to work, before the regression. That is, challenge Ingresses should default to the same class as the Ingress they are for.

@munnerz you’ve marked this a feature, not a bug, and said “This isn’t something we’re planning on changing imminently”, but it already has been changed, by some bug along the way 😄 You can verify this by comparing the current behavior to older versions or the documentation.

whereisaaron on Oct 21, 2020

@munnerz I believe this behavior is a regression, We have used cert-manager with custom classes forever and the ingress class for challenges use to behave as the documentation said (as @sagikazarmark highlighted). If you set in ingress.class on the Ingress then the challenge Ingress would use the same class. Then, after upgrading a few versions to 0.16, this suddenly stopped working and challenge Ingress’s suddenly had no ingress.class even if one was set on the original Ingress.

We were able to work around the regression bug by adding an explicit acme.cert-manager.io/http01-ingress-class, setting it to be the same value ingress.class. But we never had to do that before upgrading to 0.16.

I am not sure where the regression occurred, but I have clusters running v0.8.x that still behave as the documentation describes, but everything else that v0.16 now behaves as these various issues report, failing to default to the Ingress ingress.class. Update: Looking at #2314 it looks like the regression occurred between 0.8 and 0.11.

It really makes no sense that a challenge for an Ingress with an explicit ingress.class set would not use the same class. Why would HTTP01 challenges - by default - attempt to use a different ingress from the Ingress it is for?

whereisaaron on Oct 1, 2020

As part of today’s triage party, we have looked at this issue again. It seems like this is still a feature request even with the Ingress v1 API (i.e., with ingressClassName). We need someone willing to try to work on this for this issue to progress.

maelvls on Nov 6, 2023

Finally we managed to update cert-manager v0.9.1 to v1.6.1 in one shot, and chose to create one ClusterIssuer per Ingress Controller existing in our cluster. That means of course adding one extra annotation on each existing and future Ingress ressource, not clean and ideal but at least it’s working perfectly and we have full control on the process.

I wrote an article about it if that can help someone reading this GitHub issue thread : https://dev.to/lboix/upgrade-cert-manager-from-old-version-v010-to-v161-1i8f

If you have any question about the process, do not hesitate!

lboix on Dec 7, 2021

@lboix quick: I had to move to Cloudflare due to this issue… http01 challenge seems to be broken or I did not achieve implementing it on networking.k8s.io/v1beta1 for NGINX ingress controller v1.9.1…

If that somehow helps you out in any way I am glad to share you my experience.

Cheers mate

goddamit-io on Feb 4, 2021