cert-manager: CRDs shouldn't be templated in Helm...

Thank you everyone for all the input here!

I have a desire to have the CRDs in their own chart (option 4). Given that this topic is about whether to template or not to template those CRDs… perhaps we could:

• cert-manager-crds (templated)
• cert-manager-crds-helm (no template, put them in the /crds directory)

If we had both of those additional charts… people could pick their poison? Naming is up for debate but illustrates a possible route forward.

Apparently, the kube-prometheus-stack Helm chart, where I saw the option 3 pattern, moved to the option 4 😅

Here is the issue where they had the same discussion and a comment about automation for the CRD dedicated chat:

https://github.com/prometheus-community/helm-charts/issues/2612

if the templating was removed from the CRDs. Currently they just set labels.

My apologies for not making that clear in the list - I ran with the assumption that any templating would be removed in all scenarios. I agree with you that it’s an additional lifecycle concern that no one should have to deal with, and the other projects I’ve seen that use Helm to deploy CRDs are only applying a static dump. Thank you for bringing up that detail.

Presented options:

1. Remove the CRDs entirely

• cons: makes dev environments more difficult to setup
• pros: no more surprises with helm lifecycles

1. Place the CRDs into crds/

• cons: newer feature and already has known lifecycle caveats. doesn’t handle upgrades.
• pros: the dependency tree would be clear for multi-chart charts. easy dev bootstrapping.

1. Leave them in the chart, but create a boolean toggle for only them

• cons: invites possibilities for typos and misuse, not a common pattern
• pros: the dependency tree would be clear for multi-chart charts. easy dev bootstrapping. handles upgrades

1. Move the CRDs to a dedicated chart

• cons: ???
• pros: the dependency tree would be clear for multi-chart charts. easy dev bootstrapping. handles upgrades.

I really think option 4 is the best, given all the scenarios and concerns.

note: my deleted reply was to a mistaken interpretation. I hope this list helps create a more clear shared understanding. my apologies for coming in hot 😄

I did some tests with the FluxCD Kustomization controller and find somehow a “”“solution”“” to use the healthcheck feature. I simply patch the templatized manifests by replacing the labels with template anchors with static values.

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: cert-manager-crds
  namespace: flux-system
spec:
  interval: 12h
  url: https://github.com/cert-manager/cert-manager.git
  ref:
    tag: 'v1.12.3'
  ignore: |
    # exclude all
    /*
    # path to crds
    !/deploy/crds/
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: cert-manager-crds
  namespace: flux-system
spec:
  interval: 30m
  sourceRef:
    kind: GitRepository
    name: cert-manager-crds
    namespace: flux-system
  prune: true
  patches:
    - patch: |-
        apiVersion: apiextensions.k8s.io/v1
        kind: CustomResourceDefinition
        metadata:
          name: certificaterequests.cert-manager.io
          labels:
            app: 'cert-manager'
            app.kubernetes.io/name: 'cert-manager'
            app.kubernetes.io/instance: 'cert-manager'
      # ...omitted for brevity 
  healthChecks:
    - apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      name: certificaterequests.cert-manager.io
    # ...omitted for brevity

This is ugly but it works (having plain manifests will always better though), FluxCD is happy and the CRDs are deployed :

$ flux get kustomization
NAME                            REVISION                SUSPENDED       READY   MESSAGE
cert-manager-crds               v1.12.3@sha1:9479c815   False           True    Applied revision: v1.12.3@sha1:9479c815
...
$ k get crds | grep cert-manager.io
certificaterequests.cert-manager.io                   2023-08-09T16:43:25Z
certificates.cert-manager.io                          2023-08-09T16:43:25Z
challenges.acme.cert-manager.io                       2023-08-09T16:43:25Z
clusterissuers.cert-manager.io                        2023-08-09T16:43:25Z
issuers.cert-manager.io                               2023-08-09T16:43:25Z
orders.acme.cert-manager.io                           2023-08-09T16:43:25Z

Hello @hawksight, here is the link to the documentation about this feature : https://fluxcd.io/flux/components/kustomize/kustomization/#health-checks

For now, I use something like this in my CRDs kustomization in FluxCD, this works fine :

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - kyverno.yaml
  - metallb.yaml
  - openebs-jiva.yaml
  - openebs-localpv-lvm.yaml
  - traefik.yaml
  # Cert-Manager - there is no CRDs folder available
  - https://github.com/cert-manager/cert-manager/releases/download/v1.12.2/cert-manager.crds.yaml

For other operators available here, I use the healthcheck feature like this :

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: traefik-crds
  namespace: flux-system
spec:
  interval: 12h
  url: https://github.com/traefik/traefik-helm-chart.git
  ref:
    tag: 'v23.2.0'
  ignore: |
    # exclude all
    /*
    # path to crds
    !/traefik/crds/
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: traefik-crds
  namespace: flux-system
spec:
  interval: 30m
  sourceRef:
    kind: GitRepository
    name: traefik-crds
    namespace: flux-system
  prune: true
  healthChecks:
    - apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      name: ingressroutes.traefik.io
    - apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      name: ingressroutetcps.traefik.io
    - apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      name: ingressrouteudps.traefik.io
    - apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      name: middlewares.traefik.io
    - apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      name: middlewaretcps.traefik.io
    - apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      name: serverstransports.traefik.io
    - apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      name: tlsoptions.traefik.io
    - apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      name: tlsstores.traefik.io
    - apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      name: traefikservices.traefik.io

I just don’t see how to do it with Cert-Manager CRDs with a single link, but to be completely transparent, I’m by no mean a FluxCD guru, maybe there is an another way to do it 😃

Agreed. Helm now recommends putting crds in the crds folder of the chart. Would rather not have a separate chart for CRDs.

Sounds great! Thanks for pushing this forward when I fell off after a while.

Personally, I don’t like the idea of having 2 different charts too 😇

I take the liberty to quote myself on another issue: #5772 (comment)

-> The idea is to have a boolean that controls if CRDs are templated or not. -> Thereby we can have 2 different releases of the same chart.

• One managing the CRDs lifecycle
• The other the components

But this still doesn’t fix https://github.com/cert-manager/cert-manager/issues/6179#issuecomment-1764051690

Additionally, as @michaelajr already said, helm recommends putting the CRDs in the crds folder.

And to quote myself; if you’re doing gitops, the crds folder works perfectly and if you’re doing it manually, you might as well apply the crds folder 🤷

Since it’s just the labels that are templated, you can use spec.commonMetadata instead of patches. It makes the kustomzation a bit cleaner looking.

spec:
  commonMetadata:
    labels:
      app: "cert-manager"
      app.kubernetes.io/name: "cert-manager"
      app.kubernetes.io/instance: "cert-manager"

@hawksight Yep, that’s why I use the complete YAML file you linked with all CRDs bundled as a workaround. But as I mentioned at first when writing into this issue, the usage of this direct link doesn’t allow me to use the FluxCD healthcheck feature.

If you look at it closely, when using the single link, I use a kustomization with a kind of kustomize.config.k8s.io/v1beta1 to include all of my sub-files that include custom resources (see my Traefik example). When I want to use the healthcheck feature, the kustomization is a kind of kustomize.toolkit.fluxcd.io/v1, because it’s handled by a custom FluxCD controller with additional features including healthcheck.

Having a separate CRD folder with plain manifests without templating (inside or outside of the Helm charts) is convenient for tools like FluxCD from my point of view as a user, but I don’t know the implications on cert-manager side. For Traefik, I use the CRDs located inside the Helm chart and I disabled their management by Helm in my HelmRelease definition in Flux :

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: traefik
  namespace: flux-system
spec:
 # ...omitted for brevity
  chart:
    spec:
      chart: traefik
      version: '23.2.0'
      sourceRef:
        kind: HelmRepository
        name: traefik
        namespace: flux-system
  # ...omitted for brevity
  install:
    crds: Skip
  upgrade:
    crds: Skip

Since it looks like it’s a recommended practice to manage CRD outside of Helm (also mentioned in your documentation for production deployments), I used this general idea for all of my controller deployments including MetalLB, Kyverno or OpenEBS.

@f-bn, thank you for the example. I am completely new to flux so that’s helpful.

For cert-manager, we do publish the YAML CRDs in git here: https://github.com/cert-manager/cert-manager/tree/master/deploy/crds. So could you use that to do cert-manager CRD health checks, similarly to how you do it for the Traefik example? You may want to of course link to the v1.12.2 tag rather than master branch.

Yep, I found these CRDs, but the problem is they are templatized. However, for managing CRDs I don’t use the HelmRelease reconciliation in FluxCD, but a Kustomization reconciliation which uses kustomize under the hood.

That’s mainly the problem since I don’t see how I can provide the required values for the template with this and therefore applying the manifests correctly (I don’t find anything in the FluxCD documentation to substitute Helm template anchors with a Kustomization). If these CRDs weren’t templatized, this will solve the problem since my kustomization reconciliation will simply apply plain manifests like I did for Traefik, and then I can use healthchecks on these CRDs afterwards.

As an example for Traefik, I simply asked the Kustomization reconciliation to pull CRDs in the Traefik Helm chart and apply them as-is. Since these are plain CRDs manifests, this works perfectly and is very simple to manage.

One dirty hack I could use is kustomization patches and replace templatized metadata labels with static values. I’m not sure this will works, I need to test it.

Updating CRDs is definitely a challenge when they are put into the crds folder. The pattern we follow for charts with CRDs is to do a kubectl apply -f path/to/crds before the helm update. This ensures any CRD updates are captured.

Additionally, having them in the crds folder - and not templated in the Helm chart - allows things to continue to work if the component is ever removed. We’ve run into this when upgrading ArgoCD. Had to remove it first. But because the Application and Project CRDs and CRs remained - there were no issues.

I understand the dilemma maintainers have a round this. No easy answer.

Thanks for the input on this issues. There are some other considerations and difference with helm that we have documented here.

tl;dr our CRDs can change and we want them to be in sync with the components. Helm’s crds folder doesn’t allow uprading / removal etc.

I’m not sure I understand the problem precisely, is it that:

• You don’t want any template logic in the CRDs?
• You want the CRDs to use the crd directory in the helm chart (we do not do this).?

@bradenwright - what is your automation? Is it outputting that message as an actual blocking error, or as a warning? As a workaround, you could apply the CRDs as YAML directly and use the chart for cert-manager without the installCRDs flag being set?

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.3/cert-manager.crds.yaml
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.12.3

@f-bn could you link to the flux healthcheck feature for CRDs please? Interested to see what that requires.