cert-manager: Unable to apply Cert Manager v1.7.0 CRDs

Describe the bug: The following error occurs when updating the Cert Manager CRDs from v1.6.1 to v1.7.0.

➜  ~ kubectl apply -f certificates.cert-manager.io.yaml --server-side=true
The CustomResourceDefinition "certificates.cert-manager.io" is invalid: 
* spec.conversion.strategy: Required value
* spec.conversion.webhookClientConfig: Forbidden: should not be set when strategy is not set to Webhook

The same error occurs when applying any of the other CRDs.

Expected behaviour: The CRDs to apply successfully.

Steps to reproduce the bug: Use kubectl, with server-side apply, to apply the v1.7.0 release CRDs onto the v1.6.1 release CRDs.

Anything else we need to know?:

Environment details::

Kubernetes version: 1.21
Cloud-provider/provisioner: AWS-EKS
cert-manager version: 1.7.0
Install method: Helm (chart) & static manifests (CRDs)

/kind bug

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 14
Comments: 16 (2 by maintainers)

Most upvoted comments

for crd_name in certificaterequests.cert-manager.io certificates.cert-manager.io challenges.acme.cert-manager.io clusterissuers.cert-manager.io issuers.cert-manager.io orders.acme.cert-manager.io; do
  manager_index="$(kubectl get crd "${crd_name}" --show-managed-fields --output json | jq -r '.metadata.managedFields | map(.manager == "cainjector") | index(true)')"
  kubectl patch crd "${crd_name}" --type=json -p="[{\"op\": \"remove\", \"path\": "/metadata/managedFields/${manager_index}\"}]"
done

"/metadata needs an additional backslash:

for crd_name in certificaterequests.cert-manager.io certificates.cert-manager.io challenges.acme.cert-manager.io clusterissuers.cert-manager.io issuers.cert-manager.io orders.acme.cert-manager.io; do
  manager_index="$(kubectl get crd "${crd_name}" --show-managed-fields --output json | jq -r '.metadata.managedFields | map(.manager == "cainjector") | index(true)')"
  kubectl patch crd "${crd_name}" --type=json -p="[{\"op\": \"remove\", \"path\": \"/metadata/managedFields/${manager_index}\"}]"
done

+13

aljoshare on Feb 22, 2022

@irbekrm I’d suggest that any automated tool able to manage cert-manager CRDs will be using server-side apply so will be impacted.

I can confirm, I ran into the problem while updating cert-manager with Flux v2

+11

MattiasGees on Mar 19, 2022

FWIW, I would argue that this is not an acceptable resolution; we have multiple clusters all managed with Flux V2 and due to this change we are now forced to manually update every cluster which is antithetical to what we’re trying to do with GitOps.

Edit: also - for those needing the link to the resolution since it wasn’t linked in the closing comment, here it is https://cert-manager.io/docs/release-notes/release-notes-1.7#upgrading-with-server-side-apply

apshoemaker on Nov 3, 2022

@stevehipwell’s solution works quite well, with a small adjustment (add crd in the first command):

for crd_name in certificaterequests.cert-manager.io certificates.cert-manager.io challenges.acme.cert-manager.io clusterissuers.cert-manager.io issuers.cert-manager.io orders.acme.cert-manager.io; do
  manager_index="$(kubectl get crd "${crd_name}" --show-managed-fields --output json | jq -r '.metadata.managedFields | map(.manager == "cainjector") | index(true)')"
  kubectl patch crd "${crd_name}" --type=json -p="[{\"op\": \"remove\", \"path\": \"/metadata/managedFields/${manager_index}\"}]"
done

jonkerj on Feb 22, 2022

@irbekrm a solution is to do the following for each CRD before applying the new one over the top.

manager_index="$(kubectl get crd "${crd_name}" --show-managed-fields --output json | jq -r '.metadata.managedFields | map(.manager == "cainjector") | index(true)')"
kubectl patch crd "${crd_name}" --type=json -p="[{\"op\": \"remove\", \"path\": \"/metadata/managedFields/${manager_index}\"}]"

EDITED - Thanks @jonkerj for spotting my typo.

stevehipwell on Feb 22, 2022

manager_index=“$(kubectl get “${crd_name}” --show-managed-fields --output json | jq -r ‘.metadata.managedFields | map(.manager == “cainjector”) | index(true)’)” kubectl patch crd “${crd_name}” --type=json -p=“[{"op": "remove", "path": "/metadata/managedFields/${manager_index}"}]”

Thanks for looking into this @stevehipwell can verify that modifying managed fields to remvoe the cainjector’s ownership via those commands work and allows to upgrade to v1.7 via server side apply.

So at the moment from user perspective there are two options to upgrade - either modify managed fields or use client-side apply. We’ll be looking into how this problem can be fixed in cainjector in general. We’d briefly considered re-instating the conversion webhook till there is a fix in cainjector, but this may not be feasible.

There are other installation mechanisms that might be using server-side apply (i.e Flux), I’ve not yet looked into whether upgrading via those result in the same issue.

irbekrm on Feb 10, 2022