So in the end I ended up installing 1.5.5 which did end up just working immediately, and I can confirm that there are actually no dangling resources to be seen…

➜  ~ kubectl get challenge --all-namespaces
No resources found

So at this stage, I’m out of ideas, but I have a working setup, just with cert-manager 1.5 instead of 1.7. I can only assume that there is actually a bug in 1.7 that is causing this, as 1.5 works and does not indicate any dangling resources what so ever, yet 1.7 does not.

I also ran cmctl again just to confirm that it still says there is no upgrade work to be performed;

➜  ~ cmctl upgrade migrate-api-version
Checking all CustomResourceDefinitions have storage version set to "v1"
All CustomResourceDefinitions have "v1" configured as the storage version.
Looking for CRDs that contain resources that require migrating to "v1"...
Nothing to do. cert-manager CRDs do not have "status.storedVersions" containing old API versions. You may proceed to upgrade to cert-manager v1.7.

I’d like to upgrade to newer cert-manager versions at some stage, but right now this doesn’t seem possible.

Also thank you @mwhaubold for sharing the solution that worked for you, it’s appreciated 😃

I faced the same issue @RobinJ1995 .

We upgrade to 1.8. and weren’t able to issue new certificates.

I tried to patch the CRDs and run the cmctl upgrade like mentioned. After downgrading to 1.5 and running cmctl upgrade migrate-api-version it prompted with Nothing to do, as well as patching the CRDs. I still faced the conversion from acme.cert-manager.io/v1alpha2 issue.

I figured based on the conversation above we had an issue on one dangling challenge too. There were no challenges displayed even though we had some open certificate requests and orders.

In our case I knew which namespace was problematic, as I wasn’t able to list the challenges in the particular namespace. The same convert problem with v1alpha2 was prompted also on --all-namespace. In my case, I knew which namespace was problematic so this might not apply to the issue you are facing Robin.

After running

kubectl get challenge -n <namespace>

> failed to list *v1.Challenge: request to convert CR from an invalid group/version: acme.cert-manager.io/v1alpha2

I was able to resolve this by installing the old CRDs (from v1.4.0) while being on Version 1.5 of cert-manager via crds-managed-separately.

 kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.4.0/cert-manager.crds.yaml    

After installing the old CRDs the dangling challenge was finally displayed (it was pretty old… 🙄). And I was able to remove it simply via

kubectl  delete challenge <challange-name> -n <namespace>

After the old challenges was removed new challenges were immediatly issued, which now also use the correct v1 CRDs. I hope the upgrade will now work for us smoothly.

Maybe this will helps someone. Cheers guys for the help so far 💚

Re-installing cert-manager 0.11 is also not an option, as Kubernetes 1.22 does not supply most of the APIs anymore that were used for cert-manager 0.11’s resources.

You don’t need cert-manager v0.11 to re-instate v1alpha2 API. I’d suggest that you re-install cert-manager v1.5 that still understands v1alpha2 API, re-run the upgrade instructions as per the docs and re-attempt the upgrade then.

Thanks @irbekrm and @Sasuke0713 for your time posting on this issue.

I followed @irbekrm steps and installed v1.5.4 then ran through the steps in the resource upgrade instructions - That fixed my initial issue.

I also used the verification steps posted by @Sasuke0713; these were really helpful in proving it was all working again properly.

Thanks a lot guys!!!

Re-installing cert-manager 0.11 is also not an option, as Kubernetes 1.22 does not supply most of the APIs anymore that were used for cert-manager 0.11’s resources.

You don’t need cert-manager v0.11 to re-instate v1alpha2 API. I’d suggest that you re-install cert-manager v1.5 that still understands v1alpha2 API, re-run the upgrade instructions as per the docs and re-attempt the upgrade then.

Thanks for that suggestion. I uninstalled cert-manager 1.7 via Helm and removed its namespace, and installed cert-manager 1.5.

➜  ~ kubectl get challenge
Error from server: conversion webhook for acme.cert-manager.io/v1alpha2, Kind=Challenge failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": x509: certificate signed by unknown authority
➜  ~ kubectl get challenge --insecure-skip-tls-verify
Error from server: conversion webhook for acme.cert-manager.io/v1alpha2, Kind=Challenge failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": x509: certificate signed by unknown authority
➜  ~ cmctl upgrade migrate-api-version
Checking all CustomResourceDefinitions have storage version set to "v1"
All CustomResourceDefinitions have "v1" configured as the storage version.
Looking for CRDs that contain resources that require migrating to "v1"...
Nothing to do. cert-manager CRDs do not have "status.storedVersions" containing old API versions. You may proceed to upgrade to cert-manager v1.7.

And upon uninstalling 1.5 and installing 1.7 again I am once again greeted with the familiar

➜  ~ kubectl get challenge
Error from server: request to convert CR from an invalid group/version: acme.cert-manager.io/v1alpha2

…

Did you try patching the finalizers?

kubectl patch crd/certificaterequests.cert-manager.io -p '{"metadata":{"finalizers":[]}}' --type=merge
kubectl patch crd/orders.acme.cert-manager.io -p '{"metadata":{"finalizers":[]}}' --type=merge

Obviously yours would be with the challenges not the orders/certificaterequests

The issue is not that I wasn’t able to delete the resources, because they were gone according to the API (otherwise it would not have allowed me to delete the associated CRDs, either). The issue is that now I’m running a newer version of cert-manager and it is somehow still encountering v1alpha2 resources in my cluster even though they were all removed before proceeding with installing the new version of cert-manager.

Re-installing cert-manager 0.11 is also not an option, as Kubernetes 1.22 does not supply most of the APIs anymore that were used for cert-manager 0.11’s resources.

So in the end I ended up installing 1.5.5 which did end up just working immediately, and I can confirm that there are actually no dangling resources to be seen…

➜  ~ kubectl get challenge --all-namespaces
No resources found

So at this stage, I’m out of ideas, but I have a working setup, just with cert-manager 1.5 instead of 1.7. I can only assume that there is actually a bug in 1.7 that is causing this, as 1.5 works and does not indicate any dangling resources what so ever, yet 1.7 does not. I also ran cmctl again just to confirm that it still says there is no upgrade work to be performed;

➜  ~ cmctl upgrade migrate-api-version
Checking all CustomResourceDefinitions have storage version set to "v1"
All CustomResourceDefinitions have "v1" configured as the storage version.
Looking for CRDs that contain resources that require migrating to "v1"...
Nothing to do. cert-manager CRDs do not have "status.storedVersions" containing old API versions. You may proceed to upgrade to cert-manager v1.7.

I’d like to upgrade to newer cert-manager versions at some stage, but right now this doesn’t seem possible. Also thank you @mwhaubold for sharing the solution that worked for you, it’s appreciated 😃

I faced the same issue @RobinJ1995 .

We upgrade to 1.8. and weren’t able to issue new certificates.

I tried to patch the CRDs and run the cmctl upgrade like mentioned. After downgrading to 1.5 and running cmctl upgrade migrate-api-version it prompted with Nothing to do, as well as patching the CRDs. I still faced the conversion from acme.cert-manager.io/v1alpha2 issue.

I figured based on the conversation above we had an issue on one dangling challenge too. There were no challenges displayed even though we had some open certificate requests and orders.

In our case I knew which namespace was problematic, as I wasn’t able to list the challenges in the particular namespace. The same convert problem with v1alpha2 was prompted also on --all-namespace. In my case, I knew which namespace was problematic so this might not apply to the issue you are facing Robin.

After running

kubectl get challenge -n <namespace>

> failed to list *v1.Challenge: request to convert CR from an invalid group/version: acme.cert-manager.io/v1alpha2

I was able to resolve this by installing the old CRDs (from v1.4.0) while being on Version 1.5 of cert-manager via crds-managed-separately.

 kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.4.0/cert-manager.crds.yaml    

After installing the old CRDs the dangling challenge was finally displayed (it was pretty old… roll_eyes). And I was able to remove it simply via

kubectl  delete challenge <challange-name> -n <namespace>

After the old challenges was removed new challenges were immediatly issued, which now also use the correct v1 CRDs. I hope the upgrade will now work for us smoothly.

Maybe this will helps someone. Cheers guys for the help so far green_heart

This one helped me. Thanks @MarkWarneke

Hi @MarkWarneke

I also downgrade the cert-managet to v1.5.5 and then again downgrade by installing old CRD’s (from v1.4.0) while being on Version 1.5 of cert-manager via crds-managed-separately.

The output of applying CRD’s of v1.4.0

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.4.0/cert-manager.crds.yaml    
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io configured
Warning: Detected changes to resource clusterissuers.cert-manager.io which is currently being deleted.
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io configured

and when trying to list old dangling clusterissuers resources. I am getting below error

kubectl get clusterissuers -A
Error from server: conversion webhook for cert-manager.io/v1alpha2, Kind=ClusterIssuer failed: Post "https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s": x509: certificate signed by unknown authority

So in the end I ended up installing 1.5.5 which did end up just working immediately, and I can confirm that there are actually no dangling resources to be seen…

➜  ~ kubectl get challenge --all-namespaces
No resources found

So at this stage, I’m out of ideas, but I have a working setup, just with cert-manager 1.5 instead of 1.7. I can only assume that there is actually a bug in 1.7 that is causing this, as 1.5 works and does not indicate any dangling resources what so ever, yet 1.7 does not. I also ran cmctl again just to confirm that it still says there is no upgrade work to be performed;

➜  ~ cmctl upgrade migrate-api-version
Checking all CustomResourceDefinitions have storage version set to "v1"
All CustomResourceDefinitions have "v1" configured as the storage version.
Looking for CRDs that contain resources that require migrating to "v1"...
Nothing to do. cert-manager CRDs do not have "status.storedVersions" containing old API versions. You may proceed to upgrade to cert-manager v1.7.

I’d like to upgrade to newer cert-manager versions at some stage, but right now this doesn’t seem possible. Also thank you @mwhaubold for sharing the solution that worked for you, it’s appreciated 😃

I faced the same issue @RobinJ1995 .

We upgrade to 1.8. and weren’t able to issue new certificates.

I tried to patch the CRDs and run the cmctl upgrade like mentioned. After downgrading to 1.5 and running cmctl upgrade migrate-api-version it prompted with Nothing to do, as well as patching the CRDs. I still faced the conversion from acme.cert-manager.io/v1alpha2 issue.

I figured based on the conversation above we had an issue on one dangling challenge too. There were no challenges displayed even though we had some open certificate requests and orders.

In our case I knew which namespace was problematic, as I wasn’t able to list the challenges in the particular namespace. The same convert problem with v1alpha2 was prompted also on --all-namespace. In my case, I knew which namespace was problematic so this might not apply to the issue you are facing Robin.

After running

kubectl get challenge -n <namespace>

> failed to list *v1.Challenge: request to convert CR from an invalid group/version: acme.cert-manager.io/v1alpha2

I was able to resolve this by installing the old CRDs (from v1.4.0) while being on Version 1.5 of cert-manager via crds-managed-separately.

 kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.4.0/cert-manager.crds.yaml    

After installing the old CRDs the dangling challenge was finally displayed (it was pretty old… 🙄). And I was able to remove it simply via

kubectl  delete challenge <challange-name> -n <namespace>

After the old challenges was removed new challenges were immediatly issued, which now also use the correct v1 CRDs. I hope the upgrade will now work for us smoothly.

Maybe this will helps someone. Cheers guys for the help so far 💚

PROBLEM In case this helps anyone, we were seeing the following in the cert-manager logs after upgrading (1.5.3 -> 1.8.0):

failed to list *v1.Order: request to convert CR from an invalid group/version: acme.cert-manager.io/v1alpha2
Failed to watch *v1.Order: failed to list *v1.Order: request to convert CR from an invalid group/version: acme.cert-manager.io/v1alpha2

Running cmctl upgrade migrate-api-version --qps 5 --burst 10 showed that everything was fine:

Checking all CustomResourceDefinitions have storage version set to "v1"
All CustomResourceDefinitions have "v1" configured as the storage version.
Looking for CRDs that contain resources that require migrating to "v1"...
Nothing to do. cert-manager CRDs do not have "status.storedVersions" containing old API versions. You may proceed to upgrade to cert-manager v1.7.

Running the finalizer patches below also made no changes:

kubectl patch crd/certificaterequests.cert-manager.io -p '{"metadata":{"finalizers":[]}}' --type=merge
kubectl patch crd/orders.acme.cert-manager.io -p '{"metadata":{"finalizers":[]}}' --type=merge

CAUSE Turns out we had a ton of leftover/old certificate requests that had the following in them as found with kubectl describe certificaterequests -A (Many that did not have “Approved” as True.):

 Owner References:
    API Version:           cert-manager.io/v1alpha2

And also kubectl describe orders -A resulted in the same amount of orders that had the same v1alpha2 in them.

RESOLUTION We resolved the issue by:

• Downgrade to cert-manager 1.5.5 (we use the Helm chart)
• Run cmctl upgrade migrate-api-version --qps 5 --burst 10
• Delete all the old certificaterequest and order records with kubectl delete certificaterequest -n NAMESPACE ID1 ID2 and kubectl delete order -n NAMESPACE ID1 ID2 (We deleted anything over 90 days or that had v1alpha2.)
• Install cert-manager 1.8.0
• Check logs

Ok so I ran into this recently upgrading this system helm chart. I can probably give someone some help here if they run into this… Its easier to use the cmctl than running the curl command back to the api to update the underlying crd’s. It looks like the cmctl upgrade migrate-api-version also upgrades the underlying objects along with the crd’s. Here’s a workplan I followed(I use reckoner currently which runs helm3 under the hood).

Cert-manager upgrade from 0.16 -> 1.5.5 -> 1.7.1. The reason to go from 0.16 to 1.5.5 is because the cert-manager.io/v1alpha2 was fully removed in 1.6 and the etcd data must reflect the newer api version of cert-manager.io/v1 before you can then upgrade to 1.6 or higher.

Upgrade step 0.16 -> 1.5.5

Before upgrading in a stable environment renew the vault certificate first so if something unexpected happens and breaks we have time to fix it without certificates expiring… Check the verifications steps section for renewal steps.

BEFORE running this, ensure you backup these two objects below at the very least. https://cert-manager.io/docs/tutorials/backup/

> kc get clusterissuers <issuers> -o yaml > clusterissuers-backup.yaml
 > kc get certificates <cert> -o yaml -n blah > cert-backup.yaml

Change the apiVersion in the clusters/*/crds files from cert-manager.io/v1alpha2 to cert-manager.io/v1 Change the versions in the reckoner.yaml to 1.5.5 then:

$ reckoner plot --dry-run reckoner.yaml --heading cert-manager

➜ reckoner plot reckoner.yaml --heading cert-manager

Verification steps

Download cmctl:

OS=$(go env GOOS); ARCH=$(go env GOARCH); curl -sSL -o cmctl.tar.gz https://github.com/cert-manager/cert-manager/releases/download/v1.7.1/cmctl-$OS-$ARCH.tar.gz
tar xzf cmctl.tar.gz
sudo mv cmctl /usr/local/bin

1. Ensure that all the certificate-manager pods within the cert-manager namespace are up and running correctly. Check the logs for further verification.
2. Use cmctl to renew certificate manually.

➜ kubectl get certificate -n hashicorp

NAME                             READY   SECRET                           AGE
vault-service-cluster-zone-tls   True    vault-service-cluster-zone-tls   686d

➜ cmctl renew vault-service-cluster-zone-tls -n hashicorp
                                                 
➜ kubectl get certificaterequest -n hashicorp
NAME                              READY   AGE
vault-service-cluster-zone-tls         False    10s

Upgrade step 1.5.5 -> 1.7.1

Before running reckoner plot you must upgrade the apiVersion in the status.storedVersions.

BEFORE running this, ensure you backup these two objects below at the very least. https://cert-manager.io/docs/tutorials/backup/

> kc get clusterissuers <issuers> -o yaml > clusterissuers-backup.yaml
 > kc get certificates <cert> -o yaml -n blah > cert-backup.yaml

On your local machine run the below command to upgrade the apiVersions stored:

cmctl upgrade migrate-api-version --qps 5 --burst 10

A successful run of the migrate command looks like:

cmctl upgrade migrate-api-version --qps 5 --burst 10
Checking all CustomResourceDefinitions have storage version set to "v1"
All CustomResourceDefinitions have "v1" configured as the storage version.
Looking for CRDs that contain resources that require migrating to "v1"...
Found 6 resource types that require migration:
 - certificates.cert-manager.io (Certificate)
 - certificaterequests.cert-manager.io (CertificateRequest)
 - issuers.cert-manager.io (Issuer)
 - clusterissuers.cert-manager.io (ClusterIssuer)
 - orders.acme.cert-manager.io (Order)
 - challenges.acme.cert-manager.io (Challenge)
Migrating "Certificate" objects in group "cert-manager.io" - this may take a while (started at 21:36:14)...
 2 resources to migrate...
 Successfully migrated 2 Certificate objects in 1s
Migrating "CertificateRequest" objects in group "cert-manager.io" - this may take a while (started at 21:36:15)...
 8 resources to migrate...
 Successfully migrated 8 CertificateRequest objects in 4s
Migrating "Issuer" objects in group "cert-manager.io" - this may take a while (started at 21:36:18)...
 1 resources to migrate...
 Successfully migrated 1 Issuer objects in 1s
Migrating "ClusterIssuer" objects in group "cert-manager.io" - this may take a while (started at 21:36:18)...
 1 resources to migrate...
 Successfully migrated 1 ClusterIssuer objects in 1s
Migrating "Order" objects in group "acme.cert-manager.io" - this may take a while (started at 21:36:19)...
 7 resources to migrate...
 Successfully migrated 7 Order objects in 2s
Migrating "Challenge" objects in group "acme.cert-manager.io" - this may take a while (started at 21:36:20)...
 0 resources to migrate...
 Successfully migrated 0 Challenge objects in 1s
Patching CRD resources to set "status.storedVersions" to "v1"...
Successfully migrated all cert-manager resource types. It is now safe to upgrade to cert-manager v1.7.

Please check to see if you get this type of error when you do a kubectl get orders or kubectl get certificaterequests. Search for whatever k8s objects you would need to access for your functional use of cert-manager.

Error from server: request to convert CR from an invalid group/version: acme.cert-manager.io/v1alpha2

We may need to run what we call a kubectl patch shown below:

kubectl patch crd/certificaterequests.cert-manager.io -p '{"metadata":{"finalizers":[]}}' --type=merge
kubectl patch crd/orders.acme.cert-manager.io -p '{"metadata":{"finalizers":[]}}' --type=merge

Now run:

$ reckoner plot reckoner.yaml --heading cert-manager

PLEASE go back through the Verification Step Section!!!