rancher: Rancher not compatible with cert-manager v1.6.0 / Removed API beeing used

Rancher Server Setup

Rancher version: 2.6.2
Installation option (Docker install/Helm Chart): Helm-Chart / RKE1

Information about the Cluster

Cluster Type (Local/Downstream): Local

Describe the bug

To Reproduce

Upgrade cert-manager to 1.6.0 (or install 1.6.0)
Install or upgrade Rancher

Result

Error: UPGRADE FAILED: current release manifest contains removed kubernetes api(s) for this kubernetes version and it is therefore unable to build the kubernetes objects for performing the diff. error from kubernetes: unable to recognize "": no matches for kind "Issuer" in version "cert-manager.io/v1beta1"

Expected Result Rancher to be upgraded.

Additional context Rootcause is described here: https://github.com/jetstack/cert-manager/releases/tag/v1.6.0

Following their deprecation in version 1.5, the cert-manager APIVersions v1alpha2, v1alpha3, and v1beta1 are no longer served. This means if your deployment manifests contain any of these API versions, you will not be able to deploy them after upgrading.

Looks like Rancher still using the old API.

About this issue

Original URL
State: open
Created 3 years ago
Reactions: 13
Comments: 35 (7 by maintainers)

Most upvoted comments

Solved with helm-mapkubeapis and this config file (saved as certmanager.map.yaml)

# Run with helm mapkubeapis --mapfile certmanager.map.yaml rancher 
mappings:
 - deprecatedAPI: "apiVersion: cert-manager.io/v1beta1\nkind: Issuer"
   newAPI: "apiVersion: cert-manager.io/v1\nkind: Issuer"
   deprecatedInVersion: "v1.9"
   removedInVersion: "v1.16"

After running it (please use dry-run to ensure that output is that you want) I was able to upgrade rancher

$ helm mapkubeapis --mapfile certmanager.map.yaml   rancher
2022/05/13 15:24:57 Release 'rancher' will be checked for deprecated or removed Kubernetes APIs and will be updated if necessary to supported API versions.
2022/05/13 15:24:57 Get release 'rancher' latest version.
2022/05/13 15:24:58 Check release 'rancher' for deprecated or removed APIs...
2022/05/13 15:24:58 Found deprecated or removed Kubernetes API:
"apiVersion: cert-manager.io/v1beta1
kind: Issuer"
Supported API equivalent:
"apiVersion: cert-manager.io/v1
kind: Issuer"
2022/05/13 15:24:58 Finished checking release 'rancher' for deprecated or removed APIs.
2022/05/13 15:24:58 Deprecated or removed APIs exist, updating release: rancher.
2022/05/13 15:24:58 Set status of release version 'rancher.v12' to 'superseded'.
2022/05/13 15:24:58 Release version 'rancher.v12' updated successfully.
2022/05/13 15:24:58 Add release version 'rancher.v13' with updated supported APIs.
2022/05/13 15:24:58 Release version 'rancher.v13' added successfully.
2022/05/13 15:24:58 Release 'rancher' with deprecated or removed APIs updated successfully to new version.
2022/05/13 15:24:58 Map of release 'rancher' deprecated or removed APIs to supported versions, completed successfully.

fabn on May 13, 2022

@all4innov Solution: https://exploit.cz/rancher-upgrade-failed-cert-manager-io-v1beta1/

insekticid on Apr 4, 2022

Can confirm, cannot update from Rancher 2.6.0 to Rancher 2.6.3. Downgrading cert-manager to 1.5.3

appcoders on Dec 27, 2021

Solved with helm-mapkubeapis and this config file (saved as certmanager.map.yaml)

# Run with helm mapkubeapis --mapfile certmanager.map.yaml rancher 
mappings:
 - deprecatedAPI: "apiVersion: cert-manager.io/v1beta1\nkind: Issuer"
   newAPI: "apiVersion: cert-manager.io/v1\nkind: Issuer"
   deprecatedInVersion: "v1.9"
   removedInVersion: "v1.16"

After running it (please use dry-run to ensure that output is that you want) I was able to upgrade rancher

$ helm mapkubeapis --mapfile certmanager.map.yaml   rancher
2022/05/13 15:24:57 Release 'rancher' will be checked for deprecated or removed Kubernetes APIs and will be updated if necessary to supported API versions.
2022/05/13 15:24:57 Get release 'rancher' latest version.
2022/05/13 15:24:58 Check release 'rancher' for deprecated or removed APIs...
2022/05/13 15:24:58 Found deprecated or removed Kubernetes API:
"apiVersion: cert-manager.io/v1beta1
kind: Issuer"
Supported API equivalent:
"apiVersion: cert-manager.io/v1
kind: Issuer"
2022/05/13 15:24:58 Finished checking release 'rancher' for deprecated or removed APIs.
2022/05/13 15:24:58 Deprecated or removed APIs exist, updating release: rancher.
2022/05/13 15:24:58 Set status of release version 'rancher.v12' to 'superseded'.
2022/05/13 15:24:58 Release version 'rancher.v12' updated successfully.
2022/05/13 15:24:58 Add release version 'rancher.v13' with updated supported APIs.
2022/05/13 15:24:58 Release version 'rancher.v13' added successfully.
2022/05/13 15:24:58 Release 'rancher' with deprecated or removed APIs updated successfully to new version.
2022/05/13 15:24:58 Map of release 'rancher' deprecated or removed APIs to supported versions, completed successfully.

worked for me with (helm mapkubeapis --mapfile certmanager.map.yaml rancher -n cattle-system) upgrading 2.7.5 to 2.7.6

Privatecoder on Sep 1, 2023

Helm chart still use the correct apiVersion only as a last resort… (The release notes could be a lot simpler if #36611 is merged)

(And spamming everyone with notifications is necessary to keep the bot overlords happy)

mohag on Sep 9, 2022

Script updated to include pre-flight check, with a link to your comment in case of failure ! https://gist.github.com/AlexisDucastel/6b5e5cf79c0cd556056408934ff3029b

Thanks a lot @gbarceloPIB 😄

AlexisDucastel on Jul 4, 2022

Hi everyone, here is a simple gist with a script to help you patch your rancher release to fix cert-manager issue. https://gist.github.com/AlexisDucastel/6b5e5cf79c0cd556056408934ff3029b

It does automatically detect your rancher revision, backup the release secret in a temporary file and patch the secret if needed (so it is safe to start script multiple times).

For Mac users, be careful, OSX sed (FreeBSD) is not compatible with GNU sed. I learnt this the hard way. You may rollback with command: cat $TMP_FILE | kubectl -n cattle-system apply --force -f -

ghost on Jul 4, 2022

Hi everyone, here is a simple gist with a script to help you patch your rancher release to fix cert-manager issue. https://gist.github.com/AlexisDucastel/6b5e5cf79c0cd556056408934ff3029b

It does automatically detect your rancher revision, backup the release secret in a temporary file and patch the secret if needed (so it is safe to start script multiple times).

AlexisDucastel on Jul 4, 2022

@throrin19 you have to use correct sed replace. I added alternative sed replace to the article if you use older version than v1beta1

insekticid on Apr 5, 2022

The 2.5.11 to 2.5.12 upgrade is also broken after upgrading cert-manager to 1.6.1 (and following their guide to update the stored versions to v1)

Downgrading cert-manager to v1.5.5 seems to fix that…

(The v1 APIs have been present since cert-manager 1.0 AFAIK) (none of our own stuff ever used them, after going from the alpha APIs on 0.9.1 to v1 on ~1.2) (Rancher seems to have used the intermediate versions with the beta APIs without updating that)

mohag on Apr 4, 2022

The same problem. Rancher 2.6 doesn’t start with cert-manager 1.6.1. I rolled back to 1.5.0.

iboobel on Dec 4, 2021

Still an issue - the older cert-manager versions are getting scarce though…

mohag on Nov 1, 2023

Same issue still there… Chart still prefers the older APIs…

mohag on Nov 10, 2022

@all4innov Solution: https://exploit.cz/rancher-upgrade-failed-cert-manager-io-v1beta1/

@insekticid This solution helped me. Thanks a lot! My last sh.helm.release.v1.rancher.v* was v10 😃

daemonadmin on Jul 8, 2022

(a bit off-topic) I have all of them. But Catalina’s sed dates back from 2005’s OpenBSD! (no sed -r, out of the box at least). You may check if you do: sed --version If you have output, you may have GNU sed, otherwise check man version:

sed_version="$(sed --version 2>&1 | grep -q GNU && sed --version | sed '1!d;s/.* //' || man sed | sed '$!d; s/ *BSD *//g')"
echo $sed_version

ghost on Jul 4, 2022

Just a side note:

I have done a fresh install of Cert Manager 1.6.0 + Rancher 2.6.2 and it worked without errors.

From the Cert Manager note I understand that you can not upgrade, since by default you have the alphas and betas deployed and 1.6.0 uses v1.

For Cert Manager 1.5.x + Rancher 2.6.0 I had to modify the Rancher template files manually to contain v1 instead of the v1beta1. Otherwise the Rancher deployment/installation failed.

peteingithub on Oct 29, 2021