longhorn: [BUG] With longhorn 1.4.0, k8s 1.25, and `enablePSP: false` in Helm chart, still getting `resource mapping not found for name: "longhorn-psp" namespace: "" from "": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1"`

Describe the bug (🐛 if you encounter this issue)

With the latest Longhorn 1.4.0 (bumped from 1.2.3), the latest Helm chart (with enablePSP: false set), doing a Helm Upgrade (via Flux) still does not work:

► resuming helmrelease longhorn in longhorn-system namespace
✔ helmrelease resumed
◎ waiting for HelmRelease reconciliation
✗ Helm upgrade failed: unable to build kubernetes objects from current release manifest: resource mapping not found for name: "longhorn-psp" namespace: "" from "": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1"
ensure CRDs are installed first

To Reproduce

Steps to reproduce the behavior:

Longorn running on 1.2.3
Update Flux manifests to 1.4.0
See above error

Expected behavior

It should install correctly. My cluster does not have PSP anymore, but even with enablePSP: false it’s complaining that it can’t find PSP?

Environment

Longhorn version: 1.2.3 -> 1.4.0
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Flux/HelmRelease
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: k3s
- Number of management node in the cluster: 1
- Number of worker node in the cluster: 1 (main node is also the worker node)
Node config
- OS type and version:Arch Linux
- CPU per node:1
- Memory per node:8gb
- Disk type(e.g. SSD/NVMe):hdd
- Network bandwidth between the nodes:n/a
Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): baremetal
Number of Longhorn volumes in the cluster:9

About this issue

Original URL
State: closed
Created a year ago
Comments: 17 (3 by maintainers)

Most upvoted comments

Hello @gravufo, I’m facing the same issue. Can you elaborate what exactly you deleted? Because in my mind secret is not equal configmap. If I understand correctly you didn’t change or remove the psp resource from the release secret?

Helm creates secrets or configmaps to store the releases you install, their settings and history. They are called something similar to “sh.helm.release.v1.longhorn.v1”. Simply delete this and reinstall fresh.

Thanks for your quick reply. What I ended up doing is using the helm plugin helm-mapkubeapis which maps deprecated or removed Kubernetes APIs in a release to supported APIs (https://github.com/helm/helm-mapkubeapis). In the end is was a single command and after that the helm upgrade was working again.

helm mapkubeapis longhorn --namespace longhorn-system

thomasherrle on Jun 23, 2023

Never mind, I decided to uninstall Longhorn instead

sseneca on Jan 1, 2023

@innobead So when we’ve upgraded to longhorn 1.4.0 from 1.2.x after upgrading my cluster to 1.25, then the cluster can no longer be salvaged?

You can have fun with this: https://helm.sh/docs/topics/kubernetes_apis/#updating-api-versions-of-a-release-manifest But instead of updating the API versions, you need to actually just remove the resources.

What I did on my side was just simply delete the helm secret/configmap to make helm think nothing was installed and simply installed the new version of longhorn on top of the existing resources.

It’s really a shame that k8s/helm can’t make this a little more elegant.

gravufo on Jan 18, 2023

@innobead So when we’ve upgraded to longhorn 1.4.0 from 1.2.x after upgrading my cluster to 1.25, then the cluster can no longer be salvaged?

You can have fun with this: https://helm.sh/docs/topics/kubernetes_apis/#updating-api-versions-of-a-release-manifest But instead of updating the API versions, you need to actually just remove the resources.

What I did on my side was just simply delete the helm secret/configmap to make helm think nothing was installed and simply installed the new version of longhorn on top of the existing resources.

It’s really a shame that k8s/helm can’t make this a little more elegant.

You saved my day!

No other way was working

luizjr on Feb 16, 2023

Thanks, @sseneca update. @yangchiu good ask.

The failure is caused by upgrading to Longhorn 1.4.0 from the crashed longhorn 1.2.x after you upgraded the K8s cluster to K8s 1.25, because the originally installed Longhorn PSP should be uninstalled first when doing the helm upgrade, but it’s can’t be found from the cluster which is 1.25 already.

Helm upgrade failed: unable to build kubernetes objects from current release manifest: resource mapping not found for name: "longhorn-psp" namespace: "" from "": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1"

The right flow is to upgrade to 1.4.0 before upgrading the K8s cluster to 1.25. It’s good to know everything is fine now.

ref: https://github.com/longhorn/longhorn/issues/4003#issuecomment-1350574289

cc @longhorn/qa

innobead on Jan 3, 2023

https://docs.k3s.io/upgrades/automated

Cluster was upgraded when k8s 1.25 was marked as stable a few months back. Longhorn pods were crashing as a result, waited to update to 1.4.0, didn’t work

fwiw reinstalling Longhorn from scratch worked. I installed Longhorn and it’s running on my cluster fine

sseneca on Jan 3, 2023

With k8s v1.24.7, I can upgrade Longhorn from v1.2.3 to v1.4.0 using helm without problems, all Longhorn components are running without errors after upgrade.

Steps: (1) helm install longhorn longhorn/longhorn --namespace longhorn-system --version 1.2.3 (2) In longhorn/longhorn repository, git checkout v1.4.0, and check enablePSP value => It’s disabled by default: enablePSP: false (3) Upgrade Longhorn to v1.4.0: helm upgrade longhorn ./longhorn/chart -n longhorn-system

But the user mentioned k8s v1.25 in the title. I’m not sure when did he upgrade the k8s version (since Longhorn v1.2.3 doesn’t support k8s v1.25). @sseneca Could you help to illustrate more about your (k8s) upgrade path?

yangchiu on Jan 3, 2023