helm: [Helm 3] improper RBAC setup causes release to be stuck in `pending-install` status despite installing fine

This appears to have been role issues. Once I granted the update verb for secrets to my service account, the release was displayed properly.

Could the error handling be made better? Helm reported no errors and I had no way to know that my RBAC roles were lacking

@bacongobbler, sorry for mentioning you on an old issue. We have been experiencing this problem a lot lately and I was trying to come up with a way Helm can identify it and fix itself. I was thinking about something along the lines:

1. Check if the last release is in a pending-install state
2. If it is, get the resource list from that release and check whether the resources were created (something similar with the waitForResources (https://github.com/helm/helm/blob/master/pkg/kube/wait.go)
3. If everything was created successfully, then mark the release as deployed; otherwise, return a meaningful error

Would a PR with such solution be welcome?

I faced this issue, lack of update verb on secrets, so I ended with a bunch of installed helm charts that are working but that cannot be upgraded with Helm. I tried to manually update the label to “deployed”, but helm still believes they are in “pending-install”, I guess there is some other bookkeeping necessary in order for helm to “understand” they are in deployed status. Does anybody know what steps to take so that I can fix these deployments? (deleting them and re-installing them is not really and option for me…).

You could try helm rollback <release-name> <revision> which the revision is the latest revision (in my case is 1). I fixed this issue with that,

I could workaround it by doing this:

helm uninstall <my-release> -n <my-release-namespace>

We faced an issue with a release getting stuck in pending-rollback while all resources were finally stable. This happened since the client was not running any more on our CD system. Here is how I fixed it without uninstalling the release: https://gist.github.com/siavashs/7e3715879c0e946167749fa9a358380a

For me, a release ended up with pending-install status because an installation through Terraform had an error and aborted.

What I did to fix it (but be sure what you’re doing!) was to manually edit the secret sh.helm.release... from status: pending-install to status: deployed. Then I did a re-install/upgrade to try and get it into a known good state.

@johnhofman @bacongobbler You booth can easily simulate this problem by doing the following:

1 - Try to install any chart, using the --wait flag and the default timeout, with pod resource configuration above the limits of the node to cause the pod to not be scheduled by out of resources (CPU/MEM). 2 - Before helm cmd returns an error to you, just hit Ctrl+C 3 - Now, execute helm list -n <release-namespace> --all

My question now is: is there a workaround to fix this situation? Any ideas @bacongobbler ?

Im facing this issue with nginx-ingress chart. All other charts seem to deploy fine. The pods are launched and showin as healthy.

❯ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-26T06:17:09Z", GoVersion:"go1.14", Compiler:"gc", Platform:"darwin/amd64"}

Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.8-eks-e16311", GitCommit:"e163110a04dcb2f39c3325af96d019b4925419eb", GitTreeState:"clean", BuildDate:"2020-03-27T22:37:12Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

❯ helm version
version.BuildInfo{Version:"v3.2.4", GitCommit:"0ad800ef43d3b826f31a5ad8dfbb4fe05d143688", GitTreeState:"dirty", GoVersion:"go1.14.3"}

@artazar , Did you install nginx-ingress on k8s cluster that was running locally? And how did you create the cluster - using kubeadm or some other tool ? Reason I ask, because if someone wants to create the k8s cluster from scratch using kubeadm utility On-premise and expose the ingress externally, then what needs to be done ?

we’re also faces the issue we deployed rook and we got pending-install. we can’t delete the rook and re deploy it, because it means destroying all the kubernetes cluster

Facing the same issue. Cluster installed with kubeadm 1.18.2 and using helm 3.2.0

$ helm list -a
NAME	NAMESPACE	REVISION	UPDATED                                	STATUS         	CHART     	APP VERSION
test	default  	1       	2020-05-04 20:40:09.827303259 +0000 UTC	pending-install	kong-1.1.0	2.0.4

EDIT: I’ve updated the role to add the update verb for secrets to my service account but it had no impact:

$ helm install test bitnami/kong  --debug
install.go:159: [debug] Original chart version: ""
install.go:176: [debug] CHART PATH: /home/gtirloni/.cache/helm/repository/kong-1.1.0.tgz

client.go:108: [debug] creating 6 resource(s)
install.go:140: [debug] CRD kongconsumers.configuration.konghq.com is already present. Skipping.
install.go:153: [debug] Clearing discovery cache
wait.go:53: [debug] beginning wait for 0 resources with timeout of 1m0s
client.go:108: [debug] creating 13 resource(s)
client.go:258: [debug] Starting delete for "test-kong-migrate" Job
client.go:108: [debug] creating 1 resource(s)
client.go:467: [debug] Watching for changes to Job test-kong-migrate with timeout of 5m0s
client.go:495: [debug] Add/Modify event for test-kong-migrate: ADDED
client.go:534: [debug] test-kong-migrate: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:495: [debug] Add/Modify event for test-kong-migrate: MODIFIED
client.go:534: [debug] test-kong-migrate: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
Error: failed post-install: timed out waiting for the condition
helm.go:84: [debug] failed post-install: timed out waiting for the condition