helm: crd-install hook is not working on upgrade #4259
This is follow up for: #4259
Upgrade of cert-manager chart from version 0.4.1 to 0.5.0 added crd-install hook to CRD resources https://github.com/helm/charts/commit/e0505bbd09c4f8bf81fe583841c926e02cc82a8e
Unfortunately running helm upgrade on this chart actually removes all crd and not upgrading them.
Output of helm version:
Client: &version.Version{SemVer:"v2.10.0", GitCommit:"9ad53aac42165a5fadc6c87be0dea6b115f93090", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.10.0", GitCommit:"9ad53aac42165a5fadc6c87be0dea6b115f93090", GitTreeState:"clean"}
Output of kubectl version:
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.6", GitCommit:"a21fdbd78dde8f5447f5f6c331f7eb6f80bd684e", GitTreeState:"clean", BuildDate:"2018-07-26T10:04:08Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Cloud Provider/Platform (AKS, GKE, Minikube etc.): Kops
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 17
- Comments: 22 (5 by maintainers)
Commits related to this issue
- This is a problem that Helm is going to solve going forward, but for now if you use the crd-install hook, then you can only install CRDs, and not update them at any point during a chart lifecycle. Al... — committed to markmandel/agones by markmandel 6 years ago
- This is a problem that Helm is going to solve going forward, but for now if you use the crd-install hook, then you can only install CRDs, and not update them at any point during a chart lifecycle. Al... — committed to markmandel/agones by markmandel 6 years ago
- Remove crd-install hook, as it break CRD updates This is a problem that Helm is going to solve going forward, but for now if you use the crd-install hook, then you can only install CRDs, and not upda... — committed to markmandel/agones by markmandel 6 years ago
- Remove crd-install hook, as it break CRD updates This is a problem that Helm is going to solve going forward, but for now if you use the crd-install hook, then you can only install CRDs, and not upda... — committed to googleforgames/agones by markmandel 6 years ago
Hey, I think it’s important that whatever solution is found for this problem that the upmost is done to ensure an upgrade path for people who have either applied crd’s manually, or used the install hook.
They can’t go deleting these CRDs to get helm to track them because then they’ll loose the implementations too.
The only approach I could see would be to have helm do a “create if missing, track if exists” style operation, that way it wouldn’t require the pre-delete of the manifest.
So I think there are multiple issues here.
First, the crd-install hook only runs on install and it does a create op without checking if the crd already exists. This is the same behavior as for other hooks. But this will cause the create operation to fail if the crd already exist. It might be possible to work around this by using the before-hook-creation deletion policy, but that doesn’t sound like a good approach since it would delete all resources based on the crd.
This also means that crds are not created (or updated) as part of updating a release, so changing an existing chart to use the crd-install hook will probably not work as expected.
Both of these issues seems to be pretty well covered in the PR that introduced the crd-install hook #3982
Also, transitioning a chart that is already installing crds in the normal way (without hooks) to use the crd-install hook has some pitfalls. If crds are created in the normal way, they will be managed by helm as part of the release, but with the crd-install hook that is no longer the case. So as part of an upgrade where the chart switches to create crds with the crd-install hook, the existing crds in the release will be deleted by helm. And since it is an upgrade, helm will not execute the crd-install hook (an even if it did, this would happen before the delete step).
For any use case where helm does not create custom resources as part of the chart, the best option is probably to install crds with regular templates instead of using the crd-install hook. The hook would only be required where the chart need tthe crd to be installed before validation and creation of other resources.
Istio also impacted by this issue - and it is critical/blocking Istio 1.1 and 1.0.4. See https://github.com/istio/istio/issues/9604#issuecomment-437291974.
From my understanding/observation, this is not a support issue and is in fact a bug. Labelling as such 😄
/kind bug
/cc @technosophos @bacongobbler
EDIT: no prow bot automation 😢
related discussion about CRDs and how they are managed in Helm: #5871
Upgrades should obviously ignore CRD’s
yo! still broken? so broken 😦. When release?
Is the fix for this issue underway? Any targets. …thanks