helmfile: Cannot install prometheus-operator chart with helmfile due to helm-diff error

Versions used:

helmfile v0.100.2
helm v3.1.1
databus23/helm-diff plugin v3.1.1

Problem observed:

I can successfully install the stable/prometheus-operator chart with helm directly.

$ helm install prometheus-operator stable/prometheus-operator -n monitoring
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
NAME: prometheus-operator
LAST DEPLOYED: Wed Feb 26 13:13:58 2020
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
NOTES:
The Prometheus Operator has been installed. Check its status by running:
  kubectl --namespace monitoring get pods -l "release=prometheus-operator"

Visit https://github.com/coreos/prometheus-operator for instructions on how
to create & configure Alertmanager and Prometheus instances using the Operator.

I cannot install the same chart with helmfile:

$ helmfile apply
Comparing release=prometheus-operator, chart=stable/prometheus-operator
in ./helmfile.yaml: in .helmfiles[0]: in helmfiles/prometheus-operator.yaml: failed processing release prometheus-operator: helm exited with status 1:
  Error: Failed to render chart: exit status 1: manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
  manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
  manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
  manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
  manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
  Error: unable to build kubernetes objects from release manifest: [unable to recognize "": no matches for kind "Alertmanager" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "Prometheus" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"]

  Error: plugin "diff" exited with error

I see that the error is coming out of the helm-diff plugin. What command(s) is helmfile using that causes these errors to surface? I will open an issue in the databus23/helm-diff plugin repo once I understand how to repro this with the helm diff command directly.

Many thanks in advance!

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 19
Comments: 26 (5 by maintainers)

Commits related to this issue

Add disableValidation and disableOpenAPIValidation per release `disableOpenAPIValidation: true` might be useful for workaround for broken CRDs that is known to be exist in older OpenShift versions, a... — committed to roboll/helmfile by mumoshu 4 years ago
Add disableValidation and disableOpenAPIValidation per release (#1373) `disableOpenAPIValidation: true` might be useful for workaround for broken CRDs that is known to be exist in older OpenShift ver... — committed to roboll/helmfile by mumoshu 4 years ago

Most upvoted comments

So after #1373 this should work

releases:
- name: prometheus-operator
  chart: stable/prometheus-operator
  disableValidation: true

+14

mumoshu on Jul 22, 2020

I would rather say a helm 3 issue. Since doing helm install --dry-run don’t work for charts with crds, see https://github.com/helm/helm/issues/7449 And in the end that’s why this fails as well, you can’t verify objects that aren’t yet defined in the API-server.

In my opinion the best solution would be the one suggested in that issue, but it doesn’t seem to be a priority atm. So another solution would be to have a way to disable diff in helmfile when the release aren’t present (when installing), like a --skip-diff-if-installing flag when doing apply or something.

Xartos on Apr 24, 2020

FWIW using helmfile sync works, but using sync skips the diff. Right now I’ve been using sync on my first install and diff/apply after that.

bradenwright on Jul 1, 2020

As a workaround I installed helm-diff 3.0.0-rc.7

semoac on May 19, 2020

We currently using the following workaround:

...
    hooks:
    # Create CRDs separately in helmfile presync hooks
    # https://github.com/roboll/helmfile/issues/1124
    # https://github.com/helm/helm/issues/7449
    # https://github.com/cloudposse/helmfiles/blob/59490fd2599d6113a14103be919985f9fbcea73a/releases/prometheus-operator.yaml
    # Hooks associated to presync events are triggered before each release is applied to the remote cluster.
    # This is the ideal event to execute any commands that may mutate the cluster state as it
    # will not be run for read-only operations like lint, diff or template.
    # These hook install the prometheuses.monitoring.coreos.com CustomResourceDefinition if needed
    - events: ["presync"]
      command: "/bin/sh"
      args: ["-c", "kubectl get crd prometheuses.monitoring.coreos.com >/dev/null 2>&1 || \
             { helm pull stable/prometheus-operator --version {{`{{ .Release.Version }}`}} && tar -Oxzf prometheus-operator-{{`{{ .Release.Version }}`}}.tgz prometheus-operator/crds/crd-prometheus.yaml | kubectl apply -f -; }"]
    # This hoook installs the alertmanagers.monitoring.coreos.com CustomResourceDefinition if needed
    - events: ["presync"]
      command: "/bin/sh"
      args: ["-c", "kubectl get crd alertmanagers.monitoring.coreos.com >/dev/null 2>&1 || \
             { helm pull stable/prometheus-operator --version {{`{{ .Release.Version }}`}} && tar -Oxzf prometheus-operator-{{`{{ .Release.Version }}`}}.tgz prometheus-operator/crds/crd-alertmanager.yaml | kubectl apply -f -; }"]
    # This hoook installs the prometheusrules.monitoring.coreos.com CustomResourceDefinition if needed
    - events: ["presync"]
      command: "/bin/sh"
      args: ["-c", "kubectl get crd prometheusrules.monitoring.coreos.com >/dev/null 2>&1 || \
             { helm pull stable/prometheus-operator --version {{`{{ .Release.Version }}`}} && tar -Oxzf prometheus-operator-{{`{{ .Release.Version }}`}}.tgz prometheus-operator/crds/crd-prometheusrules.yaml | kubectl apply -f -; }"]
    # This hoook installs the servicemonitors.monitoring.coreos.com CustomResourceDefinition if needed
    - events: ["presync"]
      command: "/bin/sh"
      args: ["-c", "kubectl get crd servicemonitors.monitoring.coreos.com >/dev/null 2>&1 || \
             { helm pull stable/prometheus-operator --version {{`{{ .Release.Version }}`}} && tar -Oxzf prometheus-operator-{{`{{ .Release.Version }}`}}.tgz prometheus-operator/crds/crd-servicemonitor.yaml | kubectl apply -f -; }"]
    # This hoook installs the podmonitors.monitoring.coreos.com CustomResourceDefinition if needed
    - events: ["presync"]
      command: "/bin/sh"
      args: ["-c", "kubectl get crd podmonitors.monitoring.coreos.com >/dev/null 2>&1 || \
             { helm pull stable/prometheus-operator --version {{`{{ .Release.Version }}`}} && tar -Oxzf prometheus-operator-{{`{{ .Release.Version }}`}}.tgz prometheus-operator/crds/crd-podmonitor.yaml | kubectl apply -f -; }"]
    # This hoook installs the thanosrulers.monitoring.coreos.com CustomResourceDefinition if needed
    - events: ["presync"]
      command: "/bin/sh"
      args: ["-c", "kubectl get crd thanosrulers.monitoring.coreos.com >/dev/null 2>&1 || \
             { helm pull stable/prometheus-operator --version {{`{{ .Release.Version }}`}} && tar -Oxzf prometheus-operator-{{`{{ .Release.Version }}`}}.tgz prometheus-operator/crds/crd-thanosrulers.yaml | kubectl apply -f -; }"]
...

It’s similar to the one posted above by @agmtr but uses the CRDs directly from the chart. This increases compatibility because it uses the same version which is normally bundled with the chart - just deploys it with a presync hook.

baurmatt on May 11, 2020

I thought helm-diff recently added --disable-openapi-validation that can be used for disabling validation at all which will also makes this issue disappear.

Can we probably enhance Helmfile to add a new option under releases[] to enable it? e.g.

releases:
- name: prometheus-operator
  chart: stable/prometheus-operator
  # Set this to true if your chart contains crds or install-crd hooks.
  disableOpenAPIValidation: true

mumoshu on Jul 20, 2020

Isn’t this chart issue, rather than helm or helmfile’s?

I mean, are those charts migrated to internally use the new crds directory added in Helm 3?

mumoshu on Apr 24, 2020

With helmfile (v0.104.1), helm-diff (v3.1.1), and helm (v3.1.2) I’m no longer having this issue.

On Tue, Mar 24, 2020 at 11:39 AM agmtr notifications@github.com wrote:

I’am use hooks to workaround

events: [“prepare”] command: “/bin/sh” args: [“-c”, “kubectl get crd prometheuses.monitoring.coreos.com >/dev/null 2>&1 ||
kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml”]

events: [“prepare”] command: “/bin/sh” args: [“-c”, “kubectl get crd alertmanagers.monitoring.coreos.com >/dev/null 2>&1 ||
kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml”]

events: [“prepare”] command: “/bin/sh” args: [“-c”, “kubectl get crd prometheusrules.monitoring.coreos.com >/dev/null 2>&1 ||
kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml”]

events: [“prepare”] command: “/bin/sh” args: [“-c”, “kubectl get crd servicemonitors.monitoring.coreos.com >/dev/null 2>&1 ||
kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml”]

events: [“prepare”] command: “/bin/sh” args: [“-c”, “kubectl get crd podmonitors.monitoring.coreos.com >/dev/null 2>&1 ||
kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml”]

events: [“prepare”] command: “/bin/sh” args: [“-c”, “kubectl get crd podmonitors.monitoring.coreos.com >/dev/null 2>&1 ||
kubectl apply --validate=false -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml”] set:

name: prometheusOperator.createCustomResource value: false```

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/roboll/helmfile/issues/1124#issuecomment-603354478, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA2PMIESYDOZ5SLQBIVS6LRJDO2LANCNFSM4K4MVLUA .

mojochao on Mar 25, 2020