helm: Release "prometheus-operator" failed: rpc error: code = Canceled
Describe the bug
When I try to install prometheus operator on AKS with helm install stable/prometheus-operator --name prometheus-operator -f prometheus-operator-values.yaml I am getting this error:
prometheus-operator" failed: rpc error: code = Canceled
I checked with history:
helm history prometheus-operator -o yaml
- chart: prometheus-operator-6.3.0
description: 'Release "prometheus-operator" failed: rpc error: code = Canceled desc
= grpc: the client connection is closing'
revision: 1
status: FAILED
updated: Tue Jul 30 12:36:52 2019
Chart [stable/prometheus-operator]
Additional Info I am using below configurations to deploy a chart:
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/alertmanager.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/prometheus.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/prometheusrule.crd.yaml
kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/servicemonitor.crd.yaml
In values file: createCustomResource is set to false,
Output of helm version:
Client: &version.Version{SemVer:“v2.14.3”, GitCommit:“0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085”, GitTreeState:“clean”}
Server: &version.Version{SemVer:“v2.14.3”, GitCommit:“0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085”, GitTreeState:“clean”}
Output of kubectl version:
Client Version: version.Info{Major:“1”, Minor:“10”, GitVersion:“v1.10.4”, GitCommit:“5ca598b4ba5abb89bb773071ce452e33fb66339d”, GitTreeState:“clean”, BuildDate:“2018-06-06T08:13:03Z”, GoVersion:“go1.9.3”, Compiler:“gc”, Platform:“windows/amd64”}
Server Version: version.Info{Major:“1”, Minor:“13”, GitVersion:“v1.13.7”, GitCommit:“4683545293d792934a7a7e12f2cc47d20b2dd01b”, GitTreeState:“clean”, BuildDate:“2019-06-06T01:39:30Z”, GoVersion:“go1.11.5”, Compiler:“gc”, Platform:“linux/amd64”}
Cloud Provider/Platform (AKS, GKE, Minikube etc.): AKS
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 12
- Comments: 71 (9 by maintainers)
I was able to get around this issue by following the ‘Helm fails to create CRDs’ section in readme.md. I’m not sure how they’re related, but it worked.
Step 1: Manually create the CRDS
Step 2: Wait for CRDs to be created, which should only take a few seconds
Step 3: Install the chart, but disable the CRD provisioning by setting prometheusOperator.createCustomResource=false
I just now experienced the same issue with the CRDs currently on master. Thanks @Typositoire for your suggestion to use the currently previous version. Adapting CRDs install to the following worked for me:
That’s why fixing the version is often a good practice.
I was fighting the same issue, I had to manually install the crds specified by @JBosom and install with the web hook disabled.
helm --tls --tiller-namespace=tiller install --namespace=monitoring --name prom-mfcloud stable/prometheus-operator --set prometheusOperator.createCustomResource=false --set prometheusOperator.admissionWebhooks.enabled=false --values values.yaml --versi on 6.18.0Our workaround is to keep prometheus operator image on v0.31.1.
We use prometheus-operator in our deployment, in a nutshell, we upgraded prom-op from 6.9.3 to 8.3.3 and always failed with “Error: context canceled”. Also we always install crds before install/upgrade prometheus-operator, and ofc we didn’t change or update these crd-s.
I try to refresh crds, which in ‘github.com/helm/charts/tree/master/stable/prometheus-operator’ mentions ( like this kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/alertmanager.crd.yaml ), but these don’t exists anymore. After that I try to these from here: https://github.com/helm/charts/tree/master/stable/prometheus-operator/crds But It failed again.
I almost gave up, but with these crds, helm deploy succeeded ! yeyyyy https://github.com/coreos/kube-prometheus/tree/master/manifests/setup
My setup:
Kubernetes version:
v1.14.3Kubectl version:v1.14.2Helm version:2.14.3Prometheus-operator version:8.3.3Purge prometheus-operator from k8s !
Then:
That’s all !
Manual CRDs creation helps at least on Azure. Firstly create crds from this link https://github.com/coreos/prometheus-operator/tree/release-0.34/example/prometheus-operator-crd “kubectl create -f alertmanager.crd.yaml” and so on for all files Then helm install prometheus-operator stable/prometheus-operator --namespace monitoring --version 8.2.4 --set prometheusOperator.createCustomResource=false
Saw there was an update to promethus chart today. I bumped it to
and i’m no longer seeing the issue.
myenv: k8s 1.11.2 helm 2.13.1 tiller 2.13.1 prometheus-operator-5.5 APP VERSION 0.29 is OK!!!
but: prometheus-operator-8 APP VERSION 0.32 hava same problem: “context canceled” or “grpc: the client connection is closing”!!!
i guess the lastest version of prometheus-operator is not compatible?!!!
Yes. The original error appears to be a deadlock waiting for the admission web hook to complete, since disabling the web hook allows the chart to install without issue. Looking at Tiller’s logs should confirm the issue.
Helm 3 should report the correct error back to the user as there is no gRPC layer in the mix timing out and cancelling the request from a timeout.
Feel free to provide patches for Helm 2. Given that this has been improved for Helm 3, I went ahead and closed this as fixed in newer releases.
Hope this helps.
Try disabling the admissions controller web hook?
https://waynekhan.net/2019/10/09/prometheus-operator-release-failed.html
Same issue here trying to install on Azure AKS with kubernetes 1.13.10 and helm v2.14.3 with prometheus-operator-6.18.0. Any suggestion?
CRD installed manually.
This command failed:
helm install --name prometheus-operator stable/prometheus-operator --namespace=monitoring --set prometheusOperator.createCustomResource=falsegive the error
EDIT: installing the version 6.11.0 (as well as the 6.7.3) of the chart is working:
helm install --name prometheus-operator stable/prometheus-operator --namespace=monitoring --set prometheusOperator.createCustomResource=false --version 6.11.0Also had this issue, try to disable
admissionWebhooks. It helped in my case.So 4 days a part the workaround worked and stopped working I had to use the CRDs file from
0.32.0not master.@vsliouniaev Still same issue! Though the workaround from lethalwire works.