helm: Cannot change timeout on API calls

My organisation’s openshift cluster has many CRDs and throttles the client connection (If I understand it correctly). Often when busy the throttling/performance is so bad that helm operations fail. I’d like to increase the timeout on the API calls. Which looks like a “–timeout” setting. However, if I try to change the timeout (to a value lower than the typical throttle delay) it still appears to have a 32s timeout… (And doesn’t fail due to the request taking too long.)

helm install --timeout 10s files -f ../files.yaml  chart
I0615 14:02:04.936726   33698 request.go:668] Waited for 1.148672262s due to client-side throttling, not priority and fairness, request: GET:https://api.server:443/apis/events.k8s.io/v1?timeout=32s
I0615 14:02:14.937525   33698 request.go:668] Waited for 11.14860773s due to client-side throttling, not priority and fairness, request: GET:https://api.server:443/apis/helm.openshift.io/v1beta1?timeout=32s
NAME: files
LAST DEPLOYED: Tue Jun 15 14:02:16 2021
....etc, notes from normal install...

Example of a fail looks the same as above, but after the last “waited for” I see :

Error: release files failed, and has been uninstalled due to atomic being set: timed out waiting for the condition

(I use --atomic normally now because of this problem!)

I would like to be able to increase the timeout from 32s to a higher value. I know the API server is overloaded and would rather helm wait a few more seconds for it than ME have to wait till 4AM to deploy my helm chart when nobody else is around…

Output of helm version:

version.BuildInfo{Version:"v3.6.0", GitCommit:"7f2df6467771a75f5646b7f12afb408590ed1755", GitTreeState:"dirty", GoVersion:"go1.16.4"}

Output of kubectl version: kubectl has been removed. There was a suggestion this issue was fixed in recent versions of the openshift client (oc)

$ oc version
Client Version: 4.7.0-202104250659.p0-95881af
Kubernetes Version: v1.20.0+7d0a2b2

Cloud Provider/Platform (AKS, GKE, Minikube etc.): Openshift

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 6
  • Comments: 25 (4 by maintainers)

Most upvoted comments

We are still facing problem on clusters having 100+ CRDs

Since it’s been a while since my suggestion and there’s been no further conversation about this, I’m going to go ahead and close it.

Still relevant

From the help for install : –timeout duration time to wait for any individual Kubernetes operation (like Jobs for hooks) (default 5m0s)

Is creating an object like a secret or a deployment or …whatever it is doing… not an “individual operation” ??? What is an individual Kubernetes operation?

Going by the documentation of --timeout, this is not a feature request. It is at least a bug with the documentation of what timeout actually means. But I’d prefer it if someone fixed the timeout rather than redocumenting it.

Yes, it is a warning, but sometimes if the cluster or network is slow it is an error. “Error: timed out waiting for the condition” And if it is slow to complete then the rollback operations can be slow too and sometimes exceed the 32s timeout and the rollback fails to complete successfully leaving a mess.

This is still a problem.

The solution is simple as 2*2. One needs to add new command-line argument like “–api-server-timeout” for helm and pass it’s value to client-go library.