helm: helm (v2.15.) upgrade with --wait is waiting till timeout and not showing status even though all kubernetes resources are in ready state.

Output of helm version: Client: &version.Version{SemVer:“v2.15.0”, GitCommit:“c2440264ca6c078a06e088a838b0476d2fc14750”, GitTreeState:“clean”} Server: &version.Version{SemVer:“v2.15.0”, GitCommit:“c2440264ca6c078a06e088a838b0476d2fc14750”, GitTreeState:“clean”}

Output of kubectl version:

Client Version: version.Info{Major:“1”, Minor:“16”, GitVersion:“v1.16.2”, GitCommit:“c97fe5036ef3df2967d086711e6c0c405941e14b”, GitTreeState:“clean”, BuildDate:“2019-10-17T17:16:09Z”, GoVersion:“go1.12.10”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“14”, GitVersion:“v1.14.1”, GitCommit:“b7394102d6ef778017f2ca4046abbaa23b88c290”, GitTreeState:“clean”, BuildDate:“2019-04-08T17:02:58Z”, GoVersion:“go1.12.1”, Compiler:“gc”, Platform:“linux/amd64”}

Cloud Provider/Platform (AKS, GKE, Minikube etc.): On prem (Vmware)

I am trying to upgrade my chart. Everything is working fine in Kubernetes and all the Kubernetes resources is created and in ready state but helm is not showing status if I am using --wait.

This issue is coming only in helm v2.15.0 earlier version was working fine. The tiller pod is throwing following error.

[kube] 2019/10/22 23:21:38 Created a new Ingress called “dev-drupal” [kube] 2019/10/22 23:21:38 beginning wait for 13 resources with timeout of 10m0s [kube] 2019/10/22 23:21:40 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:21:42 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:21:44 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:21:46 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:21:48 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:21:50 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:21:52 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:21:54 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:21:56 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:21:58 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:00 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:02 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:04 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:06 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:08 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:10 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:12 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:14 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:16 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:18 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:20 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:22 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:24 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:26 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:28 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:30 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:32 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:34 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:36 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:38 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:40 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:42 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:44 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:46 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:48 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:50 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:52 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:54 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:56 Ingress is not ready: dev/drupal-ingress [kube] 2019/10/22 23:22:58 Ingress is not ready: dev/drupal-ingress

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 7
Comments: 21 (10 by maintainers)

Commits related to this issue

fix(wait): Removes ingress checks Closes #6751 After doing some more digging, I found out that updating the status of an `Ingress` object is completely optional. Because of this, Helm cannot support... — committed to thomastaylor312/helm by thomastaylor312 5 years ago

Most upvoted comments

Ok, I asked around in sig-network and got a response on Slack. Copying the message here for posterity sake:

no it’s not required. It would make it also complicated if you have one controller that creates the L4 LB and another controller that does the L7 routing. In some controllers you can have L4 and L7 in one, such that you have the optional flag.

So with that said, it looks like we need to remove the ingress checks from Helm. I’ll open up a PR soon

thomastaylor312 on Oct 25, 2019

I’m now facing this issue too. Follow the research of @thomastaylor312, I found that maybe Traefik 1.7 has already resolved this issue in containous/traefik#3324 by providing an optional config of not enabled by default. And the helm chart also has values named kubernetes.ingressEndpoint.* of this which requires configuring manually.

Also, the Nginx ingress has a doc about it published here.

Update: I’ve tried in my EKS(v1.14.7-eks-e9b1d0) + Traefik (Chart ver. 1.78.4):

# values.yaml

kubernetes:
  ingressEndpoint:
    useDefaultPublishedService: true

It works so far:

NAME            HOSTS                    ADDRESS                                                                   PORTS     AGE
example         example.com              abcd1231231231231231231231231231-1231231231.us-east-2.elb.amazonaws.com   80, 443   135d

$ kubectl get ing example -o json
...
    "status": {
        "loadBalancer": {
            "ingress": [
                {
                    "hostname": "..."
                }
            ]
        }
    }

wi1dcard on Oct 25, 2019

Same issue ! GKE + Traefik rolled-back in v2.14

mr-prud on Oct 24, 2019

@vfiset Unfortunately this is something you’d have to upgrade for. FWIW, 2.16 is basically the same, but we had to bump the gRPC version because of the Kubernetes library upgrades. Per our semver contract, it can’t be a patch release. What that means is that you should be able to switch out 2.15 for 2.16 without a big issue. Otherwise your only other option is to custom build a version by cherry picking the fix in. However, I would not recommend this

thomastaylor312 on Jan 29, 2020

status:
  loadBalancer: {}

describe gives me:

Name:             ***-frontend-ingress
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (<none>)
TLS:
  ***-dev-cert terminates some.domain.net
Rules:
  Host                   Path  Backends
  ----                   ----  --------
  some.domain.net  
                         /   ***-frontend-svc:80 (<none>)
Annotations:
  certmanager.k8s.io/cluster-issuer:              letsencrypt
  ingress.kubernetes.io/custom-response-headers:  server:
  kubernetes.io/ingress.class:                    traefik
Events:                                           <none>

This is helm 2.15.1 and an AKS cluster v1.12.6 Also using Traefik for my ingress controller

Elexy on Oct 23, 2019