istio: timed out waiting for the condition Deployment/istio-system/istiod

  • Installation

Expected behavior

Complete installation without errors

I tried to install Istio 1.5 on my local k8s cluster and encountered an error. By reading the code, it seems that it is caused by Deployment / istio-system / istiod not ready. Install Istio 1.5 on katacoda. This step only takes a few seconds, but on my local computer, He will keep Waiting for resources to become ready... until he prompts 2020-04-02T21:51:57.048098Z error installer Failed to wait for resource: resources not ready after 10m0s: timed out waiting for the condition Deployment/istio-system/istiod after 10 minutes.

Steps to reproduce the bug

install with the doc

[root@k1 istio-1.5.1]# export PATH=$PWD/bin:$PATH
[root@k1 istio-1.5.1]#
[root@k1 istio-1.5.1]# istioctl verify-install
Checking the cluster to make sure it is ready for Istio installation...

#1. Kubernetes-api
-----------------------
Can initialize the Kubernetes client.
Can query the Kubernetes API Server.

#2. Kubernetes-version
-----------------------
Istio is compatible with Kubernetes: v1.17.4.

#3. Istio-existence
-----------------------
Istio will be installed in the istio-system namespace.

#4. Kubernetes-setup
-----------------------
Can create necessary Kubernetes configurations: Namespace,ClusterRole,ClusterRoleBinding,CustomResourceDefinition,Role,ServiceAccoun                     t,Service,Deployments,ConfigMap.

#5. SideCar-Injector
-----------------------
This Kubernetes cluster supports automatic sidecar injection. To enable automatic sidecar injection see https://istio.io/docs/setup/                     kubernetes/additional-setup/sidecar-injection/#deploying-an-app

-----------------------
Install Pre-Check passed! The cluster is ready for Istio installation.

[root@k1 istio-1.5.1]# istioctl manifest apply --set profile=demo
Detected that your cluster does not support third party JWT authentication. Falling back to less secure first party                 JWT. See https://istio.io/docs/ops/best-practices/security/#configure-third-party-service-account-tokens for details                .
- Applying manifest for component Base...
✔ Finished applying manifest for component Base.
- Applying manifest for component Pilot...
✔ Finished applying manifest for component Pilot.
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
...........................
...........................
...........................
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
2020-04-02T21:51:57.048098Z     error   installer       Failed to wait for resource: resources not ready after 10m0s: timed out waiting for the condition
Deployment/istio-system/istiod
- Applying manifest for component EgressGateways...
- Applying manifest for component IngressGateways...
- Applying manifest for component AddonComponents...
✔ Finished applying manifest for component EgressGateways.
✔ Finished applying manifest for component AddonComponents.
✔ Finished applying manifest for component IngressGateways.

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

[root@k1 istio-1.5.1]# istioctl version --remote
Error: 1 error occurred:
        * error execing into istiod-646b6fcc6-9zp8k discovery container: unable to upgrade connection: container not found ("discovery")

[root@k1 istio-1.5.1]# kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T21:03:42Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T20:55:23Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed?

Installing

Environment where bug was observed (cloud vendor, OS, etc)

local k8s cluster

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 33 (9 by maintainers)

Most upvoted comments

  1. I first run the
kubectl get pods -n istio-system to get the pod information:
NAME                                    READY   STATUS    RESTARTS   AGE
istio-ingressgateway-6b989cdc67-r2c7v   1/1     Running   0          4d6h
istiod-5bcb74c764-n52gh                 1/1     Running   0          4d6h
  1. Then for debugging the istiod I run to get the detailed info kubectl describe pod istiod-5bcb74c764-n52gh -n istio-system

  2. Then I find that I wrongly set the hub and tag parameters(–istio.test.hub --istio.test.tag ) in my command. After fix it, it succeed

Not sure if is related but I have the same issue with resource timeout, without creating any pods ( the deploy objects and rs were created).

The thing was I had namespaces istio-system and istio-operator manually created. After 2h of tries, I decided to delete everything. After I deleted the ns istio-system and istio-operator and run the commands install, worked:

[charts:(⎈ |kubernetes-admin@kubernetes:istio-system)]# istioctl install --readiness-timeout 10s This will install the default Istio profile into the cluster. Proceed? (y/N) y Detected that your cluster does not support third party JWT authentication. Falling back to less secure first party JWT. See https://istio.io/docs/ops/best-practices/security/#configure-third-party-service-account-tokens for details. ✔ Istio core installed ✔ Istiod installed ✔ Ingress gateways installed ✔ Installation complete

As a mention this is a local cluster build on VMs under VMware with kubeadm.

@laurentiuspurba Yes,I am able to resolve the issues.Some pods are failing due to low memory and no of cpu .Just increase the memory and no of cpu usage.Then it will be installed.

I encountered the same issue, but it was the lack of CPU and Memory value in my case (using Minikube).

Istio’s document says:

Start minikube with 16384 MB of memory and 4 CPUs. This example uses Kubernetes version 1.20.2. You can change the version to any Kubernetes version supported by Istio by altering the --kubernetes-version value:

$ minikube start --memory=16384 --cpus=4 --kubernetes-version=v1.20.2

Depending on the hypervisor you use and the platform on which the hypervisor is run, minimum memory requirements vary. 16384 MB is sufficent to run Istio and bookinfo.

If you don’t have enough RAM allocated to the minikube virtual machine, the following errors could occur:

image pull failures
healthcheck timeout failures
kubectl failures on the host
general network instability of the virtual machine and the host
complete lock-up of the virtual machine
host NMI watchdog reboots

Installation went well after setting CPUs and Memory as designated above.

Found the actual issue:

The issue occurs if namespaces istio-system is labeled with “istio-injection=enabled”

Just remove the label and try again: kubectl label ns istio-system istio-injection-

I was able to upgrade from 1.4.3 -> 1.4.10 -> 1.5.8 -> 1.6.4. I could not have done it without help from Istio community (Prune and Vito).

Prior upgrading to other version, make sure to add newline character on ca-cert.pem as describe here.

  • 1.4.3 -> 1.4.10: I did the upgrade using istioctl experimental upgrade -f IstioControlPlane.yaml
    • You might need to use --force
  • 1.4.10 -> 1.5.8:
    • First, convert IstioControlPlane.yaml to IstioOperator.yaml
    • Before the upgrade kubectl -n istio-system delete service/istio-galley deployment.apps/istio-galley kubectl delete validatingwebhookconfiguration.admissionregistration.k8s.io/istio-galley
    • Use istioctl upgrade -f IstioOperator.yaml to upgrade (--force might be required)
    • After the upgrade kubectl -n istio-system delete deployment istio-citadel istio-galley istio-pilot istio-policy istio-sidecar-injector istio-telemetry kubectl -n istio-system delete service istio-citadel istio-policy istio-sidecar-injector istio-telemetry kubectl -n istio-system delete horizontalpodautoscaler.autoscaling/istio-pilot horizontalpodautoscaler.autoscaling/istio-telemetry kubectl -n istio-system delete pdb istio-citadel istio-galley istio-pilot istio-policy istio-sidecar-injector istio-telemetry kubectl -n istio-system delete deployment istiocoredns kubectl -n istio-system delete service istiocoredns
  • 1.5.8 to 1.64
    • Make necessary changes of IstioOperator.yaml Version 1.5.8 file to support Version 1.6.4
    • Use istioctl upgrade -f IstioOperator.yaml to upgrade (--force might be required)