istio: Timeout problem: Internal error occurred: failed calling webhook "sidecar-injector.istio.io"

I know this bug appeared here many times, but still any of the already existing issues were not sufficient to help my case

Describe the bug Replica set is not able to create any pods. In Events section there are multiple errors, but all include timeout message:

  Type     Reason        Age                 From                   Message
  ----     ------        ----                ----                   -------
  Warning  FailedCreate  50m (x2 over 61m)   replicaset-controller  Error creating: Internal error occurred: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: context deadline exceeded
  Warning  FailedCreate  33m (x15 over 81m)  replicaset-controller  Error creating: Internal error occurred: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Warning  FailedCreate  16m (x4 over 82m)   replicaset-controller  Error creating: Internal error occurred: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: dial tcp 10.243.225.152:443: i/o timeout

Expected behavior Replica set is able to post to sidecar, get automatic injects and create istio-proxied pods.

Steps to reproduce the bug

  • deploy istio using steps described below.
  • label default namespace with istio-injection=enabled
  • try deploying bookinfo sample

Version istio:

$ istioctl version --remote
Error: unknown flag: --remote

gives error so normal:

$ istioctl version
Version: 1.0.6
GitRevision: 98598f88f6ee9c1e6b3f03b652d8e0e3cd114fa2
User: root@464fc845-2bf8-11e9-b805-0a580a2c0506
Hub: docker.io/istio
GolangVersion: go1.10.4
BuildStatus: Clean

kubectl:

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-10T23:35:51Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

helm:

Client: &version.Version{SemVer:"v2.12.2", GitCommit:"7d2b0c73d734f6586ed222a567c5d103fed435be", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.2", GitCommit:"7d2b0c73d734f6586ed222a567c5d103fed435be", GitTreeState:"clean"}

Installation

// from istio-1.0.6 directory
$ kubectl apply -f install/kubernetes/helm/istio/templates/crds.yaml
$ kubectl apply -f install/kubernetes/helm/helm-service-account.yaml
$ helm init --service-account tiller --replicas=3
$ helm install --name istio --namespace istio-system \
    --set gateways.istio-egressgateway.enabled=false \
    --set galley.enabled=false \
    ./install/kubernetes/helm/istio

Environment Cluster hosted on Microsoft Azure, provisioned by aks-engine

aks-engine version
Version: v0.31.1
GitCommit: 13c427824
GitTreeState: clean

Nodes:

NAME     STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP     OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
agnt-0   Ready    agent    7d    v1.13.3   A.B.C.4         XXX             Ubuntu 16.04.5 LTS   4.15.0-1037-azure   docker://3.0.4
agnt-1   Ready    agent    7d    v1.13.3   A.B.C.6         XXX             Ubuntu 16.04.5 LTS   4.15.0-1037-azure   docker://3.0.4
agnt-2   Ready    agent    7d    v1.13.3   A.B.C.5         XXX             Ubuntu 16.04.5 LTS   4.15.0-1037-azure   docker://3.0.4
mstr-0   Ready    master   7d    v1.13.3   A.b.C.100       XXX             Ubuntu 16.04.5 LTS   4.15.0-1037-azure   docker://3.0.4

Cluster state ~Somehow the script didn’t work well for me (some errors appeared), but still was able to gather some data. istio-dump.tar.gz~ Update: istio-dump-1903.tar.gz

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 15 (1 by maintainers)

Most upvoted comments

@Morishiri Also encountered the same problem on Azure Virtual Network, but have no idea to solve it.Can you helo me ?

I had kubernetes cluster in Virtual Network in Azure, but proper route table was not assigned to VNet. So this was completely not istio issue in this case, but my fail deploying the cluster in Azure.

A fix for the dump_kubernetes.sh script that should fix the problem with reporting can be found at https://github.com/istio/istio/pull/12159

I don’t know the cause of the actual problem, the failed webhook.