istio: Timeout problem: Internal error occurred: failed calling webhook "sidecar-injector.istio.io"
I know this bug appeared here many times, but still any of the already existing issues were not sufficient to help my case
Describe the bug
Replica set is not able to create any pods. In Events
section there are multiple errors, but all include timeout message:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreate 50m (x2 over 61m) replicaset-controller Error creating: Internal error occurred: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: context deadline exceeded
Warning FailedCreate 33m (x15 over 81m) replicaset-controller Error creating: Internal error occurred: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning FailedCreate 16m (x4 over 82m) replicaset-controller Error creating: Internal error occurred: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: dial tcp 10.243.225.152:443: i/o timeout
Expected behavior Replica set is able to post to sidecar, get automatic injects and create istio-proxied pods.
Steps to reproduce the bug
- deploy istio using steps described below.
- label default namespace with
istio-injection=enabled
- try deploying
bookinfo
sample
Version istio:
$ istioctl version --remote
Error: unknown flag: --remote
gives error so normal:
$ istioctl version
Version: 1.0.6
GitRevision: 98598f88f6ee9c1e6b3f03b652d8e0e3cd114fa2
User: root@464fc845-2bf8-11e9-b805-0a580a2c0506
Hub: docker.io/istio
GolangVersion: go1.10.4
BuildStatus: Clean
kubectl:
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-10T23:35:51Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
helm:
Client: &version.Version{SemVer:"v2.12.2", GitCommit:"7d2b0c73d734f6586ed222a567c5d103fed435be", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.2", GitCommit:"7d2b0c73d734f6586ed222a567c5d103fed435be", GitTreeState:"clean"}
Installation
// from istio-1.0.6 directory
$ kubectl apply -f install/kubernetes/helm/istio/templates/crds.yaml
$ kubectl apply -f install/kubernetes/helm/helm-service-account.yaml
$ helm init --service-account tiller --replicas=3
$ helm install --name istio --namespace istio-system \
--set gateways.istio-egressgateway.enabled=false \
--set galley.enabled=false \
./install/kubernetes/helm/istio
Environment Cluster hosted on Microsoft Azure, provisioned by aks-engine
aks-engine version
Version: v0.31.1
GitCommit: 13c427824
GitTreeState: clean
Nodes:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
agnt-0 Ready agent 7d v1.13.3 A.B.C.4 XXX Ubuntu 16.04.5 LTS 4.15.0-1037-azure docker://3.0.4
agnt-1 Ready agent 7d v1.13.3 A.B.C.6 XXX Ubuntu 16.04.5 LTS 4.15.0-1037-azure docker://3.0.4
agnt-2 Ready agent 7d v1.13.3 A.B.C.5 XXX Ubuntu 16.04.5 LTS 4.15.0-1037-azure docker://3.0.4
mstr-0 Ready master 7d v1.13.3 A.b.C.100 XXX Ubuntu 16.04.5 LTS 4.15.0-1037-azure docker://3.0.4
Cluster state ~Somehow the script didn’t work well for me (some errors appeared), but still was able to gather some data. istio-dump.tar.gz~ Update: istio-dump-1903.tar.gz
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 15 (1 by maintainers)
@Morishiri Also encountered the same problem on Azure Virtual Network, but have no idea to solve it.Can you helo me ?
I had kubernetes cluster in Virtual Network in Azure, but proper route table was not assigned to VNet. So this was completely not istio issue in this case, but my fail deploying the cluster in Azure.
A fix for the dump_kubernetes.sh script that should fix the problem with reporting can be found at https://github.com/istio/istio/pull/12159
I don’t know the cause of the actual problem, the failed webhook.