istio: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "validation.istio.io": Post https://istiod.istio-system.svc:443/validate?timeout=30s:

Bug description Try to install last 1.5.0 version over Kubernetes 1.7, and consistently failed with error: Error from server (InternalError): error when creating “STDIN”: Internal error occurred: failed calling webhook “validation.istio.io”: Post https://istiod.istio-system.svc:443/validate?timeout=30s: dial tcp 10.111.86.18:443: connect: no route to host (repeated 2 times)

[ ] Configuration Infrastructure [ ] Docs [X ] Installation [ ] Networking [ ] Performance and Scalability [ ] Policies and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure

Expected behavior Expect to see istio deployed over kubernetes.

Steps to reproduce the bug istioctl manifest apply --set addonComponents.grafana.enabled=true

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm) istioctl version: 1.5.0.

kubectl version:

kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T20:55:23Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed? curl -L https://istio.io/downloadIstio | sh -

Environment where bug was observed (cloud vendor, OS, etc) Installation on premise - Centos 7

Full error output:

 istioctl manifest apply --set addonComponents.grafana.enabled=true
Detected that your cluster does not support third party JWT authentication. Falling back to less secure first party JWT. See https://istio.io/docs/ops/best-practices/security/#configure-third-party-service-account-tokens for details.
- Applying manifest for component Base...
✔ Finished applying manifest for component Base.
- Applying manifest for component Pilot...
✔ Finished applying manifest for component Pilot.
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
- Applying manifest for component IngressGateways...
- Applying manifest for component AddonComponents...
2020-03-19T12:39:04.197433Z	error	installer	error running kubectl: exit status 1
✘ Finished applying manifest for component AddonComponents.
2020-03-19T12:39:07.010758Z	error	installer	error running kubectl: exit status 1
✘ Finished applying manifest for component IngressGateways.

Component AddonComponents - manifest apply returned the following errors:
Error: error running kubectl: exit status 1

Error detail:


Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "validation.istio.io": Post https://istiod.istio-system.svc:443/validate?timeout=30s: dial tcp 10.111.86.18:443: connect: no route to host (repeated 1 times)


clusterrole.rbac.authorization.k8s.io/prometheus-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-istio-system created

serviceaccount/prometheus created
configmap/istio-grafana created
configmap/istio-grafana-configuration-dashboards-citadel-dashboard created
configmap/istio-grafana-configuration-dashboards-galley-dashboard created
configmap/istio-grafana-configuration-dashboards-istio-mesh-dashboard created
configmap/istio-grafana-configuration-dashboards-istio-performance-dashboard created
configmap/istio-grafana-configuration-dashboards-istio-service-dashboard created
configmap/istio-grafana-configuration-dashboards-istio-workload-dashboard created
configmap/istio-grafana-configuration-dashboards-mixer-dashboard created
configmap/istio-grafana-configuration-dashboards-pilot-dashboard created
configmap/prometheus created
deployment.apps/grafana created
deployment.apps/prometheus created
service/grafana created
service/prometheus created



Component IngressGateways - manifest apply returned the following errors:
Error: error running kubectl: exit status 1

Error detail:

Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "validation.istio.io": Post https://istiod.istio-system.svc:443/validate?timeout=30s: dial tcp 10.111.86.18:443: connect: no route to host (repeated 2 times)


serviceaccount/istio-ingressgateway-service-account created
deployment.apps/istio-ingressgateway created
poddisruptionbudget.policy/ingressgateway created
role.rbac.authorization.k8s.io/istio-ingressgateway-sds created
rolebinding.rbac.authorization.k8s.io/istio-ingressgateway-sds created
horizontalpodautoscaler.autoscaling/istio-ingressgateway created
service/istio-ingressgateway created




✘ Errors were logged during apply operation. Please check component installation logs above.

Error: failed to apply manifests: errors were logged during apply operation

Thanks.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 15 (2 by maintainers)

Most upvoted comments

i have same issue. k8s: 1.17.3 istio: 1.5.1

A little different:istio-ingressgateway is running. But the log of kube-apiserver still reports errors:

W0401 09:49:27.065594       1 dispatcher.go:128] Failed calling webhook, failing open validation.istio.io: failed calling webhook "validation.istio.io": Post https://istiod.istio-system.svc:443/validate?timeout=30s: context deadline exceeded
E0401 09:49:27.065620       1 dispatcher.go:129] failed calling webhook "validation.istio.io": Post https://istiod.istio-system.svc:443/validate?timeout=30s: context deadline exceeded

the log of istiod reports info:

2020-04-01T09:51:27.101082Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-04-01T09:51:27.101740Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=205973) is up-to-date. No change required.
2020-04-01T09:51:27.101773Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config

@feliperfmarques I am facing a similar issue with Centos 7 on-premise development. There is no firewall running and automatic sidecar injection for the pod is also failing.

kubernetes version: 1.18.1 istio version : 1.5.2

Any help much appreciated.

Error: Kube-api server:

– Failed calling webhook, failing open validation.istio.io: failed calling webhook “validation.istio.io”: Post https://istiod.istio-system.svc:443/validate?timeout=30s: context deadline exceeded

  • istiod-logs:

validationController Not ready to switch validation to fail-closed: dummy invalid config not rejected validationController validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=205973) is up-to-date. No change required. validationController Reconcile(enter): retry dry-run creation of invalid config

It’s a flannel issue. For now able to fix the issue by changing the flannel backend to host-gw from vxlan. Not sure why it didn’t work with vxlan. Thanks for @nustiueudinastea (Alex) for debugging the issue.

This issue can also to be a problem of webhook validation. For webhook works, firewall rule needs setting to port 15017 instead of 9443 on Istio 1.5, as mentioned here #19532 (comment). Try add this firewall rule.