istio: 1.10 regression: idle_timeout leads to NACK

[ ] Docs [ ] Installation [ ] Networking [ ] Performance and Scalability [ ] Extensions and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure [ *] Upgrade

Upgrade to version 1.10.0, publish istio service, istio-proxy cannot be started, error prompt:

[root@node0 maixiaolan]# kubectl describe pod basemq-57f9b7b96c-xvr7t

...
...
  Normal   Created    <invalid>                       kubelet            Created container istio-proxy
  Normal   Started    <invalid>                       kubelet            Started container istio-proxy
  Warning  Unhealthy  <invalid> (x2 over <invalid>)   kubelet            Readiness probe failed: Get "http://10.244.166.181:15021/healthz/ready": dial tcp 10.244.166.181:15021: connect: connection refused
  Warning  Unhealthy  <invalid> (x13 over <invalid>)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503

[root@node0 maixiaolan]# kubectl logs basemq-c6bd6f877-n2lbs -c istio-proxy -f

2021-05-26T12:28:57.793386Z	info	FLAG: --concurrency="2"
2021-05-26T12:28:57.793486Z	info	FLAG: --domain="maixiaolan-test.svc.cluster.local"
2021-05-26T12:28:57.793504Z	info	FLAG: --help="false"
2021-05-26T12:28:57.793515Z	info	FLAG: --log_as_json="false"
2021-05-26T12:28:57.793525Z	info	FLAG: --log_caller=""
2021-05-26T12:28:57.793536Z	info	FLAG: --log_output_level="default:info"
2021-05-26T12:28:57.793545Z	info	FLAG: --log_rotate=""
2021-05-26T12:28:57.793555Z	info	FLAG: --log_rotate_max_age="30"
2021-05-26T12:28:57.793566Z	info	FLAG: --log_rotate_max_backups="1000"
2021-05-26T12:28:57.793577Z	info	FLAG: --log_rotate_max_size="104857600"
2021-05-26T12:28:57.793587Z	info	FLAG: --log_stacktrace_level="default:none"
2021-05-26T12:28:57.793607Z	info	FLAG: --log_target="[stdout]"
2021-05-26T12:28:57.793624Z	info	FLAG: --meshConfig="./etc/istio/config/mesh"
2021-05-26T12:28:57.793635Z	info	FLAG: --outlierLogPath=""
2021-05-26T12:28:57.793645Z	info	FLAG: --proxyComponentLogLevel="misc:error"
2021-05-26T12:28:57.793655Z	info	FLAG: --proxyLogLevel="warning"
2021-05-26T12:28:57.793678Z	info	FLAG: --serviceCluster="basemq.maixiaolan-test"
2021-05-26T12:28:57.793688Z	info	FLAG: --stsPort="0"
2021-05-26T12:28:57.793698Z	info	FLAG: --templateFile=""
2021-05-26T12:28:57.793767Z	info	FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2021-05-26T12:28:57.793781Z	info	Version 1.10.0-d26cba7e341587453ffeb978f5cf6fbc32f346f8-Clean
2021-05-26T12:28:57.794230Z	info	Proxy role	ips=[10.244.166.182] type=sidecar id=basemq-c6bd6f877-n2lbs.maixiaolan-test domain=maixiaolan-test.svc.cluster.local
2021-05-26T12:28:57.794453Z	info	Apply proxy config from env {}

2021-05-26T12:28:57.797559Z	info	Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: basemq.maixiaolan-test
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
  zipkin:
    address: zipkin.istio-system:9411

2021-05-26T12:28:57.797595Z	info	JWT policy is third-party-jwt
2021-05-26T12:28:57.797619Z	info	Pilot SAN: [istiod.istio-system.svc]
2021-05-26T12:28:57.797641Z	info	CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2021-05-26T12:28:57.797749Z	info	Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2021-05-26T12:28:57.798150Z	info	citadelclient	Citadel client using custom root cert: istiod.istio-system.svc:15012
2021-05-26T12:28:57.895999Z	info	ads	All caches have been synced up in 110.192332ms, marking server ready
2021-05-26T12:28:57.896945Z	info	sds	SDS server for workload certificates started, listening on "./etc/istio/proxy/SDS"
2021-05-26T12:28:57.897042Z	info	xdsproxy	Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"
2021-05-26T12:28:57.897112Z	info	sds	Start SDS grpc server
2021-05-26T12:28:57.897862Z	info	Opening status port 15020
2021-05-26T12:28:59.404147Z	info	cache	generated new workload certificate	latency=1.506980225s ttl=23h57m11.595946512s
2021-05-26T12:28:59.404403Z	info	cache	Root cert has changed, start rotating root cert
2021-05-26T12:28:59.404501Z	info	ads	XDS: Incremental Pushing:0 ConnectedEndpoints:0 Version:
2021-05-26T12:28:59.404667Z	info	cache	returned workload trust anchor from cache	ttl=23h57m11.595358411s
2021-05-26T12:28:59.901592Z	info	Starting proxy agent
2021-05-26T12:28:59.901737Z	info	Epoch 0 starting
2021-05-26T12:28:59.908585Z	info	Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --drain-strategy immediate --parent-shutdown-time-s 60 --service-cluster basemq.maixiaolan-test --service-node sidecar~10.244.166.182~basemq-c6bd6f877-n2lbs.maixiaolan-test~maixiaolan-test.svc.cluster.local --local-address-ip-version v4 --bootstrap-version 3 --disable-hot-restart --log-format %Y-%m-%dT%T.%fZ	%l	envoy %n	%v -l warning --component-log-level misc:error --concurrency 2]
2021-05-26T12:29:00.173351Z	info	xdsproxy	connected to upstream XDS server: istiod.istio-system.svc:15012
2021-05-26T12:29:00.217932Z	warning	envoy config	gRPC config for type.googleapis.com/envoy.config.cluster.v3.Cluster rejected: Error adding/updating cluster(s) outbound|8080||basemq.maixiaolan-test.svc.cluster.local: Proto constraint validation failed (field: "upstream_protocol_options", reason: is required): common_http_protocol_options {
  idle_timeout {
    seconds: 60
  }
}
, outbound|8080|v22483|basemq.maixiaolan-test.svc.cluster.local: Proto constraint validation failed (field: "upstream_protocol_options", reason: is required): common_http_protocol_options {
  idle_timeout {
    seconds: 60
  }
}
, inbound|8080||: Proto constraint validation failed (field: "upstream_protocol_options", reason: is required): common_http_protocol_options {
  idle_timeout {
    seconds: 60
  }
}

2021-05-26T12:29:00.226936Z	info	ads	ADS: new connection for node:sidecar~10.244.166.182~basemq-c6bd6f877-n2lbs.maixiaolan-test~maixiaolan-test.svc.cluster.local-1
2021-05-26T12:29:00.227126Z	info	cache	returned workload certificate from cache	ttl=23h57m10.772887875s
2021-05-26T12:29:00.227511Z	info	ads	ADS: new connection for node:sidecar~10.244.166.182~basemq-c6bd6f877-n2lbs.maixiaolan-test~maixiaolan-test.svc.cluster.local-2
2021-05-26T12:29:00.227792Z	info	cache	returned workload trust anchor from cache	ttl=23h57m10.772224073s
2021-05-26T12:29:00.227948Z	info	sds	SDS: PUSH	resource=default
2021-05-26T12:29:00.228312Z	info	sds	SDS: PUSH	resource=ROOTCA
2021-05-26T12:29:01.883253Z	warn	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 1 rejected; lds updates: 1 successful, 0 rejected
2021-05-26T12:29:03.882752Z	warn	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 1 rejected; lds updates: 1 successful, 0 rejected
2021-05-26T12:29:05.882075Z	warn	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 1 rejected; lds updates: 1 successful, 0 rejected
2021-05-26T12:29:07.882387Z	warn	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 1 rejected; lds updates: 1 successful, 0 rejected
2021-05-26T12:29:09.881921Z	warn	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 1 rejected; lds updates: 1 successful, 0 rejected
2021-05-26T12:29:11.882289Z	warn	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 1 rejected; lds updates: 1 successful, 0 rejected
2021-05-26T12:29:1

[root@node0 maixiaolan]# kubectl get pods

NAME                                 READY   STATUS    RESTARTS   AGE
basemq-57f9b7b96c-xvr7t              1/2     Running   0          62s

[root@node0 maixiaolan]# kubectl get svc -n istio-system

NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                                                                                              AGE
istio-ingressgateway   LoadBalancer   10.245.40.6      <pending>     8901:8901/TCP,8900:8900/TCP,8080:8080/TCP,58080:58080/TCP,15021:18907/TCP,80:52604/TCP,443:697/TCP,**15012**:10434/TCP,15443:64884/TCP   22h
istiod                 ClusterIP      10.245.148.175   <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP

no problem with version 1.9.5. How to solve it?

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 19 (11 by maintainers)

Most upvoted comments

1.10.1 will be released this week, targeting Thursday 06/10.

@yu-shiba @howardjohn This problem is solved first. Thank you very much for your help !

It seems that version 1.10.0 has changed a lot. Some of the configuration properties of evoryfilter have also changed. I will continue to modify it…

@yu-shiba Remove “idleTimeout” from DestinationRule, running successfully!

Is this version not supported for this parameter?

This is the definition of something like this.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: hoge.fuga
  namespace: develop
spec:
  host: hoge.fuga
  trafficPolicy:
    connectionPool:
      http:
        idleTimeout: 55s
        maxRetries: 0

In my environment, it happens when I do a fresh install to switch to 1.10.0. I was able to successfully install 1.9.4 with the same settings.