istio: 1.10 regression: idle_timeout leads to NACK
[ ] Docs [ ] Installation [ ] Networking [ ] Performance and Scalability [ ] Extensions and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure [ *] Upgrade
Upgrade to version 1.10.0, publish istio service, istio-proxy cannot be started, error prompt:
[root@node0 maixiaolan]# kubectl describe pod basemq-57f9b7b96c-xvr7t
...
...
Normal Created <invalid> kubelet Created container istio-proxy
Normal Started <invalid> kubelet Started container istio-proxy
Warning Unhealthy <invalid> (x2 over <invalid>) kubelet Readiness probe failed: Get "http://10.244.166.181:15021/healthz/ready": dial tcp 10.244.166.181:15021: connect: connection refused
Warning Unhealthy <invalid> (x13 over <invalid>) kubelet Readiness probe failed: HTTP probe failed with statuscode: 503
[root@node0 maixiaolan]# kubectl logs basemq-c6bd6f877-n2lbs -c istio-proxy -f
2021-05-26T12:28:57.793386Z info FLAG: --concurrency="2"
2021-05-26T12:28:57.793486Z info FLAG: --domain="maixiaolan-test.svc.cluster.local"
2021-05-26T12:28:57.793504Z info FLAG: --help="false"
2021-05-26T12:28:57.793515Z info FLAG: --log_as_json="false"
2021-05-26T12:28:57.793525Z info FLAG: --log_caller=""
2021-05-26T12:28:57.793536Z info FLAG: --log_output_level="default:info"
2021-05-26T12:28:57.793545Z info FLAG: --log_rotate=""
2021-05-26T12:28:57.793555Z info FLAG: --log_rotate_max_age="30"
2021-05-26T12:28:57.793566Z info FLAG: --log_rotate_max_backups="1000"
2021-05-26T12:28:57.793577Z info FLAG: --log_rotate_max_size="104857600"
2021-05-26T12:28:57.793587Z info FLAG: --log_stacktrace_level="default:none"
2021-05-26T12:28:57.793607Z info FLAG: --log_target="[stdout]"
2021-05-26T12:28:57.793624Z info FLAG: --meshConfig="./etc/istio/config/mesh"
2021-05-26T12:28:57.793635Z info FLAG: --outlierLogPath=""
2021-05-26T12:28:57.793645Z info FLAG: --proxyComponentLogLevel="misc:error"
2021-05-26T12:28:57.793655Z info FLAG: --proxyLogLevel="warning"
2021-05-26T12:28:57.793678Z info FLAG: --serviceCluster="basemq.maixiaolan-test"
2021-05-26T12:28:57.793688Z info FLAG: --stsPort="0"
2021-05-26T12:28:57.793698Z info FLAG: --templateFile=""
2021-05-26T12:28:57.793767Z info FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2021-05-26T12:28:57.793781Z info Version 1.10.0-d26cba7e341587453ffeb978f5cf6fbc32f346f8-Clean
2021-05-26T12:28:57.794230Z info Proxy role ips=[10.244.166.182] type=sidecar id=basemq-c6bd6f877-n2lbs.maixiaolan-test domain=maixiaolan-test.svc.cluster.local
2021-05-26T12:28:57.794453Z info Apply proxy config from env {}
2021-05-26T12:28:57.797559Z info Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: basemq.maixiaolan-test
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
zipkin:
address: zipkin.istio-system:9411
2021-05-26T12:28:57.797595Z info JWT policy is third-party-jwt
2021-05-26T12:28:57.797619Z info Pilot SAN: [istiod.istio-system.svc]
2021-05-26T12:28:57.797641Z info CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2021-05-26T12:28:57.797749Z info Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2021-05-26T12:28:57.798150Z info citadelclient Citadel client using custom root cert: istiod.istio-system.svc:15012
2021-05-26T12:28:57.895999Z info ads All caches have been synced up in 110.192332ms, marking server ready
2021-05-26T12:28:57.896945Z info sds SDS server for workload certificates started, listening on "./etc/istio/proxy/SDS"
2021-05-26T12:28:57.897042Z info xdsproxy Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"
2021-05-26T12:28:57.897112Z info sds Start SDS grpc server
2021-05-26T12:28:57.897862Z info Opening status port 15020
2021-05-26T12:28:59.404147Z info cache generated new workload certificate latency=1.506980225s ttl=23h57m11.595946512s
2021-05-26T12:28:59.404403Z info cache Root cert has changed, start rotating root cert
2021-05-26T12:28:59.404501Z info ads XDS: Incremental Pushing:0 ConnectedEndpoints:0 Version:
2021-05-26T12:28:59.404667Z info cache returned workload trust anchor from cache ttl=23h57m11.595358411s
2021-05-26T12:28:59.901592Z info Starting proxy agent
2021-05-26T12:28:59.901737Z info Epoch 0 starting
2021-05-26T12:28:59.908585Z info Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --drain-strategy immediate --parent-shutdown-time-s 60 --service-cluster basemq.maixiaolan-test --service-node sidecar~10.244.166.182~basemq-c6bd6f877-n2lbs.maixiaolan-test~maixiaolan-test.svc.cluster.local --local-address-ip-version v4 --bootstrap-version 3 --disable-hot-restart --log-format %Y-%m-%dT%T.%fZ %l envoy %n %v -l warning --component-log-level misc:error --concurrency 2]
2021-05-26T12:29:00.173351Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-05-26T12:29:00.217932Z warning envoy config gRPC config for type.googleapis.com/envoy.config.cluster.v3.Cluster rejected: Error adding/updating cluster(s) outbound|8080||basemq.maixiaolan-test.svc.cluster.local: Proto constraint validation failed (field: "upstream_protocol_options", reason: is required): common_http_protocol_options {
idle_timeout {
seconds: 60
}
}
, outbound|8080|v22483|basemq.maixiaolan-test.svc.cluster.local: Proto constraint validation failed (field: "upstream_protocol_options", reason: is required): common_http_protocol_options {
idle_timeout {
seconds: 60
}
}
, inbound|8080||: Proto constraint validation failed (field: "upstream_protocol_options", reason: is required): common_http_protocol_options {
idle_timeout {
seconds: 60
}
}
2021-05-26T12:29:00.226936Z info ads ADS: new connection for node:sidecar~10.244.166.182~basemq-c6bd6f877-n2lbs.maixiaolan-test~maixiaolan-test.svc.cluster.local-1
2021-05-26T12:29:00.227126Z info cache returned workload certificate from cache ttl=23h57m10.772887875s
2021-05-26T12:29:00.227511Z info ads ADS: new connection for node:sidecar~10.244.166.182~basemq-c6bd6f877-n2lbs.maixiaolan-test~maixiaolan-test.svc.cluster.local-2
2021-05-26T12:29:00.227792Z info cache returned workload trust anchor from cache ttl=23h57m10.772224073s
2021-05-26T12:29:00.227948Z info sds SDS: PUSH resource=default
2021-05-26T12:29:00.228312Z info sds SDS: PUSH resource=ROOTCA
2021-05-26T12:29:01.883253Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 1 rejected; lds updates: 1 successful, 0 rejected
2021-05-26T12:29:03.882752Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 1 rejected; lds updates: 1 successful, 0 rejected
2021-05-26T12:29:05.882075Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 1 rejected; lds updates: 1 successful, 0 rejected
2021-05-26T12:29:07.882387Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 1 rejected; lds updates: 1 successful, 0 rejected
2021-05-26T12:29:09.881921Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 1 rejected; lds updates: 1 successful, 0 rejected
2021-05-26T12:29:11.882289Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 1 rejected; lds updates: 1 successful, 0 rejected
2021-05-26T12:29:1
[root@node0 maixiaolan]# kubectl get pods
NAME READY STATUS RESTARTS AGE
basemq-57f9b7b96c-xvr7t 1/2 Running 0 62s
[root@node0 maixiaolan]# kubectl get svc -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istio-ingressgateway LoadBalancer 10.245.40.6 <pending> 8901:8901/TCP,8900:8900/TCP,8080:8080/TCP,58080:58080/TCP,15021:18907/TCP,80:52604/TCP,443:697/TCP,**15012**:10434/TCP,15443:64884/TCP 22h
istiod ClusterIP 10.245.148.175 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP
no problem with version 1.9.5. How to solve it?
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 19 (11 by maintainers)
1.10.1
will be released this week, targeting Thursday 06/10.@yu-shiba @howardjohn This problem is solved first. Thank you very much for your help !
It seems that version 1.10.0 has changed a lot. Some of the configuration properties of evoryfilter have also changed. I will continue to modify it…
https://github.com/envoyproxy/envoy/blob/main/api/envoy/extensions/upstreams/http/v3/http_protocol_options.proto#L135
@yu-shiba Remove “idleTimeout” from DestinationRule, running successfully!
Is this version not supported for this parameter?
This is the definition of something like this.
In my environment, it happens when I do a fresh install to switch to 1.10.0. I was able to successfully install 1.9.4 with the same settings.