istio: LDS update causing 503 no healthy upstream spike
Is this the right place to submit this?
- This is not a security vulnerability or a crashing bug
- This is not a question about how to use Istio
Bug Description
We applied below EnvoyFilter to update the AccessLog sampling and format:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: sample-200-response-code-accesslog-1.17
namespace: istio-production
spec:
configPatches:
- applyTo: NETWORK_FILTER
match:
context: ANY
listener:
filterChain:
filter:
name: envoy.filters.network.http_connection_manager
proxy:
proxyVersion: ^1\.17.*
patch:
operation: MERGE
value:
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
access_log:
- filter:
and_filter:
filters:
- not_health_check_filter: {}
- or_filter:
filters:
- and_filter:
filters:
- runtime_filter:
percent_sampled:
denominator: HUNDRED
numerator: 1
runtime_key: http_ok_response_sampling_fraction
use_independent_randomness: true
- status_code_filter:
comparison:
op: EQ
value:
default_value: 200
runtime_key: http_ok_response_sampling_status_eq
- and_filter:
filters:
- runtime_filter:
percent_sampled:
denominator: HUNDRED
numerator: 100
runtime_key: http_err_response_sampling_fraction
use_independent_randomness: true
- or_filter:
filters:
- status_code_filter:
comparison:
op: LE
value:
default_value: 199
runtime_key: http_err_response_sampling_status_le
- status_code_filter:
comparison:
op: GE
value:
default_value: 201
runtime_key: http_err_response_sampling_status_ge
name: envoy.access_loggers.file
typed_config:
'@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
log_format:
json_format:
authority: '%REQ(:AUTHORITY)%'
bytes_received: '%BYTES_RECEIVED%'
bytes_sent: '%BYTES_SENT%'
connection_termination_details: '%CONNECTION_TERMINATION_DETAILS%'
downstream_local_address: '%DOWNSTREAM_LOCAL_ADDRESS%'
downstream_local_uri_san: '%DOWNSTREAM_LOCAL_URI_SAN%'
downstream_peer_issuer: '%DOWNSTREAM_PEER_ISSUER%'
downstream_peer_subject: '%DOWNSTREAM_PEER_SUBJECT%'
downstream_peer_uri_san: '%DOWNSTREAM_PEER_URI_SAN%'
downstream_remote_address: '%DOWNSTREAM_REMOTE_ADDRESS%'
downstream_tls_cipher: '%DOWNSTREAM_TLS_CIPHER%'
downstream_tls_version: '%DOWNSTREAM_TLS_VERSION%'
duration: '%DURATION%'
method: '%REQ(:METHOD)%'
path: '%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%'
protocol: '%PROTOCOL%'
request_id: '%REQ(X-REQUEST-ID)%'
requested_server_name: '%REQUESTED_SERVER_NAME%'
response_code: '%RESPONSE_CODE%'
response_code_details: '%RESPONSE_CODE_DETAILS%'
response_flags: '%RESPONSE_FLAGS%'
rlog_id: '%RESP(RLOGID)%'
route_name: '%ROUTE_NAME%'
start_time: '%START_TIME%'
upstream_cluster: '%UPSTREAM_CLUSTER%'
upstream_host: '%UPSTREAM_HOST%'
upstream_local_address: '%UPSTREAM_LOCAL_ADDRESS%'
upstream_service_time: '%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%'
upstream_transport_failure_reason: '%UPSTREAM_TRANSPORT_FAILURE_REASON%'
user_agent: '%REQ(USER-AGENT)%'
x_ebay_request_id: '%REQ(X-EBAY-REQUEST-ID)%'
x_forwarded_for: '%REQ(X-FORWARDED-FOR)%'
path: /var/log/proxy/access.log
And after applying the EnvoyFilter, the gateway upstream connection dropped to the floor:
The 5xx spike came after the listener drain spike:
And gateway is using below drain strategy:
--drain-time-s 45 --drain-strategy immediate
This issue can be reproduced in kind:
2023-11-20T08:03:23.518272Z debug envoy init external/envoy/source/common/init/manager_impl.cc:24 added shared target RdsRouteConfigSubscription RDS init http.8080.192.168.119.5 to init manager Listener-local-init-manager 192.168.119.5_8080 5254667097225700325 thread=22
2023-11-20T08:03:23.523895Z debug envoy config external/envoy/source/extensions/listener_managers/listener_manager/listener_impl.cc:968 add warming listener: name=192.168.119.5_8080, hash=5254667097225700325, tag=5, address=192.168.119.5:8080 thread=22 // listener is in warming state
2023-11-20T08:03:23.523943Z debug envoy misc external/envoy/source/extensions/listener_managers/listener_manager/listener_impl.cc:977 Initialize listener 192.168.119.5_8080 local-init-manager. thread=22
2023-11-20T08:03:23.523961Z debug envoy init external/envoy/source/common/init/manager_impl.cc:53 init manager Listener-local-init-manager 192.168.119.5_8080 5254667097225700325 initializing thread=22
2023-11-20T08:03:23.523981Z debug envoy init external/envoy/source/common/init/target_impl.cc:15 init manager Listener-local-init-manager 192.168.119.5_8080 5254667097225700325 initializing shared target RdsRouteConfigSubscription RDS init http.8080.192.168.119.5 thread=22 // init rds for listener which includes cds and eds
2023-11-20T08:03:23.524011Z debug envoy init external/envoy/source/common/init/watcher_impl.cc:14 shared target RdsRouteConfigSubscription RDS init http.8080.192.168.119.5 initialized, notifying init manager Listener-local-init-manager 192.168.119.5_8080 5254667097225700325 thread=22
2023-11-20T08:03:23.524027Z debug envoy init external/envoy/source/common/init/watcher_impl.cc:14 init manager Listener-local-init-manager 192.168.119.5_8080 5254667097225700325 initialized, notifying Listener-local-init-watcher 192.168.119.5_8080 thread=22
2023-11-20T08:03:23.524049Z debug envoy config external/envoy/source/extensions/listener_managers/listener_manager/listener_impl.cc:968 execute in place filter chain update: name=192.168.119.5_8080, hash=5254667097225700325, tag=5, address=192.168.119.5:8080 thread=22
2023-11-20T08:03:23.526576Z debug envoy config external/envoy/source/extensions/listener_managers/listener_manager/listener_impl.cc:968 draining 1 filter chains in listener 192.168.119.5_8080: name=192.168.119.5_8080, hash=10684809757490173150, tag=5, address=192.168.119.5:8080 thread=22 //started draining
2023-11-20T08:03:23.526683Z info envoy upstream external/envoy/source/extensions/listener_managers/listener_manager/lds_api.cc:82 lds: add/update listener '192.168.119.5_8080' thread=22
2023-11-20T08:03:24.101972Z debug envoy config external/envoy/source/common/config/grpc_subscription_impl.cc:89 gRPC config update took 611 ms! Resources names: 192.168.119.5_8080,192.168.119.8_8443,192.168.119.4_8443,192.168.119.10_8080,192.168.119.7_8080,192.168.119.7_8443,192.168.119.8_8080,192.168.119.3_8080,192.168.119.6_8080,192.168.119.3_8443,192.168.119.9_8443,192.168.119.9_8080,192.168.119.10_8443,192.168.119.5_8443,192.168.119.6_8443,192.168.119.4_8080 thread=22
2023-11-20T08:04:08.496535Z debug envoy config external/envoy/source/extensions/listener_managers/listener_manager/listener_impl.cc:968 removing draining filter chains from listener 192.168.119.5_8080: name=192.168.119.5_8080, hash=10684809757490173150, tag=5, address=192.168.119.5:8080 thread=22
2023-11-20T08:04:08.508755Z debug envoy config external/envoy/source/extensions/listener_managers/listener_manager/listener_impl.cc:968 draining filter chains from listener 192.168.119.5_8080 complete: name=192.168.119.5_8080, hash=10684809757490173150, tag=5, address=192.168.119.5:8080 thread=22
2023-11-20T08:04:08.508772Z debug envoy init external/envoy/source/common/init/watcher_impl.cc:31 Listener-local-init-watcher 192.168.119.5_8080 destroyed thread=22
As per https://github.com/envoyproxy/envoy/issues/26775 The ordering at initialization is specified at https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/init.html. At runtime, there is no relationship between CDS and LDS, they are independent subsystem and do not warm on each other. See also issues such as #5174.
And https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/lds#config-listeners-lds Listeners are effectively constant once created. Thus, when a listener is updated, an entirely new listener is created (if the listener’s address is unchanged, the new one uses the same listen socket). This listener goes through the same warming process described above for a newly added listener.
So there could be a time gap between the listener warming and draining, especially when the mesh size is big with hundreds of routes/clusters and even bigger EDS(for the production case)
2023-11-20T08:03:23.523895Z added warming listener 192.168.119.5_8080,
2023-11-20T08:03:23.526576Z started draining listener 192.168.119.5_8080,
The new connections comes to the Envoy in between will hit the warming listener and since the RDS/CDS/LDS are initialized independently with LDS, it will result to 503 no healthy upstream spike.
The gateway xds config size is much smaller than production so the time gap here is less than 1s, but in production it takes longer and it took few minutes(~3mins)to drain for 200 listeners, and will also take longer to warm up the new listeners.
Expected:
Ideally lds change should not cause data plane impact if it can make sure the warming listeners are ready before draining the listener.
Version
istio 1.17
k8s 1.23
Additional Information
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 16 (14 by maintainers)
Previously I was thinking the problem is in LDS but seems not. Listener update will trigger listener rebuild and add new listeners with warming state, meanwhile it will wait for RDS to be ready and only after RDS ready the listener will be active. This can explain why no 404 response code with NR(no route) since RDS is ready. However it doesn’t guarantee CDS/EDS are ready since they are initialized independently. But for this case CDS should be ready otherwise the response flag will be NC( no cluster), and 503 no healthy upstream means something is wrong with the EDS.
The problem here is EDS, there was istiod rollout but no ingressgateway rollout.
For the ingressgateway pods got connection drop, they connected to a BAD istiod pod which pushed 321 empty clusters to the ingressgateway:
While meantime there was a good istiod after rollout and in the initial push the empty cluster number is just 17:
For the bad istiod, discovery took aboout 5 mins to initialize:
While for the good istiod it just took 50seconds to initialize:
Meanwhile there was apiserver rollout which could cause some churn, but it seems there was also eds bug here? Why all the clusters eds built were empty? cc @hzxuzhonghu @howardjohn
@hzxuzhonghu I think similar issue here https://github.com/envoyproxy/envoy/issues/26775, my uderstading is that LDS warming doesn’t guarantee CDS/EDS is ready.