istio: LDS update causing 503 no healthy upstream spike

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
  • This is not a question about how to use Istio

Bug Description

We applied below EnvoyFilter to update the AccessLog sampling and format:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: sample-200-response-code-accesslog-1.17
  namespace: istio-production
spec:
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: ANY
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
      proxy:
        proxyVersion: ^1\.17.*
    patch:
      operation: MERGE
      value:
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          access_log:
          - filter:
              and_filter:
                filters:
                - not_health_check_filter: {}
                - or_filter:
                    filters:
                    - and_filter:
                        filters:
                        - runtime_filter:
                            percent_sampled:
                              denominator: HUNDRED
                              numerator: 1
                            runtime_key: http_ok_response_sampling_fraction
                            use_independent_randomness: true
                        - status_code_filter:
                            comparison:
                              op: EQ
                              value:
                                default_value: 200
                                runtime_key: http_ok_response_sampling_status_eq
                    - and_filter:
                        filters:
                        - runtime_filter:
                            percent_sampled:
                              denominator: HUNDRED
                              numerator: 100
                            runtime_key: http_err_response_sampling_fraction
                            use_independent_randomness: true
                        - or_filter:
                            filters:
                            - status_code_filter:
                                comparison:
                                  op: LE
                                  value:
                                    default_value: 199
                                    runtime_key: http_err_response_sampling_status_le
                            - status_code_filter:
                                comparison:
                                  op: GE
                                  value:
                                    default_value: 201
                                    runtime_key: http_err_response_sampling_status_ge
            name: envoy.access_loggers.file
            typed_config:
              '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
              log_format:
                json_format:
                  authority: '%REQ(:AUTHORITY)%'
                  bytes_received: '%BYTES_RECEIVED%'
                  bytes_sent: '%BYTES_SENT%'
                  connection_termination_details: '%CONNECTION_TERMINATION_DETAILS%'
                  downstream_local_address: '%DOWNSTREAM_LOCAL_ADDRESS%'
                  downstream_local_uri_san: '%DOWNSTREAM_LOCAL_URI_SAN%'
                  downstream_peer_issuer: '%DOWNSTREAM_PEER_ISSUER%'
                  downstream_peer_subject: '%DOWNSTREAM_PEER_SUBJECT%'
                  downstream_peer_uri_san: '%DOWNSTREAM_PEER_URI_SAN%'
                  downstream_remote_address: '%DOWNSTREAM_REMOTE_ADDRESS%'
                  downstream_tls_cipher: '%DOWNSTREAM_TLS_CIPHER%'
                  downstream_tls_version: '%DOWNSTREAM_TLS_VERSION%'
                  duration: '%DURATION%'
                  method: '%REQ(:METHOD)%'
                  path: '%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%'
                  protocol: '%PROTOCOL%'
                  request_id: '%REQ(X-REQUEST-ID)%'
                  requested_server_name: '%REQUESTED_SERVER_NAME%'
                  response_code: '%RESPONSE_CODE%'
                  response_code_details: '%RESPONSE_CODE_DETAILS%'
                  response_flags: '%RESPONSE_FLAGS%'
                  rlog_id: '%RESP(RLOGID)%'
                  route_name: '%ROUTE_NAME%'
                  start_time: '%START_TIME%'
                  upstream_cluster: '%UPSTREAM_CLUSTER%'
                  upstream_host: '%UPSTREAM_HOST%'
                  upstream_local_address: '%UPSTREAM_LOCAL_ADDRESS%'
                  upstream_service_time: '%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%'
                  upstream_transport_failure_reason: '%UPSTREAM_TRANSPORT_FAILURE_REASON%'
                  user_agent: '%REQ(USER-AGENT)%'
                  x_ebay_request_id: '%REQ(X-EBAY-REQUEST-ID)%'
                  x_forwarded_for: '%REQ(X-FORWARDED-FOR)%'
              path: /var/log/proxy/access.log

And after applying the EnvoyFilter, the gateway upstream connection dropped to the floor: Screenshot 2023-11-23 at 14 09 41

The 5xx spike came after the listener drain spike: Screenshot 2023-11-23 at 14 15 37

Screenshot 2023-11-23 at 15 11 41

And gateway is using below drain strategy:

--drain-time-s 45 --drain-strategy immediate

This issue can be reproduced in kind:

2023-11-20T08:03:23.518272Z debug envoy init external/envoy/source/common/init/manager_impl.cc:24 added shared target RdsRouteConfigSubscription RDS init http.8080.192.168.119.5 to init manager Listener-local-init-manager 192.168.119.5_8080 5254667097225700325  thread=22
2023-11-20T08:03:23.523895Z debug envoy config external/envoy/source/extensions/listener_managers/listener_manager/listener_impl.cc:968 add warming listener: name=192.168.119.5_8080, hash=5254667097225700325, tag=5, address=192.168.119.5:8080  thread=22 // listener is in warming state
2023-11-20T08:03:23.523943Z debug envoy misc external/envoy/source/extensions/listener_managers/listener_manager/listener_impl.cc:977 Initialize listener 192.168.119.5_8080 local-init-manager.  thread=22
2023-11-20T08:03:23.523961Z debug envoy init external/envoy/source/common/init/manager_impl.cc:53 init manager Listener-local-init-manager 192.168.119.5_8080 5254667097225700325 initializing  thread=22
2023-11-20T08:03:23.523981Z debug envoy init external/envoy/source/common/init/target_impl.cc:15  init manager Listener-local-init-manager 192.168.119.5_8080 5254667097225700325 initializing shared target RdsRouteConfigSubscription RDS init http.8080.192.168.119.5  thread=22 // init rds for listener which includes cds and eds
2023-11-20T08:03:23.524011Z debug envoy init external/envoy/source/common/init/watcher_impl.cc:14 shared target RdsRouteConfigSubscription RDS init http.8080.192.168.119.5 initialized, notifying init manager Listener-local-init-manager 192.168.119.5_8080 5254667097225700325  thread=22
2023-11-20T08:03:23.524027Z debug envoy init external/envoy/source/common/init/watcher_impl.cc:14 init manager Listener-local-init-manager 192.168.119.5_8080 5254667097225700325 initialized, notifying Listener-local-init-watcher 192.168.119.5_8080 thread=22
2023-11-20T08:03:23.524049Z debug envoy config external/envoy/source/extensions/listener_managers/listener_manager/listener_impl.cc:968 execute in place filter chain update: name=192.168.119.5_8080, hash=5254667097225700325, tag=5, address=192.168.119.5:8080  thread=22
2023-11-20T08:03:23.526576Z debug envoy config external/envoy/source/extensions/listener_managers/listener_manager/listener_impl.cc:968 draining 1 filter chains in listener 192.168.119.5_8080: name=192.168.119.5_8080,  hash=10684809757490173150, tag=5, address=192.168.119.5:8080 thread=22 //started draining
2023-11-20T08:03:23.526683Z info  envoy upstream external/envoy/source/extensions/listener_managers/listener_manager/lds_api.cc:82  lds: add/update listener '192.168.119.5_8080' thread=22
2023-11-20T08:03:24.101972Z debug envoy config external/envoy/source/common/config/grpc_subscription_impl.cc:89 gRPC config update took 611 ms! Resources names: 192.168.119.5_8080,192.168.119.8_8443,192.168.119.4_8443,192.168.119.10_8080,192.168.119.7_8080,192.168.119.7_8443,192.168.119.8_8080,192.168.119.3_8080,192.168.119.6_8080,192.168.119.3_8443,192.168.119.9_8443,192.168.119.9_8080,192.168.119.10_8443,192.168.119.5_8443,192.168.119.6_8443,192.168.119.4_8080  thread=22
2023-11-20T08:04:08.496535Z debug envoy config external/envoy/source/extensions/listener_managers/listener_manager/listener_impl.cc:968 removing draining filter chains from listener 192.168.119.5_8080: name=192.168.119.5_8080, hash=10684809757490173150, tag=5, address=192.168.119.5:8080 thread=22
2023-11-20T08:04:08.508755Z debug envoy config external/envoy/source/extensions/listener_managers/listener_manager/listener_impl.cc:968 draining filter chains from listener 192.168.119.5_8080 complete: name=192.168.119.5_8080, hash=10684809757490173150, tag=5, address=192.168.119.5:8080 thread=22
2023-11-20T08:04:08.508772Z debug envoy init external/envoy/source/common/init/watcher_impl.cc:31 Listener-local-init-watcher 192.168.119.5_8080 destroyed  thread=22

As per https://github.com/envoyproxy/envoy/issues/26775 The ordering at initialization is specified at https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/init.html. At runtime, there is no relationship between CDS and LDS, they are independent subsystem and do not warm on each other. See also issues such as #5174.

And https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/lds#config-listeners-lds Listeners are effectively constant once created. Thus, when a listener is updated, an entirely new listener is created (if the listener’s address is unchanged, the new one uses the same listen socket). This listener goes through the same warming process described above for a newly added listener.

So there could be a time gap between the listener warming and draining, especially when the mesh size is big with hundreds of routes/clusters and even bigger EDS(for the production case) Screenshot 2023-11-23 at 14 28 45

2023-11-20T08:03:23.523895Z added warming listener 192.168.119.5_8080,
2023-11-20T08:03:23.526576Z started draining listener 192.168.119.5_8080,

The new connections comes to the Envoy in between will hit the warming listener and since the RDS/CDS/LDS are initialized independently with LDS, it will result to 503 no healthy upstream spike.

The gateway xds config size is much smaller than production so the time gap here is less than 1s, but in production it takes longer and it took few minutes(~3mins)to drain for 200 listeners, and will also take longer to warm up the new listeners. Screenshot 2023-11-23 at 15 23 27

Expected:

Ideally lds change should not cause data plane impact if it can make sure the warming listeners are ready before draining the listener.

Version

istio 1.17
k8s 1.23

Additional Information

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 16 (14 by maintainers)

Most upvoted comments

Previously I was thinking the problem is in LDS but seems not. Listener update will trigger listener rebuild and add new listeners with warming state, meanwhile it will wait for RDS to be ready and only after RDS ready the listener will be active. This can explain why no 404 response code with NR(no route) since RDS is ready. However it doesn’t guarantee CDS/EDS are ready since they are initialized independently. But for this case CDS should be ready otherwise the response flag will be NC( no cluster), and 503 no healthy upstream means something is wrong with the EDS.

The problem here is EDS, there was istiod rollout but no ingressgateway rollout.

For the ingressgateway pods got connection drop, they connected to a BAD istiod pod which pushed 321 empty clusters to the ingressgateway:

2023-11-15 08:14:30.001 | 2023-11-15T00:14:30.876928Z	info	ads	EDS: PUSH request for node:istio-ingressgateway-789bc75c9b-6thwx.istio-production resources:321 size:25.7kB empty:321 cached:0/321

While meantime there was a good istiod after rollout and in the initial push the empty cluster number is just 17:

2023-11-15 08:15:26.001 | 2023-11-15T00:15:26.678536Z	info	ads	EDS: PUSH request for node:istio-ingressgateway-789bc75c9b-k4jpz.istio-production resources:321 size:2.3MB empty:17 cached:0/321

For the bad istiod, discovery took aboout 5 mins to initialize:

2023-11-15 08:08:59.000 
2023-11-15T00:08:59.420010Z info    multicluster remote secrets controller cache synced in 32.043631ms
2023-11-15 08:09:39.000 
2023-11-15T00:09:39.489739Z info    kube    kube controller for Kubernetes synced after 40.101600165s
2023-11-15 08:13:58.001 
2023-11-15T00:13:58.534439Z info    kube    Pilot K8S CRD controller synced in 4m59.146207749s  controller=crd-controller
2023-11-15 08:13:58.001 
2023-11-15T00:13:58.590071Z info    Waiting for caches to be synced
2023-11-15 08:13:58.001 
2023-11-15T00:13:58.590099Z info    All controller caches have been synced up in 29.12µs
2023-11-15 08:14:00.001 
2023-11-15T00:14:00.023396Z info    ads All caches have been synced up in 5m1.111042773s, marking server ready

While for the good istiod it just took 50seconds to initialize:

2023-11-15 08:14:06.000 
2023-11-15T00:14:06.459821Z info    multicluster remote secrets controller cache synced in 14.530202ms
2023-11-15 08:14:06.000 
2023-11-15T00:14:06.674221Z info    kube    Pilot K8S CRD controller synced in 228.607788ms controller=crd-controller
2023-11-15 08:14:51.000 
2023-11-15T00:14:51.547500Z info    Waiting for caches to be synced
2023-11-15 08:14:52.000 
2023-11-15T00:14:52.764738Z info    kube    kube controller for Kubernetes synced after 46.319102979s
2023-11-15 08:14:52.000 
2023-11-15T00:14:52.778233Z info    All controller caches have been synced up in 1.230737187s
2023-11-15 08:14:56.001 
2023-11-15T00:14:56.334701Z info    ads All caches have been synced up in 50.417063025s, marking server ready

Meanwhile there was apiserver rollout which could cause some churn, but it seems there was also eds bug here? Why all the clusters eds built were empty? cc @hzxuzhonghu @howardjohn

OK. the CDS and EDS should be ready before warming. because they are pushed to envoy before listener. So looks weired

@hzxuzhonghu I think similar issue here https://github.com/envoyproxy/envoy/issues/26775, my uderstading is that LDS warming doesn’t guarantee CDS/EDS is ready.