envoy: Memory Leak After 19->21 Upgrade

We’re using Envoy as our edge proxy at Railway (20k routes, virtual hosts, cluster; 5k secrets) and started running into an issue where heap grows uncontrollably. There was already some leaking, but it increased dramatically after switching from Dockerized Envoy 1.19 to Linux Brew Envoy 1.21. We’re currently restarting each of our nodes every few hours to get around this (as noticeable on the posted graphs below).
My question is, does anything jump out immediately as to why this might be happening, and what is the best way to go about debugging this issue if not?
3 VMs, each running af50070ee60866874b0a9383daf9364e884ded22/1.21.1/Modified/RELEASE/BoringSSL on Debian
Envoy’s are behind GCP external network LB (TCP passthrough)
HTTPS listener, cluster, etc are updated every ~10 seconds
Heap size doesn’t seem to be correlated to num active connections, num draining listeners, or anything else I could find in the stats
I removed one of the instances from the LB (but still connected to the go-control-plane) and the leak stopped
We have an Overload Manager configured to shrink heap around 70GB (doesn’t seem to do anything) and stop accepting connections at 100GB. If we let this run, it simply reaches 100GB and stops accepting.
Let me know if you need any additional info.
📊 Here's also a dump of /stats from one of the nodes:
cluster_manager.cds.version_text: "snapshot-2022-04-12T17:59:50Z-536"
control_plane.identifier: ""
http.http.rds.http_route_config.version_text: "snapshot-2022-04-12T17:59:50Z-378"
http.https.rds.https_route_config.version_text: "snapshot-2022-04-12T17:59:50Z-536"
listener_manager.lds.version_text: "snapshot-2022-04-12T17:59:50Z-536"
cluster.http_cluster.assignment_stale: 0
cluster.http_cluster.assignment_timeout_received: 0
cluster.http_cluster.bind_errors: 0
cluster.http_cluster.circuit_breakers.default.cx_open: 0
cluster.http_cluster.circuit_breakers.default.cx_pool_open: 0
cluster.http_cluster.circuit_breakers.default.rq_open: 0
cluster.http_cluster.circuit_breakers.default.rq_pending_open: 0
cluster.http_cluster.circuit_breakers.default.rq_retry_open: 0
cluster.http_cluster.circuit_breakers.high.cx_open: 0
cluster.http_cluster.circuit_breakers.high.cx_pool_open: 0
cluster.http_cluster.circuit_breakers.high.rq_open: 0
cluster.http_cluster.circuit_breakers.high.rq_pending_open: 0
cluster.http_cluster.circuit_breakers.high.rq_retry_open: 0
cluster.http_cluster.default.total_match_count: 1
cluster.http_cluster.external.upstream_rq_404: 310
cluster.http_cluster.external.upstream_rq_4xx: 310
cluster.http_cluster.external.upstream_rq_completed: 310
cluster.http_cluster.http1.dropped_headers_with_underscores: 0
cluster.http_cluster.http1.metadata_not_supported_error: 0
cluster.http_cluster.http1.requests_rejected_with_underscores_in_headers: 0
cluster.http_cluster.http1.response_flood: 0
cluster.http_cluster.lb_healthy_panic: 0
cluster.http_cluster.lb_local_cluster_not_ok: 0
cluster.http_cluster.lb_recalculate_zone_structures: 0
cluster.http_cluster.lb_subsets_active: 0
cluster.http_cluster.lb_subsets_created: 0
cluster.http_cluster.lb_subsets_fallback: 0
cluster.http_cluster.lb_subsets_fallback_panic: 0
cluster.http_cluster.lb_subsets_removed: 0
cluster.http_cluster.lb_subsets_selected: 0
cluster.http_cluster.lb_zone_cluster_too_small: 0
cluster.http_cluster.lb_zone_no_capacity_left: 0
cluster.http_cluster.lb_zone_number_differs: 0
cluster.http_cluster.lb_zone_routing_all_directly: 0
cluster.http_cluster.lb_zone_routing_cross_zone: 0
cluster.http_cluster.lb_zone_routing_sampled: 0
cluster.http_cluster.max_host_weight: 0
cluster.http_cluster.membership_change: 1
cluster.http_cluster.membership_degraded: 0
cluster.http_cluster.membership_excluded: 0
cluster.http_cluster.membership_healthy: 1
cluster.http_cluster.membership_total: 1
cluster.http_cluster.original_dst_host_invalid: 0
cluster.http_cluster.retry_or_shadow_abandoned: 0
cluster.http_cluster.update_attempt: 0
cluster.http_cluster.update_empty: 0
cluster.http_cluster.update_failure: 0
cluster.http_cluster.update_no_rebuild: 0
cluster.http_cluster.update_success: 0
cluster.http_cluster.upstream_cx_active: 0
cluster.http_cluster.upstream_cx_close_notify: 0
cluster.http_cluster.upstream_cx_connect_attempts_exceeded: 0
cluster.http_cluster.upstream_cx_connect_fail: 0
cluster.http_cluster.upstream_cx_connect_timeout: 0
cluster.http_cluster.upstream_cx_destroy: 306
cluster.http_cluster.upstream_cx_destroy_local: 0
cluster.http_cluster.upstream_cx_destroy_local_with_active_rq: 0
cluster.http_cluster.upstream_cx_destroy_remote: 306
cluster.http_cluster.upstream_cx_destroy_remote_with_active_rq: 0
cluster.http_cluster.upstream_cx_destroy_with_active_rq: 0
cluster.http_cluster.upstream_cx_http1_total: 306
cluster.http_cluster.upstream_cx_http2_total: 0
cluster.http_cluster.upstream_cx_http3_total: 0
cluster.http_cluster.upstream_cx_idle_timeout: 0
cluster.http_cluster.upstream_cx_max_duration_reached: 0
cluster.http_cluster.upstream_cx_max_requests: 0
cluster.http_cluster.upstream_cx_none_healthy: 0
cluster.http_cluster.upstream_cx_overflow: 0
cluster.http_cluster.upstream_cx_pool_overflow: 0
cluster.http_cluster.upstream_cx_protocol_error: 0
cluster.http_cluster.upstream_cx_rx_bytes_buffered: 0
cluster.http_cluster.upstream_cx_rx_bytes_total: 54560
cluster.http_cluster.upstream_cx_total: 306
cluster.http_cluster.upstream_cx_tx_bytes_buffered: 0
cluster.http_cluster.upstream_cx_tx_bytes_total: 148563
cluster.http_cluster.upstream_flow_control_backed_up_total: 0
cluster.http_cluster.upstream_flow_control_drained_total: 0
cluster.http_cluster.upstream_flow_control_paused_reading_total: 0
cluster.http_cluster.upstream_flow_control_resumed_reading_total: 0
cluster.http_cluster.upstream_internal_redirect_failed_total: 0
cluster.http_cluster.upstream_internal_redirect_succeeded_total: 0
cluster.http_cluster.upstream_rq_404: 310
cluster.http_cluster.upstream_rq_4xx: 310
cluster.http_cluster.upstream_rq_active: 0
cluster.http_cluster.upstream_rq_cancelled: 0
cluster.http_cluster.upstream_rq_completed: 310
cluster.http_cluster.upstream_rq_maintenance_mode: 0
cluster.http_cluster.upstream_rq_max_duration_reached: 0
cluster.http_cluster.upstream_rq_pending_active: 0
cluster.http_cluster.upstream_rq_pending_failure_eject: 0
cluster.http_cluster.upstream_rq_pending_overflow: 0
cluster.http_cluster.upstream_rq_pending_total: 306
cluster.http_cluster.upstream_rq_per_try_idle_timeout: 0
cluster.http_cluster.upstream_rq_per_try_timeout: 0
cluster.http_cluster.upstream_rq_retry: 0
cluster.http_cluster.upstream_rq_retry_backoff_exponential: 0
cluster.http_cluster.upstream_rq_retry_backoff_ratelimited: 0
cluster.http_cluster.upstream_rq_retry_limit_exceeded: 0
cluster.http_cluster.upstream_rq_retry_overflow: 0
cluster.http_cluster.upstream_rq_retry_success: 0
cluster.http_cluster.upstream_rq_rx_reset: 0
cluster.http_cluster.upstream_rq_timeout: 0
cluster.http_cluster.upstream_rq_total: 310
cluster.http_cluster.upstream_rq_tx_reset: 0
cluster.http_cluster.version: 0
cluster.railway_xds_cluster.assignment_stale: 0
cluster.railway_xds_cluster.assignment_timeout_received: 0
cluster.railway_xds_cluster.bind_errors: 0
cluster.railway_xds_cluster.circuit_breakers.default.cx_open: 0
cluster.railway_xds_cluster.circuit_breakers.default.cx_pool_open: 0
cluster.railway_xds_cluster.circuit_breakers.default.rq_open: 0
cluster.railway_xds_cluster.circuit_breakers.default.rq_pending_open: 0
cluster.railway_xds_cluster.circuit_breakers.default.rq_retry_open: 0
cluster.railway_xds_cluster.circuit_breakers.high.cx_open: 0
cluster.railway_xds_cluster.circuit_breakers.high.cx_pool_open: 0
cluster.railway_xds_cluster.circuit_breakers.high.rq_open: 0
cluster.railway_xds_cluster.circuit_breakers.high.rq_pending_open: 0
cluster.railway_xds_cluster.circuit_breakers.high.rq_retry_open: 0
cluster.railway_xds_cluster.default.total_match_count: 1
cluster.railway_xds_cluster.http2.dropped_headers_with_underscores: 0
cluster.railway_xds_cluster.http2.header_overflow: 0
cluster.railway_xds_cluster.http2.headers_cb_no_stream: 0
cluster.railway_xds_cluster.http2.inbound_empty_frames_flood: 0
cluster.railway_xds_cluster.http2.inbound_priority_frames_flood: 0
cluster.railway_xds_cluster.http2.inbound_window_update_frames_flood: 0
cluster.railway_xds_cluster.http2.keepalive_timeout: 0
cluster.railway_xds_cluster.http2.metadata_empty_frames: 0
cluster.railway_xds_cluster.http2.outbound_control_flood: 0
cluster.railway_xds_cluster.http2.outbound_flood: 0
cluster.railway_xds_cluster.http2.pending_send_bytes: 0
cluster.railway_xds_cluster.http2.requests_rejected_with_underscores_in_headers: 0
cluster.railway_xds_cluster.http2.rx_messaging_error: 0
cluster.railway_xds_cluster.http2.rx_reset: 0
cluster.railway_xds_cluster.http2.stream_refused_errors: 0
cluster.railway_xds_cluster.http2.streams_active: 1
cluster.railway_xds_cluster.http2.trailers: 0
cluster.railway_xds_cluster.http2.tx_flush_timeout: 0
cluster.railway_xds_cluster.http2.tx_reset: 0
cluster.railway_xds_cluster.internal.upstream_rq_200: 1
cluster.railway_xds_cluster.internal.upstream_rq_2xx: 1
cluster.railway_xds_cluster.internal.upstream_rq_completed: 1
cluster.railway_xds_cluster.lb_healthy_panic: 0
cluster.railway_xds_cluster.lb_local_cluster_not_ok: 0
cluster.railway_xds_cluster.lb_recalculate_zone_structures: 0
cluster.railway_xds_cluster.lb_subsets_active: 0
cluster.railway_xds_cluster.lb_subsets_created: 0
cluster.railway_xds_cluster.lb_subsets_fallback: 0
cluster.railway_xds_cluster.lb_subsets_fallback_panic: 0
cluster.railway_xds_cluster.lb_subsets_removed: 0
cluster.railway_xds_cluster.lb_subsets_selected: 0
cluster.railway_xds_cluster.lb_zone_cluster_too_small: 0
cluster.railway_xds_cluster.lb_zone_no_capacity_left: 0
cluster.railway_xds_cluster.lb_zone_number_differs: 0
cluster.railway_xds_cluster.lb_zone_routing_all_directly: 0
cluster.railway_xds_cluster.lb_zone_routing_cross_zone: 0
cluster.railway_xds_cluster.lb_zone_routing_sampled: 0
cluster.railway_xds_cluster.max_host_weight: 0
cluster.railway_xds_cluster.membership_change: 1
cluster.railway_xds_cluster.membership_degraded: 0
cluster.railway_xds_cluster.membership_excluded: 0
cluster.railway_xds_cluster.membership_healthy: 1
cluster.railway_xds_cluster.membership_total: 1
cluster.railway_xds_cluster.original_dst_host_invalid: 0
cluster.railway_xds_cluster.retry_or_shadow_abandoned: 0
cluster.railway_xds_cluster.update_attempt: 0
cluster.railway_xds_cluster.update_empty: 0
cluster.railway_xds_cluster.update_failure: 0
cluster.railway_xds_cluster.update_no_rebuild: 0
cluster.railway_xds_cluster.update_success: 0
cluster.railway_xds_cluster.upstream_cx_active: 1
cluster.railway_xds_cluster.upstream_cx_close_notify: 0
cluster.railway_xds_cluster.upstream_cx_connect_attempts_exceeded: 0
cluster.railway_xds_cluster.upstream_cx_connect_fail: 0
cluster.railway_xds_cluster.upstream_cx_connect_timeout: 0
cluster.railway_xds_cluster.upstream_cx_destroy: 0
cluster.railway_xds_cluster.upstream_cx_destroy_local: 0
cluster.railway_xds_cluster.upstream_cx_destroy_local_with_active_rq: 0
cluster.railway_xds_cluster.upstream_cx_destroy_remote: 0
cluster.railway_xds_cluster.upstream_cx_destroy_remote_with_active_rq: 0
cluster.railway_xds_cluster.upstream_cx_destroy_with_active_rq: 0
cluster.railway_xds_cluster.upstream_cx_http1_total: 0
cluster.railway_xds_cluster.upstream_cx_http2_total: 1
cluster.railway_xds_cluster.upstream_cx_http3_total: 0
cluster.railway_xds_cluster.upstream_cx_idle_timeout: 0
cluster.railway_xds_cluster.upstream_cx_max_duration_reached: 0
cluster.railway_xds_cluster.upstream_cx_max_requests: 0
cluster.railway_xds_cluster.upstream_cx_none_healthy: 0
cluster.railway_xds_cluster.upstream_cx_overflow: 0
cluster.railway_xds_cluster.upstream_cx_pool_overflow: 0
cluster.railway_xds_cluster.upstream_cx_protocol_error: 0
cluster.railway_xds_cluster.upstream_cx_rx_bytes_buffered: 30
cluster.railway_xds_cluster.upstream_cx_rx_bytes_total: 2296019362
cluster.railway_xds_cluster.upstream_cx_total: 1
cluster.railway_xds_cluster.upstream_cx_tx_bytes_buffered: 0
cluster.railway_xds_cluster.upstream_cx_tx_bytes_total: 51817804
cluster.railway_xds_cluster.upstream_flow_control_backed_up_total: 0
cluster.railway_xds_cluster.upstream_flow_control_drained_total: 0
cluster.railway_xds_cluster.upstream_flow_control_paused_reading_total: 0
cluster.railway_xds_cluster.upstream_flow_control_resumed_reading_total: 0
cluster.railway_xds_cluster.upstream_internal_redirect_failed_total: 0
cluster.railway_xds_cluster.upstream_internal_redirect_succeeded_total: 0
cluster.railway_xds_cluster.upstream_rq_200: 1
cluster.railway_xds_cluster.upstream_rq_2xx: 1
cluster.railway_xds_cluster.upstream_rq_active: 1
cluster.railway_xds_cluster.upstream_rq_cancelled: 0
cluster.railway_xds_cluster.upstream_rq_completed: 1
cluster.railway_xds_cluster.upstream_rq_maintenance_mode: 0
cluster.railway_xds_cluster.upstream_rq_max_duration_reached: 0
cluster.railway_xds_cluster.upstream_rq_pending_active: 0
cluster.railway_xds_cluster.upstream_rq_pending_failure_eject: 0
cluster.railway_xds_cluster.upstream_rq_pending_overflow: 0
cluster.railway_xds_cluster.upstream_rq_pending_total: 1
cluster.railway_xds_cluster.upstream_rq_per_try_idle_timeout: 0
cluster.railway_xds_cluster.upstream_rq_per_try_timeout: 0
cluster.railway_xds_cluster.upstream_rq_retry: 0
cluster.railway_xds_cluster.upstream_rq_retry_backoff_exponential: 0
cluster.railway_xds_cluster.upstream_rq_retry_backoff_ratelimited: 0
cluster.railway_xds_cluster.upstream_rq_retry_limit_exceeded: 0
cluster.railway_xds_cluster.upstream_rq_retry_overflow: 0
cluster.railway_xds_cluster.upstream_rq_retry_success: 0
cluster.railway_xds_cluster.upstream_rq_rx_reset: 0
cluster.railway_xds_cluster.upstream_rq_timeout: 0
cluster.railway_xds_cluster.upstream_rq_total: 1
cluster.railway_xds_cluster.upstream_rq_tx_reset: 0
cluster.railway_xds_cluster.version: 0
cluster_manager.active_clusters: 19947
cluster_manager.cds.init_fetch_timeout: 0
cluster_manager.cds.update_attempt: 120
cluster_manager.cds.update_failure: 0
cluster_manager.cds.update_rejected: 0
cluster_manager.cds.update_success: 119
cluster_manager.cds.update_time: 1649802536288
cluster_manager.cds.version: 2486829869471955647
cluster_manager.cluster_added: 20028
cluster_manager.cluster_modified: 0
cluster_manager.cluster_removed: 81
cluster_manager.cluster_updated: 0
cluster_manager.cluster_updated_via_merge: 0
cluster_manager.update_merge_cancelled: 0
cluster_manager.update_out_of_merge_window: 0
cluster_manager.warming_clusters: 0
control_plane.connected_state: 1
control_plane.pending_requests: 0
control_plane.rate_limit_enforced: 0
envoy.overload_actions.reset_high_memory_stream.count: 0
filesystem.flushed_by_timer: 0
filesystem.reopen_failed: 0
filesystem.write_buffered: 0
filesystem.write_completed: 0
filesystem.write_failed: 0
filesystem.write_total_buffered: 0
http.admin.downstream_cx_active: 1
http.admin.downstream_cx_delayed_close_timeout: 0
http.admin.downstream_cx_destroy: 713
http.admin.downstream_cx_destroy_active_rq: 0
http.admin.downstream_cx_destroy_local: 0
http.admin.downstream_cx_destroy_local_active_rq: 0
http.admin.downstream_cx_destroy_remote: 713
http.admin.downstream_cx_destroy_remote_active_rq: 0
http.admin.downstream_cx_drain_close: 0
http.admin.downstream_cx_http1_active: 1
http.admin.downstream_cx_http1_total: 714
http.admin.downstream_cx_http2_active: 0
http.admin.downstream_cx_http2_total: 0
http.admin.downstream_cx_http3_active: 0
http.admin.downstream_cx_http3_total: 0
http.admin.downstream_cx_idle_timeout: 0
http.admin.downstream_cx_max_duration_reached: 0
http.admin.downstream_cx_max_requests_reached: 0
http.admin.downstream_cx_overload_disable_keepalive: 0
http.admin.downstream_cx_protocol_error: 0
http.admin.downstream_cx_rx_bytes_buffered: 84
http.admin.downstream_cx_rx_bytes_total: 111602
http.admin.downstream_cx_ssl_active: 0
http.admin.downstream_cx_ssl_total: 0
http.admin.downstream_cx_total: 714
http.admin.downstream_cx_tx_bytes_buffered: 0
http.admin.downstream_cx_tx_bytes_total: 22759646
http.admin.downstream_cx_upgrades_active: 0
http.admin.downstream_cx_upgrades_total: 0
http.admin.downstream_flow_control_paused_reading_total: 0
http.admin.downstream_flow_control_resumed_reading_total: 0
http.admin.downstream_rq_1xx: 0
http.admin.downstream_rq_2xx: 713
http.admin.downstream_rq_3xx: 0
http.admin.downstream_rq_4xx: 0
http.admin.downstream_rq_5xx: 0
http.admin.downstream_rq_active: 1
http.admin.downstream_rq_completed: 713
http.admin.downstream_rq_failed_path_normalization: 0
http.admin.downstream_rq_header_timeout: 0
http.admin.downstream_rq_http1_total: 714
http.admin.downstream_rq_http2_total: 0
http.admin.downstream_rq_http3_total: 0
http.admin.downstream_rq_idle_timeout: 0
http.admin.downstream_rq_max_duration_reached: 0
http.admin.downstream_rq_non_relative_path: 0
http.admin.downstream_rq_overload_close: 0
http.admin.downstream_rq_redirected_with_normalized_path: 0
http.admin.downstream_rq_rejected_via_ip_detection: 0
http.admin.downstream_rq_response_before_rq_complete: 0
http.admin.downstream_rq_rx_reset: 0
http.admin.downstream_rq_timeout: 0
http.admin.downstream_rq_too_large: 0
http.admin.downstream_rq_total: 714
http.admin.downstream_rq_tx_reset: 0
http.admin.downstream_rq_ws_on_non_ws_route: 0
http.admin.rs_too_large: 0
http.async-client.no_cluster: 0
http.async-client.no_route: 0
http.async-client.passthrough_internal_redirect_bad_location: 0
http.async-client.passthrough_internal_redirect_no_route: 0
http.async-client.passthrough_internal_redirect_predicate: 0
http.async-client.passthrough_internal_redirect_too_many_redirects: 0
http.async-client.passthrough_internal_redirect_unsafe_scheme: 0
http.async-client.rq_direct_response: 0
http.async-client.rq_redirect: 0
http.async-client.rq_reset_after_downstream_response_started: 0
http.async-client.rq_total: 1
http.http.downstream_cx_active: 1224
http.http.downstream_cx_delayed_close_timeout: 339
http.http.downstream_cx_destroy: 1399
http.http.downstream_cx_destroy_active_rq: 0
http.http.downstream_cx_destroy_local: 339
http.http.downstream_cx_destroy_local_active_rq: 0
http.http.downstream_cx_destroy_remote: 1060
http.http.downstream_cx_destroy_remote_active_rq: 0
http.http.downstream_cx_drain_close: 0
http.http.downstream_cx_http1_active: 1224
http.http.downstream_cx_http1_total: 2623
http.http.downstream_cx_http2_active: 0
http.http.downstream_cx_http2_total: 0
http.http.downstream_cx_http3_active: 0
http.http.downstream_cx_http3_total: 0
http.http.downstream_cx_idle_timeout: 0
http.http.downstream_cx_max_duration_reached: 0
http.http.downstream_cx_max_requests_reached: 0
http.http.downstream_cx_overload_disable_keepalive: 0
http.http.downstream_cx_protocol_error: 2
http.http.downstream_cx_rx_bytes_buffered: 694759
http.http.downstream_cx_rx_bytes_total: 1675285917
http.http.downstream_cx_ssl_active: 0
http.http.downstream_cx_ssl_total: 0
http.http.downstream_cx_total: 2623
http.http.downstream_cx_tx_bytes_buffered: 0
http.http.downstream_cx_tx_bytes_total: 1172707534
http.http.downstream_cx_upgrades_active: 0
http.http.downstream_cx_upgrades_total: 0
http.http.downstream_flow_control_paused_reading_total: 0
http.http.downstream_flow_control_resumed_reading_total: 0
http.http.downstream_rq_1xx: 0
http.http.downstream_rq_2xx: 0
http.http.downstream_rq_3xx: 2956423
http.http.downstream_rq_4xx: 315
http.http.downstream_rq_5xx: 0
http.http.downstream_rq_active: 0
http.http.downstream_rq_completed: 2956738
http.http.downstream_rq_failed_path_normalization: 0
http.http.downstream_rq_header_timeout: 0
http.http.downstream_rq_http1_total: 2956738
http.http.downstream_rq_http2_total: 0
http.http.downstream_rq_http3_total: 0
http.http.downstream_rq_idle_timeout: 0
http.http.downstream_rq_max_duration_reached: 0
http.http.downstream_rq_non_relative_path: 0
http.http.downstream_rq_overload_close: 0
http.http.downstream_rq_redirected_with_normalized_path: 0
http.http.downstream_rq_rejected_via_ip_detection: 0
http.http.downstream_rq_response_before_rq_complete: 106
http.http.downstream_rq_rx_reset: 106
http.http.downstream_rq_timeout: 0
http.http.downstream_rq_too_large: 0
http.http.downstream_rq_total: 2956738
http.http.downstream_rq_tx_reset: 0
http.http.downstream_rq_ws_on_non_ws_route: 0
http.http.no_cluster: 0
http.http.no_route: 2
http.http.passthrough_internal_redirect_bad_location: 0
http.http.passthrough_internal_redirect_no_route: 0
http.http.passthrough_internal_redirect_predicate: 0
http.http.passthrough_internal_redirect_too_many_redirects: 0
http.http.passthrough_internal_redirect_unsafe_scheme: 0
http.http.rds.http_route_config.config_reload: 1
http.http.rds.http_route_config.config_reload_time_ms: 1649797937577
http.http.rds.http_route_config.init_fetch_timeout: 1
http.http.rds.http_route_config.update_attempt: 3
http.http.rds.http_route_config.update_empty: 0
http.http.rds.http_route_config.update_failure: 0
http.http.rds.http_route_config.update_rejected: 0
http.http.rds.http_route_config.update_success: 1
http.http.rds.http_route_config.update_time: 1649797937577
http.http.rds.http_route_config.version: 10194100587458576616
http.http.rq_direct_response: 2956423
http.http.rq_redirect: 0
http.http.rq_reset_after_downstream_response_started: 0
http.http.rq_total: 2956735
http.http.rs_too_large: 0
http.http.tracing.client_enabled: 0
http.http.tracing.health_check: 0
http.http.tracing.not_traceable: 0
http.http.tracing.random_sampling: 0
http.http.tracing.service_forced: 0
http.https.downstream_cx_active: 5643
http.https.downstream_cx_delayed_close_timeout: 3542
http.https.downstream_cx_destroy: 175304
http.https.downstream_cx_destroy_active_rq: 5329
http.https.downstream_cx_destroy_local: 9729
http.https.downstream_cx_destroy_local_active_rq: 0
http.https.downstream_cx_destroy_remote: 165575
http.https.downstream_cx_destroy_remote_active_rq: 5329
http.https.downstream_cx_drain_close: 40644
http.https.downstream_cx_http1_active: 990
http.https.downstream_cx_http1_total: 123681
http.https.downstream_cx_http2_active: 4624
http.https.downstream_cx_http2_total: 53927
http.https.downstream_cx_http3_active: 0
http.https.downstream_cx_http3_total: 0
http.https.downstream_cx_idle_timeout: 18
http.https.downstream_cx_max_duration_reached: 0
http.https.downstream_cx_max_requests_reached: 0
http.https.downstream_cx_overload_disable_keepalive: 0
http.https.downstream_cx_protocol_error: 12
http.https.downstream_cx_rx_bytes_buffered: 1189909
http.https.downstream_cx_rx_bytes_total: 3785524095
http.https.downstream_cx_ssl_active: 5643
http.https.downstream_cx_ssl_total: 180947
http.https.downstream_cx_total: 180947
http.https.downstream_cx_tx_bytes_buffered: 4930064
http.https.downstream_cx_tx_bytes_total: 11391226786
http.https.downstream_cx_upgrades_active: 933
http.https.downstream_cx_upgrades_total: 10883
http.https.downstream_flow_control_paused_reading_total: 28
http.https.downstream_flow_control_resumed_reading_total: 28
http.https.downstream_rq_1xx: 6351
http.https.downstream_rq_2xx: 906891
http.https.downstream_rq_3xx: 13561
http.https.downstream_rq_4xx: 103747
http.https.downstream_rq_5xx: 11552
http.https.downstream_rq_active: 943
http.https.downstream_rq_completed: 1042102
http.https.downstream_rq_failed_path_normalization: 0
http.https.downstream_rq_header_timeout: 0
http.https.downstream_rq_http1_total: 325770
http.https.downstream_rq_http2_total: 717528
http.https.downstream_rq_http3_total: 0
http.https.downstream_rq_idle_timeout: 42
http.https.downstream_rq_max_duration_reached: 0
http.https.downstream_rq_non_relative_path: 0
http.https.downstream_rq_overload_close: 0
http.https.downstream_rq_redirected_with_normalized_path: 0
http.https.downstream_rq_rejected_via_ip_detection: 0
http.https.downstream_rq_response_before_rq_complete: 15012
http.https.downstream_rq_rx_reset: 15473
http.https.downstream_rq_timeout: 0
http.https.downstream_rq_too_large: 0
http.https.downstream_rq_total: 1043298
http.https.downstream_rq_tx_reset: 1390
http.https.downstream_rq_ws_on_non_ws_route: 0
http.https.no_cluster: 10
http.https.no_route: 100
http.https.passthrough_internal_redirect_bad_location: 0
http.https.passthrough_internal_redirect_no_route: 0
http.https.passthrough_internal_redirect_predicate: 0
http.https.passthrough_internal_redirect_too_many_redirects: 0
http.https.passthrough_internal_redirect_unsafe_scheme: 0
http.https.rds.https_route_config.config_reload: 142
http.https.rds.https_route_config.config_reload_time_ms: 1649802538356
http.https.rds.https_route_config.init_fetch_timeout: 1
http.https.rds.https_route_config.update_attempt: 144
http.https.rds.https_route_config.update_empty: 0
http.https.rds.https_route_config.update_failure: 0
http.https.rds.https_route_config.update_rejected: 0
http.https.rds.https_route_config.update_success: 142
http.https.rds.https_route_config.update_time: 1649802538357
http.https.rds.https_route_config.version: 2486829869471955647
http.https.rq_direct_response: 28262
http.https.rq_redirect: 0
http.https.rq_reset_after_downstream_response_started: 0
http.https.rq_total: 1042353
http.https.rs_too_large: 0
http.https.tracing.client_enabled: 0
http.https.tracing.health_check: 0
http.https.tracing.not_traceable: 0
http.https.tracing.random_sampling: 0
http.https.tracing.service_forced: 0
http.https.user_agent.android.downstream_cx_destroy_remote_active_rq: 1
http.https.user_agent.android.downstream_cx_total: 41976
http.https.user_agent.android.downstream_rq_total: 41976
http.https.user_agent.ios.downstream_cx_destroy_remote_active_rq: 0
http.https.user_agent.ios.downstream_cx_total: 245
http.https.user_agent.ios.downstream_rq_total: 245
http1.dropped_headers_with_underscores: 0
http1.metadata_not_supported_error: 0
http1.requests_rejected_with_underscores_in_headers: 0
http1.response_flood: 0
http2.dropped_headers_with_underscores: 0
http2.header_overflow: 0
http2.headers_cb_no_stream: 0
http2.inbound_empty_frames_flood: 0
http2.inbound_priority_frames_flood: 0
http2.inbound_window_update_frames_flood: 0
http2.keepalive_timeout: 0
http2.metadata_empty_frames: 0
http2.outbound_control_flood: 0
http2.outbound_flood: 0
http2.pending_send_bytes: 0
http2.requests_rejected_with_underscores_in_headers: 0
http2.rx_messaging_error: 0
http2.rx_reset: 87
http2.stream_refused_errors: 0
http2.streams_active: 7
http2.trailers: 0
http2.tx_flush_timeout: 0
http2.tx_reset: 2667
http_inspector.http10_found: 1
http_inspector.http11_found: 2620
http_inspector.http2_found: 0
http_inspector.http_not_found: 2
http_inspector.read_error: 3
listener.0.0.0.0_443.downstream_cx_active: 5643
listener.0.0.0.0_443.downstream_cx_destroy: 175304
listener.0.0.0.0_443.downstream_cx_overflow: 0
listener.0.0.0.0_443.downstream_cx_overload_reject: 0
listener.0.0.0.0_443.downstream_cx_total: 180947
listener.0.0.0.0_443.downstream_cx_transport_socket_connect_timeout: 0
listener.0.0.0.0_443.downstream_global_cx_overflow: 0
listener.0.0.0.0_443.downstream_pre_cx_active: 3
listener.0.0.0.0_443.downstream_pre_cx_timeout: 65
listener.0.0.0.0_443.http.https.downstream_rq_1xx: 6351
listener.0.0.0.0_443.http.https.downstream_rq_2xx: 906891
listener.0.0.0.0_443.http.https.downstream_rq_3xx: 13561
listener.0.0.0.0_443.http.https.downstream_rq_4xx: 103747
listener.0.0.0.0_443.http.https.downstream_rq_5xx: 11552
listener.0.0.0.0_443.http.https.downstream_rq_completed: 1042102
listener.0.0.0.0_443.no_filter_chain_match: 0
listener.0.0.0.0_443.server_ssl_socket_factory.downstream_context_secrets_not_ready: 0
listener.0.0.0.0_443.server_ssl_socket_factory.ssl_context_update_by_sds: 1986
listener.0.0.0.0_443.server_ssl_socket_factory.upstream_context_secrets_not_ready: 0
listener.0.0.0.0_443.ssl.ciphers.AES128-SHA: 2
listener.0.0.0.0_443.ssl.ciphers.ECDHE-RSA-AES128-GCM-SHA256: 817
listener.0.0.0.0_443.ssl.ciphers.ECDHE-RSA-AES128-SHA: 972
listener.0.0.0.0_443.ssl.ciphers.ECDHE-RSA-CHACHA20-POLY1305: 47131
listener.0.0.0.0_443.ssl.ciphers.TLS_AES_128_GCM_SHA256: 14968
listener.0.0.0.0_443.ssl.ciphers.TLS_AES_256_GCM_SHA384: 114174
listener.0.0.0.0_443.ssl.ciphers.TLS_CHACHA20_POLY1305_SHA256: 487
listener.0.0.0.0_443.ssl.connection_error: 765
listener.0.0.0.0_443.ssl.curves.P-256: 1607
listener.0.0.0.0_443.ssl.curves.X25519: 176942
listener.0.0.0.0_443.ssl.fail_verify_cert_hash: 0
listener.0.0.0.0_443.ssl.fail_verify_error: 0
listener.0.0.0.0_443.ssl.fail_verify_no_cert: 0
listener.0.0.0.0_443.ssl.fail_verify_san: 0
listener.0.0.0.0_443.ssl.handshake: 178551
listener.0.0.0.0_443.ssl.no_certificate: 178551
listener.0.0.0.0_443.ssl.ocsp_staple_failed: 0
listener.0.0.0.0_443.ssl.ocsp_staple_omitted: 12656
listener.0.0.0.0_443.ssl.ocsp_staple_requests: 12656
listener.0.0.0.0_443.ssl.ocsp_staple_responses: 0
listener.0.0.0.0_443.ssl.session_reused: 29364
listener.0.0.0.0_443.ssl.versions.TLSv1: 31
listener.0.0.0.0_443.ssl.versions.TLSv1.1: 1
listener.0.0.0.0_443.ssl.versions.TLSv1.2: 48890
listener.0.0.0.0_443.ssl.versions.TLSv1.3: 129629
listener.0.0.0.0_443.worker_0.downstream_cx_active: 83
listener.0.0.0.0_443.worker_0.downstream_cx_total: 2859
listener.0.0.0.0_443.worker_1.downstream_cx_active: 90
listener.0.0.0.0_443.worker_1.downstream_cx_total: 2950
listener.0.0.0.0_443.worker_10.downstream_cx_active: 86
listener.0.0.0.0_443.worker_10.downstream_cx_total: 2859
listener.0.0.0.0_443.worker_11.downstream_cx_active: 94
listener.0.0.0.0_443.worker_11.downstream_cx_total: 2841
listener.0.0.0.0_443.worker_12.downstream_cx_active: 88
listener.0.0.0.0_443.worker_12.downstream_cx_total: 2875
listener.0.0.0.0_443.worker_13.downstream_cx_active: 89
listener.0.0.0.0_443.worker_13.downstream_cx_total: 2827
listener.0.0.0.0_443.worker_14.downstream_cx_active: 70
listener.0.0.0.0_443.worker_14.downstream_cx_total: 2893
listener.0.0.0.0_443.worker_15.downstream_cx_active: 87
listener.0.0.0.0_443.worker_15.downstream_cx_total: 2854
listener.0.0.0.0_443.worker_16.downstream_cx_active: 75
listener.0.0.0.0_443.worker_16.downstream_cx_total: 2815
listener.0.0.0.0_443.worker_17.downstream_cx_active: 77
listener.0.0.0.0_443.worker_17.downstream_cx_total: 2873
listener.0.0.0.0_443.worker_18.downstream_cx_active: 77
listener.0.0.0.0_443.worker_18.downstream_cx_total: 2731
listener.0.0.0.0_443.worker_19.downstream_cx_active: 89
listener.0.0.0.0_443.worker_19.downstream_cx_total: 2883
listener.0.0.0.0_443.worker_2.downstream_cx_active: 109
listener.0.0.0.0_443.worker_2.downstream_cx_total: 2756
listener.0.0.0.0_443.worker_20.downstream_cx_active: 80
listener.0.0.0.0_443.worker_20.downstream_cx_total: 2813
listener.0.0.0.0_443.worker_21.downstream_cx_active: 94
listener.0.0.0.0_443.worker_21.downstream_cx_total: 2878
listener.0.0.0.0_443.worker_22.downstream_cx_active: 90
listener.0.0.0.0_443.worker_22.downstream_cx_total: 2796
listener.0.0.0.0_443.worker_23.downstream_cx_active: 76
listener.0.0.0.0_443.worker_23.downstream_cx_total: 2830
listener.0.0.0.0_443.worker_24.downstream_cx_active: 81
listener.0.0.0.0_443.worker_24.downstream_cx_total: 2841
listener.0.0.0.0_443.worker_25.downstream_cx_active: 104
listener.0.0.0.0_443.worker_25.downstream_cx_total: 2885
listener.0.0.0.0_443.worker_26.downstream_cx_active: 85
listener.0.0.0.0_443.worker_26.downstream_cx_total: 2766
listener.0.0.0.0_443.worker_27.downstream_cx_active: 99
listener.0.0.0.0_443.worker_27.downstream_cx_total: 2724
listener.0.0.0.0_443.worker_28.downstream_cx_active: 92
listener.0.0.0.0_443.worker_28.downstream_cx_total: 2980
listener.0.0.0.0_443.worker_29.downstream_cx_active: 100
listener.0.0.0.0_443.worker_29.downstream_cx_total: 2892
listener.0.0.0.0_443.worker_3.downstream_cx_active: 90
listener.0.0.0.0_443.worker_3.downstream_cx_total: 2678
listener.0.0.0.0_443.worker_30.downstream_cx_active: 83
listener.0.0.0.0_443.worker_30.downstream_cx_total: 2785
listener.0.0.0.0_443.worker_31.downstream_cx_active: 81
listener.0.0.0.0_443.worker_31.downstream_cx_total: 2768
listener.0.0.0.0_443.worker_32.downstream_cx_active: 88
listener.0.0.0.0_443.worker_32.downstream_cx_total: 2720
listener.0.0.0.0_443.worker_33.downstream_cx_active: 81
listener.0.0.0.0_443.worker_33.downstream_cx_total: 2876
listener.0.0.0.0_443.worker_34.downstream_cx_active: 74
listener.0.0.0.0_443.worker_34.downstream_cx_total: 2806
listener.0.0.0.0_443.worker_35.downstream_cx_active: 81
listener.0.0.0.0_443.worker_35.downstream_cx_total: 2835
listener.0.0.0.0_443.worker_36.downstream_cx_active: 77
listener.0.0.0.0_443.worker_36.downstream_cx_total: 2822
listener.0.0.0.0_443.worker_37.downstream_cx_active: 95
listener.0.0.0.0_443.worker_37.downstream_cx_total: 2843
listener.0.0.0.0_443.worker_38.downstream_cx_active: 95
listener.0.0.0.0_443.worker_38.downstream_cx_total: 2952
listener.0.0.0.0_443.worker_39.downstream_cx_active: 88
listener.0.0.0.0_443.worker_39.downstream_cx_total: 2679
listener.0.0.0.0_443.worker_4.downstream_cx_active: 100
listener.0.0.0.0_443.worker_4.downstream_cx_total: 2772
listener.0.0.0.0_443.worker_40.downstream_cx_active: 96
listener.0.0.0.0_443.worker_40.downstream_cx_total: 2844
listener.0.0.0.0_443.worker_41.downstream_cx_active: 71
listener.0.0.0.0_443.worker_41.downstream_cx_total: 2707
listener.0.0.0.0_443.worker_42.downstream_cx_active: 93
listener.0.0.0.0_443.worker_42.downstream_cx_total: 2769
listener.0.0.0.0_443.worker_43.downstream_cx_active: 88
listener.0.0.0.0_443.worker_43.downstream_cx_total: 2755
listener.0.0.0.0_443.worker_44.downstream_cx_active: 86
listener.0.0.0.0_443.worker_44.downstream_cx_total: 2808
listener.0.0.0.0_443.worker_45.downstream_cx_active: 90
listener.0.0.0.0_443.worker_45.downstream_cx_total: 2727
listener.0.0.0.0_443.worker_46.downstream_cx_active: 89
listener.0.0.0.0_443.worker_46.downstream_cx_total: 2803
listener.0.0.0.0_443.worker_47.downstream_cx_active: 95
listener.0.0.0.0_443.worker_47.downstream_cx_total: 2880
listener.0.0.0.0_443.worker_48.downstream_cx_active: 97
listener.0.0.0.0_443.worker_48.downstream_cx_total: 2770
listener.0.0.0.0_443.worker_49.downstream_cx_active: 88
listener.0.0.0.0_443.worker_49.downstream_cx_total: 2769
listener.0.0.0.0_443.worker_5.downstream_cx_active: 82
listener.0.0.0.0_443.worker_5.downstream_cx_total: 2887
listener.0.0.0.0_443.worker_50.downstream_cx_active: 90
listener.0.0.0.0_443.worker_50.downstream_cx_total: 2879
listener.0.0.0.0_443.worker_51.downstream_cx_active: 84
listener.0.0.0.0_443.worker_51.downstream_cx_total: 2852
listener.0.0.0.0_443.worker_52.downstream_cx_active: 76
listener.0.0.0.0_443.worker_52.downstream_cx_total: 2912
listener.0.0.0.0_443.worker_53.downstream_cx_active: 85
listener.0.0.0.0_443.worker_53.downstream_cx_total: 2939
listener.0.0.0.0_443.worker_54.downstream_cx_active: 102
listener.0.0.0.0_443.worker_54.downstream_cx_total: 2843
listener.0.0.0.0_443.worker_55.downstream_cx_active: 106
listener.0.0.0.0_443.worker_55.downstream_cx_total: 2727
listener.0.0.0.0_443.worker_56.downstream_cx_active: 95
listener.0.0.0.0_443.worker_56.downstream_cx_total: 2787
listener.0.0.0.0_443.worker_57.downstream_cx_active: 86
listener.0.0.0.0_443.worker_57.downstream_cx_total: 2865
listener.0.0.0.0_443.worker_58.downstream_cx_active: 76
listener.0.0.0.0_443.worker_58.downstream_cx_total: 2922
listener.0.0.0.0_443.worker_59.downstream_cx_active: 100
listener.0.0.0.0_443.worker_59.downstream_cx_total: 2951
listener.0.0.0.0_443.worker_6.downstream_cx_active: 102
listener.0.0.0.0_443.worker_6.downstream_cx_total: 2979
listener.0.0.0.0_443.worker_60.downstream_cx_active: 93
listener.0.0.0.0_443.worker_60.downstream_cx_total: 2860
listener.0.0.0.0_443.worker_61.downstream_cx_active: 70
listener.0.0.0.0_443.worker_61.downstream_cx_total: 2786
listener.0.0.0.0_443.worker_62.downstream_cx_active: 93
listener.0.0.0.0_443.worker_62.downstream_cx_total: 2948
listener.0.0.0.0_443.worker_63.downstream_cx_active: 88
listener.0.0.0.0_443.worker_63.downstream_cx_total: 2785
listener.0.0.0.0_443.worker_7.downstream_cx_active: 99
listener.0.0.0.0_443.worker_7.downstream_cx_total: 2795
listener.0.0.0.0_443.worker_8.downstream_cx_active: 74
listener.0.0.0.0_443.worker_8.downstream_cx_total: 2613
listener.0.0.0.0_443.worker_9.downstream_cx_active: 101
listener.0.0.0.0_443.worker_9.downstream_cx_total: 2828
listener.0.0.0.0_80.downstream_cx_active: 1224
listener.0.0.0.0_80.downstream_cx_destroy: 1399
listener.0.0.0.0_80.downstream_cx_overflow: 0
listener.0.0.0.0_80.downstream_cx_overload_reject: 0
listener.0.0.0.0_80.downstream_cx_total: 2623
listener.0.0.0.0_80.downstream_cx_transport_socket_connect_timeout: 0
listener.0.0.0.0_80.downstream_global_cx_overflow: 0
listener.0.0.0.0_80.downstream_pre_cx_active: 0
listener.0.0.0.0_80.downstream_pre_cx_timeout: 6
listener.0.0.0.0_80.http.http.downstream_rq_1xx: 0
listener.0.0.0.0_80.http.http.downstream_rq_2xx: 0
listener.0.0.0.0_80.http.http.downstream_rq_3xx: 2956422
listener.0.0.0.0_80.http.http.downstream_rq_4xx: 315
listener.0.0.0.0_80.http.http.downstream_rq_5xx: 0
listener.0.0.0.0_80.http.http.downstream_rq_completed: 2956738
listener.0.0.0.0_80.no_filter_chain_match: 0
listener.0.0.0.0_80.worker_0.downstream_cx_active: 12
listener.0.0.0.0_80.worker_0.downstream_cx_total: 42
listener.0.0.0.0_80.worker_1.downstream_cx_active: 17
listener.0.0.0.0_80.worker_1.downstream_cx_total: 40
listener.0.0.0.0_80.worker_10.downstream_cx_active: 25
listener.0.0.0.0_80.worker_10.downstream_cx_total: 44
listener.0.0.0.0_80.worker_11.downstream_cx_active: 12
listener.0.0.0.0_80.worker_11.downstream_cx_total: 33
listener.0.0.0.0_80.worker_12.downstream_cx_active: 24
listener.0.0.0.0_80.worker_12.downstream_cx_total: 52
listener.0.0.0.0_80.worker_13.downstream_cx_active: 21
listener.0.0.0.0_80.worker_13.downstream_cx_total: 44
listener.0.0.0.0_80.worker_14.downstream_cx_active: 25
listener.0.0.0.0_80.worker_14.downstream_cx_total: 47
listener.0.0.0.0_80.worker_15.downstream_cx_active: 20
listener.0.0.0.0_80.worker_15.downstream_cx_total: 43
listener.0.0.0.0_80.worker_16.downstream_cx_active: 24
listener.0.0.0.0_80.worker_16.downstream_cx_total: 52
listener.0.0.0.0_80.worker_17.downstream_cx_active: 21
listener.0.0.0.0_80.worker_17.downstream_cx_total: 45
listener.0.0.0.0_80.worker_18.downstream_cx_active: 15
listener.0.0.0.0_80.worker_18.downstream_cx_total: 40
listener.0.0.0.0_80.worker_19.downstream_cx_active: 23
listener.0.0.0.0_80.worker_19.downstream_cx_total: 48
listener.0.0.0.0_80.worker_2.downstream_cx_active: 18
listener.0.0.0.0_80.worker_2.downstream_cx_total: 39
listener.0.0.0.0_80.worker_20.downstream_cx_active: 16
listener.0.0.0.0_80.worker_20.downstream_cx_total: 36
listener.0.0.0.0_80.worker_21.downstream_cx_active: 13
listener.0.0.0.0_80.worker_21.downstream_cx_total: 31
listener.0.0.0.0_80.worker_22.downstream_cx_active: 13
listener.0.0.0.0_80.worker_22.downstream_cx_total: 36
listener.0.0.0.0_80.worker_23.downstream_cx_active: 24
listener.0.0.0.0_80.worker_23.downstream_cx_total: 47
listener.0.0.0.0_80.worker_24.downstream_cx_active: 15
listener.0.0.0.0_80.worker_24.downstream_cx_total: 34
listener.0.0.0.0_80.worker_25.downstream_cx_active: 19
listener.0.0.0.0_80.worker_25.downstream_cx_total: 51
listener.0.0.0.0_80.worker_26.downstream_cx_active: 21
listener.0.0.0.0_80.worker_26.downstream_cx_total: 42
listener.0.0.0.0_80.worker_27.downstream_cx_active: 23
listener.0.0.0.0_80.worker_27.downstream_cx_total: 47
listener.0.0.0.0_80.worker_28.downstream_cx_active: 25
listener.0.0.0.0_80.worker_28.downstream_cx_total: 45
listener.0.0.0.0_80.worker_29.downstream_cx_active: 16
listener.0.0.0.0_80.worker_29.downstream_cx_total: 34
listener.0.0.0.0_80.worker_3.downstream_cx_active: 16
listener.0.0.0.0_80.worker_3.downstream_cx_total: 47
listener.0.0.0.0_80.worker_30.downstream_cx_active: 19
listener.0.0.0.0_80.worker_30.downstream_cx_total: 47
listener.0.0.0.0_80.worker_31.downstream_cx_active: 14
listener.0.0.0.0_80.worker_31.downstream_cx_total: 31
listener.0.0.0.0_80.worker_32.downstream_cx_active: 19
listener.0.0.0.0_80.worker_32.downstream_cx_total: 42
listener.0.0.0.0_80.worker_33.downstream_cx_active: 21
listener.0.0.0.0_80.worker_33.downstream_cx_total: 48
listener.0.0.0.0_80.worker_34.downstream_cx_active: 28
listener.0.0.0.0_80.worker_34.downstream_cx_total: 53
listener.0.0.0.0_80.worker_35.downstream_cx_active: 15
listener.0.0.0.0_80.worker_35.downstream_cx_total: 43
listener.0.0.0.0_80.worker_36.downstream_cx_active: 17
listener.0.0.0.0_80.worker_36.downstream_cx_total: 43
listener.0.0.0.0_80.worker_37.downstream_cx_active: 19
listener.0.0.0.0_80.worker_37.downstream_cx_total: 39
listener.0.0.0.0_80.worker_38.downstream_cx_active: 16
listener.0.0.0.0_80.worker_38.downstream_cx_total: 37
listener.0.0.0.0_80.worker_39.downstream_cx_active: 19
listener.0.0.0.0_80.worker_39.downstream_cx_total: 43
listener.0.0.0.0_80.worker_4.downstream_cx_active: 17
listener.0.0.0.0_80.worker_4.downstream_cx_total: 28
listener.0.0.0.0_80.worker_40.downstream_cx_active: 27
listener.0.0.0.0_80.worker_40.downstream_cx_total: 49
listener.0.0.0.0_80.worker_41.downstream_cx_active: 12
listener.0.0.0.0_80.worker_41.downstream_cx_total: 31
listener.0.0.0.0_80.worker_42.downstream_cx_active: 21
listener.0.0.0.0_80.worker_42.downstream_cx_total: 47
listener.0.0.0.0_80.worker_43.downstream_cx_active: 16
listener.0.0.0.0_80.worker_43.downstream_cx_total: 32
listener.0.0.0.0_80.worker_44.downstream_cx_active: 20
listener.0.0.0.0_80.worker_44.downstream_cx_total: 33
listener.0.0.0.0_80.worker_45.downstream_cx_active: 24
listener.0.0.0.0_80.worker_45.downstream_cx_total: 41
listener.0.0.0.0_80.worker_46.downstream_cx_active: 20
listener.0.0.0.0_80.worker_46.downstream_cx_total: 42
listener.0.0.0.0_80.worker_47.downstream_cx_active: 19
listener.0.0.0.0_80.worker_47.downstream_cx_total: 40
listener.0.0.0.0_80.worker_48.downstream_cx_active: 17
listener.0.0.0.0_80.worker_48.downstream_cx_total: 32
listener.0.0.0.0_80.worker_49.downstream_cx_active: 19
listener.0.0.0.0_80.worker_49.downstream_cx_total: 34
listener.0.0.0.0_80.worker_5.downstream_cx_active: 18
listener.0.0.0.0_80.worker_5.downstream_cx_total: 42
listener.0.0.0.0_80.worker_50.downstream_cx_active: 26
listener.0.0.0.0_80.worker_50.downstream_cx_total: 52
listener.0.0.0.0_80.worker_51.downstream_cx_active: 20
listener.0.0.0.0_80.worker_51.downstream_cx_total: 45
listener.0.0.0.0_80.worker_52.downstream_cx_active: 19
listener.0.0.0.0_80.worker_52.downstream_cx_total: 40
listener.0.0.0.0_80.worker_53.downstream_cx_active: 19
listener.0.0.0.0_80.worker_53.downstream_cx_total: 46
listener.0.0.0.0_80.worker_54.downstream_cx_active: 22
listener.0.0.0.0_80.worker_54.downstream_cx_total: 45
listener.0.0.0.0_80.worker_55.downstream_cx_active: 24
listener.0.0.0.0_80.worker_55.downstream_cx_total: 54
listener.0.0.0.0_80.worker_56.downstream_cx_active: 12
listener.0.0.0.0_80.worker_56.downstream_cx_total: 34
listener.0.0.0.0_80.worker_57.downstream_cx_active: 13
listener.0.0.0.0_80.worker_57.downstream_cx_total: 32
listener.0.0.0.0_80.worker_58.downstream_cx_active: 19
listener.0.0.0.0_80.worker_58.downstream_cx_total: 39
listener.0.0.0.0_80.worker_59.downstream_cx_active: 23
listener.0.0.0.0_80.worker_59.downstream_cx_total: 42
listener.0.0.0.0_80.worker_6.downstream_cx_active: 24
listener.0.0.0.0_80.worker_6.downstream_cx_total: 39
listener.0.0.0.0_80.worker_60.downstream_cx_active: 22
listener.0.0.0.0_80.worker_60.downstream_cx_total: 40
listener.0.0.0.0_80.worker_61.downstream_cx_active: 15
listener.0.0.0.0_80.worker_61.downstream_cx_total: 34
listener.0.0.0.0_80.worker_62.downstream_cx_active: 19
listener.0.0.0.0_80.worker_62.downstream_cx_total: 41
listener.0.0.0.0_80.worker_63.downstream_cx_active: 19
listener.0.0.0.0_80.worker_63.downstream_cx_total: 35
listener.0.0.0.0_80.worker_7.downstream_cx_active: 22
listener.0.0.0.0_80.worker_7.downstream_cx_total: 41
listener.0.0.0.0_80.worker_8.downstream_cx_active: 20
listener.0.0.0.0_80.worker_8.downstream_cx_total: 42
listener.0.0.0.0_80.worker_9.downstream_cx_active: 11
listener.0.0.0.0_80.worker_9.downstream_cx_total: 26
listener.admin.downstream_cx_active: 1
listener.admin.downstream_cx_destroy: 713
listener.admin.downstream_cx_overflow: 0
listener.admin.downstream_cx_overload_reject: 0
listener.admin.downstream_cx_total: 714
listener.admin.downstream_cx_transport_socket_connect_timeout: 0
listener.admin.downstream_global_cx_overflow: 0
listener.admin.downstream_pre_cx_active: 0
listener.admin.downstream_pre_cx_timeout: 0
listener.admin.http.admin.downstream_rq_1xx: 0
listener.admin.http.admin.downstream_rq_2xx: 713
listener.admin.http.admin.downstream_rq_3xx: 0
listener.admin.http.admin.downstream_rq_4xx: 0
listener.admin.http.admin.downstream_rq_5xx: 0
listener.admin.http.admin.downstream_rq_completed: 713
listener.admin.main_thread.downstream_cx_active: 1
listener.admin.main_thread.downstream_cx_total: 714
listener.admin.no_filter_chain_match: 0
listener_manager.lds.init_fetch_timeout: 0
listener_manager.lds.update_attempt: 144
listener_manager.lds.update_failure: 0
listener_manager.lds.update_rejected: 0
listener_manager.lds.update_success: 143
listener_manager.lds.update_time: 1649802540829
listener_manager.lds.version: 2486829869471955647
listener_manager.listener_added: 2
listener_manager.listener_create_failure: 0
listener_manager.listener_create_success: 9216
listener_manager.listener_in_place_updated: 142
listener_manager.listener_modified: 142
listener_manager.listener_removed: 0
listener_manager.listener_stopped: 0
listener_manager.total_filter_chains_draining: 22
listener_manager.total_listeners_active: 2
listener_manager.total_listeners_draining: 0
listener_manager.total_listeners_warming: 0
listener_manager.workers_started: 1
main_thread.watchdog_mega_miss: 287
main_thread.watchdog_miss: 289
overload.envoy.overload_actions.shrink_heap.active: 0
overload.envoy.overload_actions.shrink_heap.scale_percent: 0
overload.envoy.overload_actions.shrink_heap.shrink_count: 0
overload.envoy.overload_actions.stop_accepting_requests.active: 0
overload.envoy.overload_actions.stop_accepting_requests.scale_percent: 0
overload.envoy.resource_monitors.fixed_heap.failed_updates: 0
overload.envoy.resource_monitors.fixed_heap.pressure: 23
overload.envoy.resource_monitors.fixed_heap.skipped_updates: 0
runtime.admin_overrides_active: 0
runtime.deprecated_feature_seen_since_process_start: 0
runtime.deprecated_feature_use: 0
runtime.load_error: 0
runtime.load_success: 1
runtime.num_keys: 0
runtime.num_layers: 0
runtime.override_dir_exists: 0
runtime.override_dir_not_exists: 1
server.compilation_settings.fips_mode: 0
server.concurrency: 64
server.days_until_first_cert_expiring: 0
server.debug_assertion_failures: 0
server.dropped_stat_flushes: 0
server.dynamic_unknown_fields: 0
server.envoy_bug_failures: 0
server.hot_restart_epoch: 0
server.hot_restart_generation: 1
server.live: 1
server.main_thread.watchdog_mega_miss: 287
server.main_thread.watchdog_miss: 289
server.memory_allocated: 22625564096
server.memory_heap_size: 24519901184
server.memory_physical_size: 23603857926
server.parent_connections: 0
server.seconds_until_first_ocsp_response_expiring: 0
server.state: 0
server.static_unknown_fields: 0
server.stats_recent_lookups: 6514310
server.total_connections: 6876
server.uptime: 5344
server.version: 11489287
server.wip_protos: 0
server.worker_0.watchdog_mega_miss: 0
server.worker_0.watchdog_miss: 0
server.worker_1.watchdog_mega_miss: 0
server.worker_1.watchdog_miss: 0
server.worker_10.watchdog_mega_miss: 0
server.worker_10.watchdog_miss: 1
server.worker_11.watchdog_mega_miss: 0
server.worker_11.watchdog_miss: 0
server.worker_12.watchdog_mega_miss: 0
server.worker_12.watchdog_miss: 1
server.worker_13.watchdog_mega_miss: 0
server.worker_13.watchdog_miss: 0
server.worker_14.watchdog_mega_miss: 0
server.worker_14.watchdog_miss: 0
server.worker_15.watchdog_mega_miss: 0
server.worker_15.watchdog_miss: 0
server.worker_16.watchdog_mega_miss: 0
server.worker_16.watchdog_miss: 0
server.worker_17.watchdog_mega_miss: 0
server.worker_17.watchdog_miss: 0
server.worker_18.watchdog_mega_miss: 0
server.worker_18.watchdog_miss: 0
server.worker_19.watchdog_mega_miss: 0
server.worker_19.watchdog_miss: 0
server.worker_2.watchdog_mega_miss: 0
server.worker_2.watchdog_miss: 0
server.worker_20.watchdog_mega_miss: 0
server.worker_20.watchdog_miss: 0
server.worker_21.watchdog_mega_miss: 0
server.worker_21.watchdog_miss: 0
server.worker_22.watchdog_mega_miss: 0
server.worker_22.watchdog_miss: 0
server.worker_23.watchdog_mega_miss: 0
server.worker_23.watchdog_miss: 0
server.worker_24.watchdog_mega_miss: 0
server.worker_24.watchdog_miss: 2
server.worker_25.watchdog_mega_miss: 0
server.worker_25.watchdog_miss: 0
server.worker_26.watchdog_mega_miss: 0
server.worker_26.watchdog_miss: 0
server.worker_27.watchdog_mega_miss: 0
server.worker_27.watchdog_miss: 0
server.worker_28.watchdog_mega_miss: 0
server.worker_28.watchdog_miss: 2
server.worker_29.watchdog_mega_miss: 0
server.worker_29.watchdog_miss: 0
server.worker_3.watchdog_mega_miss: 0
server.worker_3.watchdog_miss: 0
server.worker_30.watchdog_mega_miss: 0
server.worker_30.watchdog_miss: 0
server.worker_31.watchdog_mega_miss: 0
server.worker_31.watchdog_miss: 0
server.worker_32.watchdog_mega_miss: 0
server.worker_32.watchdog_miss: 1
server.worker_33.watchdog_mega_miss: 0
server.worker_33.watchdog_miss: 0
server.worker_34.watchdog_mega_miss: 0
server.worker_34.watchdog_miss: 1
server.worker_35.watchdog_mega_miss: 0
server.worker_35.watchdog_miss: 0
server.worker_36.watchdog_mega_miss: 0
server.worker_36.watchdog_miss: 0
server.worker_37.watchdog_mega_miss: 0
server.worker_37.watchdog_miss: 1
server.worker_38.watchdog_mega_miss: 0
server.worker_38.watchdog_miss: 1
server.worker_39.watchdog_mega_miss: 0
server.worker_39.watchdog_miss: 1
server.worker_4.watchdog_mega_miss: 0
server.worker_4.watchdog_miss: 0
server.worker_40.watchdog_mega_miss: 0
server.worker_40.watchdog_miss: 1
server.worker_41.watchdog_mega_miss: 0
server.worker_41.watchdog_miss: 1
server.worker_42.watchdog_mega_miss: 0
server.worker_42.watchdog_miss: 0
server.worker_43.watchdog_mega_miss: 0
server.worker_43.watchdog_miss: 0
server.worker_44.watchdog_mega_miss: 0
server.worker_44.watchdog_miss: 0
server.worker_45.watchdog_mega_miss: 0
server.worker_45.watchdog_miss: 1
server.worker_46.watchdog_mega_miss: 0
server.worker_46.watchdog_miss: 0
server.worker_47.watchdog_mega_miss: 0
server.worker_47.watchdog_miss: 0
server.worker_48.watchdog_mega_miss: 0
server.worker_48.watchdog_miss: 0
server.worker_49.watchdog_mega_miss: 0
server.worker_49.watchdog_miss: 0
server.worker_5.watchdog_mega_miss: 0
server.worker_5.watchdog_miss: 0
server.worker_50.watchdog_mega_miss: 0
server.worker_50.watchdog_miss: 0
server.worker_51.watchdog_mega_miss: 0
server.worker_51.watchdog_miss: 0
server.worker_52.watchdog_mega_miss: 0
server.worker_52.watchdog_miss: 1
server.worker_53.watchdog_mega_miss: 0
server.worker_53.watchdog_miss: 0
server.worker_54.watchdog_mega_miss: 0
server.worker_54.watchdog_miss: 0
server.worker_55.watchdog_mega_miss: 0
server.worker_55.watchdog_miss: 1
server.worker_56.watchdog_mega_miss: 0
server.worker_56.watchdog_miss: 0
server.worker_57.watchdog_mega_miss: 0
server.worker_57.watchdog_miss: 0
server.worker_58.watchdog_mega_miss: 0
server.worker_58.watchdog_miss: 1
server.worker_59.watchdog_mega_miss: 0
server.worker_59.watchdog_miss: 2
server.worker_6.watchdog_mega_miss: 0
server.worker_6.watchdog_miss: 0
server.worker_60.watchdog_mega_miss: 0
server.worker_60.watchdog_miss: 0
server.worker_61.watchdog_mega_miss: 0
server.worker_61.watchdog_miss: 2
server.worker_62.watchdog_mega_miss: 0
server.worker_62.watchdog_miss: 0
server.worker_63.watchdog_mega_miss: 0
server.worker_63.watchdog_miss: 0
server.worker_7.watchdog_mega_miss: 0
server.worker_7.watchdog_miss: 0
server.worker_8.watchdog_mega_miss: 0
server.worker_8.watchdog_miss: 0
server.worker_9.watchdog_mega_miss: 0
server.worker_9.watchdog_miss: 0
tls_inspector.alpn_found: 57616
tls_inspector.alpn_not_found: 123041
tls_inspector.client_hello_too_large: 0
tls_inspector.connection_closed: 1498
tls_inspector.read_error: 7
tls_inspector.sni_found: 135963
tls_inspector.sni_not_found: 44694
tls_inspector.tls_found: 180657
tls_inspector.tls_not_found: 290
vhost.fallback_virtualhost.vcluster.other.upstream_rq_retry: 0
vhost.fallback_virtualhost.vcluster.other.upstream_rq_retry_limit_exceeded: 0
vhost.fallback_virtualhost.vcluster.other.upstream_rq_retry_overflow: 0
vhost.fallback_virtualhost.vcluster.other.upstream_rq_retry_success: 0
vhost.fallback_virtualhost.vcluster.other.upstream_rq_timeout: 0
vhost.fallback_virtualhost.vcluster.other.upstream_rq_total: 0
vhost.health_virtualhost.vcluster.other.upstream_rq_retry: 0
vhost.health_virtualhost.vcluster.other.upstream_rq_retry_limit_exceeded: 0
vhost.health_virtualhost.vcluster.other.upstream_rq_retry_overflow: 0
vhost.health_virtualhost.vcluster.other.upstream_rq_retry_success: 0
vhost.health_virtualhost.vcluster.other.upstream_rq_timeout: 0
vhost.health_virtualhost.vcluster.other.upstream_rq_total: 0
vhost.http_virtualhost.vcluster.other.upstream_rq_retry: 0
vhost.http_virtualhost.vcluster.other.upstream_rq_retry_limit_exceeded: 0
vhost.http_virtualhost.vcluster.other.upstream_rq_retry_overflow: 0
vhost.http_virtualhost.vcluster.other.upstream_rq_retry_success: 0
vhost.http_virtualhost.vcluster.other.upstream_rq_timeout: 0
vhost.http_virtualhost.vcluster.other.upstream_rq_total: 0
workers.watchdog_mega_miss: 0
workers.watchdog_miss: 21
Also tagging @JakeCooper and @soggycactus from the team on this one.
About this issue

Original URL
State: open
Created 2 years ago
Reactions: 4
Comments: 40 (21 by maintainers)
Commits related to this issue

router: only cache single route for the active request (#26045) * router: only cache single route for the active request Signed-off-by: wbpcode <wangbaiping@corp.netease.com> In the previous im... — committed to envoyproxy/envoy by wbpcode a year ago
router: only cache single route for the active request (#26045) * router: only cache single route for the active request Signed-off-by: wbpcode <wangbaiping@corp.netease.com> In the previous impl o... — committed to RiverPhillips/envoy by wbpcode a year ago
Most upvoted comments

Confirmed via a test, rough figures with an 18mb~ route config and ramping up to 8k websocket connections. Base mem use at idle was about ~ 1.5GB.
32GB peak RAM use @ 0.1 RDS updates per second
8GB peak RAM use @ 0.02 RDS updates per second
1.5GB peak RAM use @ 0.02 RDS updates per second with a build of #26045 - can’t see any difference in the mem usage graph during the test 🎉
char8 on Mar 14, 2023
Yes that is correct. Something is really wrong if it’s taking 7 hours to build on that machine. It should build in probably 10-20 minutes I would guess.
mattklein123 on Apr 29, 2022
Should be same as https://github.com/envoyproxy/envoy/issues/20919. You can set envoy.reloadable_features.no_extension_lookup_by_name to false.
suniltheta on Apr 29, 2022
Thanks for the quick response!
In a previous issue around performance, you can find more info on our setup as well as a snippet of our config_dump https://github.com/envoyproxy/envoy/issues/19946. We’re not doing anything fancy.
We can start working on getting a memory profile next week.
gschier on Apr 14, 2022