envoy: extauthz can't send raw bytes when using with_request_body

Description:

Hello! I’m trying to use the extauthz filter with the with_request_body option and am running into issues. Specifically, I am trying to forward the body of an incoming GRPC request to an external auth service (so that it can decode the GRPC body and perform auth as necessary). When I do this, I am seeing the following error in the envoy logs:

[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/wire_format_lite.cc:584] String field 'envoy.service.auth.v2.AttributeContext.HttpRequest.body' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.

I expected the extauthz filter to be totally agnostic to the format of the underlying request body that it was forwarding but it seems that it can only send UTF-8 data. My current workaround is using the lua filter instead of the extauthz one.

I suspect that this is the result of how the proto is defined here https://github.com/envoyproxy/envoy/blob/062c895f499382ae61dead16db2a7e78b9146525/api/envoy/service/auth/v2/attribute_context.proto#L131 but will defer to those with more experience.

Repro steps: This same error can be reproduced more simply by issuing a curl request with binary data.

echo -e '\x03\xF1' | curl -X POST --data-binary @- https://localhost:10002

which yields in the logs:

[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/wire_format_lite.cc:584] String field 'envoy.service.auth.v2.AttributeContext.HttpRequest.body' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
[2020-01-24T17:13:52.119Z] "POST / HTTP/1.1" 403 UAEX 3 0 1 - "-" "curl/7.64.1" "548b0ddb-cd8b-4cc6-ac60-0edf40b96b74" "localhost:10002" "-"

If I run the same command without binary data

echo 'a' | curl -X POST --data-binary @- https://localhost:10002

I don’t get the error message, just the UAEX access log (which is expected)

[2020-01-24T17:20:25.055Z] "POST / HTTP/1.1" 403 UAEX 2 0 0 - "-" "curl/7.64.1" "71470fb0-bf77-4f27-bdcc-701b60aea3e5" "localhost:10002" "-"

Logs and other output

/clusters : Click to expand! 
extauthz_cluster::default_priority::max_connections::1024
extauthz_cluster::default_priority::max_pending_requests::1024
extauthz_cluster::default_priority::max_requests::1024
extauthz_cluster::default_priority::max_retries::3
extauthz_cluster::high_priority::max_connections::1024
extauthz_cluster::high_priority::max_pending_requests::1024
extauthz_cluster::high_priority::max_requests::1024
extauthz_cluster::high_priority::max_retries::3
extauthz_cluster::added_via_api::false
extauthz_cluster::192.168.65.2:9123::cx_active::0
extauthz_cluster::192.168.65.2:9123::cx_connect_fail::0
extauthz_cluster::192.168.65.2:9123::cx_total::128
extauthz_cluster::192.168.65.2:9123::rq_active::0
extauthz_cluster::192.168.65.2:9123::rq_error::128
extauthz_cluster::192.168.65.2:9123::rq_success::0
extauthz_cluster::192.168.65.2:9123::rq_timeout::0
extauthz_cluster::192.168.65.2:9123::rq_total::128
extauthz_cluster::192.168.65.2:9123::hostname::host.docker.internal
extauthz_cluster::192.168.65.2:9123::health_flags::healthy
extauthz_cluster::192.168.65.2:9123::weight::1
extauthz_cluster::192.168.65.2:9123::region::
extauthz_cluster::192.168.65.2:9123::zone::
extauthz_cluster::192.168.65.2:9123::sub_zone::
extauthz_cluster::192.168.65.2:9123::canary::false
extauthz_cluster::192.168.65.2:9123::priority::0
extauthz_cluster::192.168.65.2:9123::success_rate::-1
extauthz_cluster::192.168.65.2:9123::local_origin_success_rate::-1
backend_cluster::default_priority::max_connections::1024
backend_cluster::default_priority::max_pending_requests::1024
backend_cluster::default_priority::max_requests::1024
backend_cluster::default_priority::max_retries::3
backend_cluster::high_priority::max_connections::1024
backend_cluster::high_priority::max_pending_requests::1024
backend_cluster::high_priority::max_requests::1024
backend_cluster::high_priority::max_retries::3
backend_cluster::added_via_api::false
backend_cluster::192.168.65.2:9998::cx_active::0
backend_cluster::192.168.65.2:9998::cx_connect_fail::0
backend_cluster::192.168.65.2:9998::cx_total::0
backend_cluster::192.168.65.2:9998::rq_active::0
backend_cluster::192.168.65.2:9998::rq_error::0
backend_cluster::192.168.65.2:9998::rq_success::0
backend_cluster::192.168.65.2:9998::rq_timeout::0
backend_cluster::192.168.65.2:9998::rq_total::0
backend_cluster::192.168.65.2:9998::hostname::host.docker.internal
backend_cluster::192.168.65.2:9998::health_flags::healthy
backend_cluster::192.168.65.2:9998::weight::1
backend_cluster::192.168.65.2:9998::region::
backend_cluster::192.168.65.2:9998::zone::
backend_cluster::192.168.65.2:9998::sub_zone::
backend_cluster::192.168.65.2:9998::canary::false
backend_cluster::192.168.65.2:9998::priority::0
backend_cluster::192.168.65.2:9998::success_rate::-1
backend_cluster::192.168.65.2:9998::local_origin_success_rate::-1
/stats : Click to expand! 
cluster.backend_cluster.assignment_stale: 0
cluster.backend_cluster.assignment_timeout_received: 0
cluster.backend_cluster.bind_errors: 0
cluster.backend_cluster.circuit_breakers.default.cx_open: 0
cluster.backend_cluster.circuit_breakers.default.cx_pool_open: 0
cluster.backend_cluster.circuit_breakers.default.rq_open: 0
cluster.backend_cluster.circuit_breakers.default.rq_pending_open: 0
cluster.backend_cluster.circuit_breakers.default.rq_retry_open: 0
cluster.backend_cluster.circuit_breakers.high.cx_open: 0
cluster.backend_cluster.circuit_breakers.high.cx_pool_open: 0
cluster.backend_cluster.circuit_breakers.high.rq_open: 0
cluster.backend_cluster.circuit_breakers.high.rq_pending_open: 0
cluster.backend_cluster.circuit_breakers.high.rq_retry_open: 0
cluster.backend_cluster.default.total_match_count: 7
cluster.backend_cluster.ext_authz.error: 710
cluster.backend_cluster.lb_healthy_panic: 0
cluster.backend_cluster.lb_local_cluster_not_ok: 0
cluster.backend_cluster.lb_recalculate_zone_structures: 0
cluster.backend_cluster.lb_subsets_active: 0
cluster.backend_cluster.lb_subsets_created: 0
cluster.backend_cluster.lb_subsets_fallback: 0
cluster.backend_cluster.lb_subsets_fallback_panic: 0
cluster.backend_cluster.lb_subsets_removed: 0
cluster.backend_cluster.lb_subsets_selected: 0
cluster.backend_cluster.lb_zone_cluster_too_small: 0
cluster.backend_cluster.lb_zone_no_capacity_left: 0
cluster.backend_cluster.lb_zone_number_differs: 0
cluster.backend_cluster.lb_zone_routing_all_directly: 0
cluster.backend_cluster.lb_zone_routing_cross_zone: 0
cluster.backend_cluster.lb_zone_routing_sampled: 0
cluster.backend_cluster.max_host_weight: 1
cluster.backend_cluster.membership_change: 1
cluster.backend_cluster.membership_degraded: 0
cluster.backend_cluster.membership_excluded: 0
cluster.backend_cluster.membership_healthy: 1
cluster.backend_cluster.membership_total: 1
cluster.backend_cluster.original_dst_host_invalid: 0
cluster.backend_cluster.retry_or_shadow_abandoned: 0
cluster.backend_cluster.update_attempt: 7
cluster.backend_cluster.update_empty: 0
cluster.backend_cluster.update_failure: 0
cluster.backend_cluster.update_no_rebuild: 6
cluster.backend_cluster.update_success: 7
cluster.backend_cluster.upstream_cx_active: 0
cluster.backend_cluster.upstream_cx_close_notify: 0
cluster.backend_cluster.upstream_cx_connect_attempts_exceeded: 0
cluster.backend_cluster.upstream_cx_connect_fail: 0
cluster.backend_cluster.upstream_cx_connect_timeout: 0
cluster.backend_cluster.upstream_cx_destroy: 0
cluster.backend_cluster.upstream_cx_destroy_local: 0
cluster.backend_cluster.upstream_cx_destroy_local_with_active_rq: 0
cluster.backend_cluster.upstream_cx_destroy_remote: 0
cluster.backend_cluster.upstream_cx_destroy_remote_with_active_rq: 0
cluster.backend_cluster.upstream_cx_destroy_with_active_rq: 0
cluster.backend_cluster.upstream_cx_http1_total: 0
cluster.backend_cluster.upstream_cx_http2_total: 0
cluster.backend_cluster.upstream_cx_idle_timeout: 0
cluster.backend_cluster.upstream_cx_max_requests: 0
cluster.backend_cluster.upstream_cx_none_healthy: 0
cluster.backend_cluster.upstream_cx_overflow: 0
cluster.backend_cluster.upstream_cx_pool_overflow: 0
cluster.backend_cluster.upstream_cx_protocol_error: 0
cluster.backend_cluster.upstream_cx_rx_bytes_buffered: 0
cluster.backend_cluster.upstream_cx_rx_bytes_total: 0
cluster.backend_cluster.upstream_cx_total: 0
cluster.backend_cluster.upstream_cx_tx_bytes_buffered: 0
cluster.backend_cluster.upstream_cx_tx_bytes_total: 0
cluster.backend_cluster.upstream_flow_control_backed_up_total: 0
cluster.backend_cluster.upstream_flow_control_drained_total: 0
cluster.backend_cluster.upstream_flow_control_paused_reading_total: 0
cluster.backend_cluster.upstream_flow_control_resumed_reading_total: 0
cluster.backend_cluster.upstream_internal_redirect_failed_total: 0
cluster.backend_cluster.upstream_internal_redirect_succeeded_total: 0
cluster.backend_cluster.upstream_rq_active: 0
cluster.backend_cluster.upstream_rq_cancelled: 0
cluster.backend_cluster.upstream_rq_completed: 0
cluster.backend_cluster.upstream_rq_maintenance_mode: 0
cluster.backend_cluster.upstream_rq_pending_active: 0
cluster.backend_cluster.upstream_rq_pending_failure_eject: 0
cluster.backend_cluster.upstream_rq_pending_overflow: 0
cluster.backend_cluster.upstream_rq_pending_total: 0
cluster.backend_cluster.upstream_rq_per_try_timeout: 0
cluster.backend_cluster.upstream_rq_retry: 0
cluster.backend_cluster.upstream_rq_retry_overflow: 0
cluster.backend_cluster.upstream_rq_retry_success: 0
cluster.backend_cluster.upstream_rq_rx_reset: 0
cluster.backend_cluster.upstream_rq_timeout: 0
cluster.backend_cluster.upstream_rq_total: 0
cluster.backend_cluster.upstream_rq_tx_reset: 0
cluster.backend_cluster.version: 0
cluster.extauthz_cluster.assignment_stale: 0
cluster.extauthz_cluster.assignment_timeout_received: 0
cluster.extauthz_cluster.bind_errors: 0
cluster.extauthz_cluster.circuit_breakers.default.cx_open: 0
cluster.extauthz_cluster.circuit_breakers.default.cx_pool_open: 0
cluster.extauthz_cluster.circuit_breakers.default.rq_open: 0
cluster.extauthz_cluster.circuit_breakers.default.rq_pending_open: 0
cluster.extauthz_cluster.circuit_breakers.default.rq_retry_open: 0
cluster.extauthz_cluster.circuit_breakers.high.cx_open: 0
cluster.extauthz_cluster.circuit_breakers.high.cx_pool_open: 0
cluster.extauthz_cluster.circuit_breakers.high.rq_open: 0
cluster.extauthz_cluster.circuit_breakers.high.rq_pending_open: 0
cluster.extauthz_cluster.circuit_breakers.high.rq_retry_open: 0
cluster.extauthz_cluster.default.total_match_count: 7
cluster.extauthz_cluster.http1.metadata_not_supported_error: 0
cluster.extauthz_cluster.internal.upstream_rq_503: 710
cluster.extauthz_cluster.internal.upstream_rq_5xx: 710
cluster.extauthz_cluster.internal.upstream_rq_completed: 710
cluster.extauthz_cluster.lb_healthy_panic: 0
cluster.extauthz_cluster.lb_local_cluster_not_ok: 0
cluster.extauthz_cluster.lb_recalculate_zone_structures: 0
cluster.extauthz_cluster.lb_subsets_active: 0
cluster.extauthz_cluster.lb_subsets_created: 0
cluster.extauthz_cluster.lb_subsets_fallback: 0
cluster.extauthz_cluster.lb_subsets_fallback_panic: 0
cluster.extauthz_cluster.lb_subsets_removed: 0
cluster.extauthz_cluster.lb_subsets_selected: 0
cluster.extauthz_cluster.lb_zone_cluster_too_small: 0
cluster.extauthz_cluster.lb_zone_no_capacity_left: 0
cluster.extauthz_cluster.lb_zone_number_differs: 0
cluster.extauthz_cluster.lb_zone_routing_all_directly: 0
cluster.extauthz_cluster.lb_zone_routing_cross_zone: 0
cluster.extauthz_cluster.lb_zone_routing_sampled: 0
cluster.extauthz_cluster.max_host_weight: 1
cluster.extauthz_cluster.membership_change: 1
cluster.extauthz_cluster.membership_degraded: 0
cluster.extauthz_cluster.membership_excluded: 0
cluster.extauthz_cluster.membership_healthy: 1
cluster.extauthz_cluster.membership_total: 1
cluster.extauthz_cluster.original_dst_host_invalid: 0
cluster.extauthz_cluster.retry_or_shadow_abandoned: 0
cluster.extauthz_cluster.update_attempt: 7
cluster.extauthz_cluster.update_empty: 0
cluster.extauthz_cluster.update_failure: 0
cluster.extauthz_cluster.update_no_rebuild: 6
cluster.extauthz_cluster.update_success: 7
cluster.extauthz_cluster.upstream_cx_active: 0
cluster.extauthz_cluster.upstream_cx_close_notify: 0
cluster.extauthz_cluster.upstream_cx_connect_attempts_exceeded: 0
cluster.extauthz_cluster.upstream_cx_connect_fail: 0
cluster.extauthz_cluster.upstream_cx_connect_timeout: 0
cluster.extauthz_cluster.upstream_cx_destroy: 710
cluster.extauthz_cluster.upstream_cx_destroy_local: 708
cluster.extauthz_cluster.upstream_cx_destroy_local_with_active_rq: 708
cluster.extauthz_cluster.upstream_cx_destroy_remote: 2
cluster.extauthz_cluster.upstream_cx_destroy_remote_with_active_rq: 2
cluster.extauthz_cluster.upstream_cx_destroy_with_active_rq: 710
cluster.extauthz_cluster.upstream_cx_http1_total: 710
cluster.extauthz_cluster.upstream_cx_http2_total: 0
cluster.extauthz_cluster.upstream_cx_idle_timeout: 0
cluster.extauthz_cluster.upstream_cx_max_requests: 0
cluster.extauthz_cluster.upstream_cx_none_healthy: 0
cluster.extauthz_cluster.upstream_cx_overflow: 0
cluster.extauthz_cluster.upstream_cx_pool_overflow: 0
cluster.extauthz_cluster.upstream_cx_protocol_error: 708
cluster.extauthz_cluster.upstream_cx_rx_bytes_buffered: 0
cluster.extauthz_cluster.upstream_cx_rx_bytes_total: 10620
cluster.extauthz_cluster.upstream_cx_total: 710
cluster.extauthz_cluster.upstream_cx_tx_bytes_buffered: 0
cluster.extauthz_cluster.upstream_cx_tx_bytes_total: 12800921
cluster.extauthz_cluster.upstream_flow_control_backed_up_total: 0
cluster.extauthz_cluster.upstream_flow_control_drained_total: 0
cluster.extauthz_cluster.upstream_flow_control_paused_reading_total: 0
cluster.extauthz_cluster.upstream_flow_control_resumed_reading_total: 0
cluster.extauthz_cluster.upstream_internal_redirect_failed_total: 0
cluster.extauthz_cluster.upstream_internal_redirect_succeeded_total: 0
cluster.extauthz_cluster.upstream_rq_503: 710
cluster.extauthz_cluster.upstream_rq_5xx: 710
cluster.extauthz_cluster.upstream_rq_active: 0
cluster.extauthz_cluster.upstream_rq_cancelled: 0
cluster.extauthz_cluster.upstream_rq_completed: 710
cluster.extauthz_cluster.upstream_rq_maintenance_mode: 0
cluster.extauthz_cluster.upstream_rq_pending_active: 0
cluster.extauthz_cluster.upstream_rq_pending_failure_eject: 0
cluster.extauthz_cluster.upstream_rq_pending_overflow: 0
cluster.extauthz_cluster.upstream_rq_pending_total: 710
cluster.extauthz_cluster.upstream_rq_per_try_timeout: 0
cluster.extauthz_cluster.upstream_rq_retry: 0
cluster.extauthz_cluster.upstream_rq_retry_overflow: 0
cluster.extauthz_cluster.upstream_rq_retry_success: 0
cluster.extauthz_cluster.upstream_rq_rx_reset: 0
cluster.extauthz_cluster.upstream_rq_timeout: 0
cluster.extauthz_cluster.upstream_rq_total: 710
cluster.extauthz_cluster.upstream_rq_tx_reset: 0
cluster.extauthz_cluster.version: 0
cluster_manager.active_clusters: 2
cluster_manager.cluster_added: 2
cluster_manager.cluster_modified: 0
cluster_manager.cluster_removed: 0
cluster_manager.cluster_updated: 0
cluster_manager.cluster_updated_via_merge: 0
cluster_manager.update_merge_cancelled: 0
cluster_manager.update_out_of_merge_window: 0
cluster_manager.warming_clusters: 0
filesystem.flushed_by_timer: 3
filesystem.reopen_failed: 0
filesystem.write_buffered: 712
filesystem.write_completed: 30
filesystem.write_failed: 0
filesystem.write_total_buffered: 8574
http.admin.downstream_cx_active: 2
http.admin.downstream_cx_delayed_close_timeout: 0
http.admin.downstream_cx_destroy: 1
http.admin.downstream_cx_destroy_active_rq: 0
http.admin.downstream_cx_destroy_local: 0
http.admin.downstream_cx_destroy_local_active_rq: 0
http.admin.downstream_cx_destroy_remote: 1
http.admin.downstream_cx_destroy_remote_active_rq: 0
http.admin.downstream_cx_drain_close: 0
http.admin.downstream_cx_http1_active: 1
http.admin.downstream_cx_http1_total: 1
http.admin.downstream_cx_http2_active: 0
http.admin.downstream_cx_http2_total: 0
http.admin.downstream_cx_http3_active: 0
http.admin.downstream_cx_http3_total: 0
http.admin.downstream_cx_idle_timeout: 0
http.admin.downstream_cx_max_duration_reached: 0
http.admin.downstream_cx_overload_disable_keepalive: 0
http.admin.downstream_cx_protocol_error: 0
http.admin.downstream_cx_rx_bytes_buffered: 669
http.admin.downstream_cx_rx_bytes_total: 2145
http.admin.downstream_cx_ssl_active: 0
http.admin.downstream_cx_ssl_total: 0
http.admin.downstream_cx_total: 3
http.admin.downstream_cx_tx_bytes_buffered: 0
http.admin.downstream_cx_tx_bytes_total: 6073
http.admin.downstream_cx_upgrades_active: 0
http.admin.downstream_cx_upgrades_total: 0
http.admin.downstream_flow_control_paused_reading_total: 0
http.admin.downstream_flow_control_resumed_reading_total: 0
http.admin.downstream_rq_1xx: 0
http.admin.downstream_rq_2xx: 2
http.admin.downstream_rq_3xx: 0
http.admin.downstream_rq_4xx: 0
http.admin.downstream_rq_5xx: 0
http.admin.downstream_rq_active: 1
http.admin.downstream_rq_completed: 2
http.admin.downstream_rq_http1_total: 3
http.admin.downstream_rq_http2_total: 0
http.admin.downstream_rq_http3_total: 0
http.admin.downstream_rq_idle_timeout: 0
http.admin.downstream_rq_non_relative_path: 0
http.admin.downstream_rq_overload_close: 0
http.admin.downstream_rq_response_before_rq_complete: 0
http.admin.downstream_rq_rx_reset: 0
http.admin.downstream_rq_timeout: 0
http.admin.downstream_rq_too_large: 0
http.admin.downstream_rq_total: 3
http.admin.downstream_rq_tx_reset: 0
http.admin.downstream_rq_ws_on_non_ws_route: 0
http.admin.rs_too_large: 0
http.async-client.no_cluster: 0
http.async-client.no_route: 0
http.async-client.rq_direct_response: 0
http.async-client.rq_redirect: 0
http.async-client.rq_reset_after_downstream_response_started: 0
http.async-client.rq_retry_skipped_request_not_complete: 0
http.async-client.rq_total: 710
http.ingress.downstream_cx_active: 50
http.ingress.downstream_cx_delayed_close_timeout: 0
http.ingress.downstream_cx_destroy: 0
http.ingress.downstream_cx_destroy_active_rq: 0
http.ingress.downstream_cx_destroy_local: 0
http.ingress.downstream_cx_destroy_local_active_rq: 0
http.ingress.downstream_cx_destroy_remote: 0
http.ingress.downstream_cx_destroy_remote_active_rq: 0
http.ingress.downstream_cx_drain_close: 0
http.ingress.downstream_cx_http1_active: 48
http.ingress.downstream_cx_http1_total: 48
http.ingress.downstream_cx_http2_active: 2
http.ingress.downstream_cx_http2_total: 2
http.ingress.downstream_cx_http3_active: 0
http.ingress.downstream_cx_http3_total: 0
http.ingress.downstream_cx_idle_timeout: 0
http.ingress.downstream_cx_max_duration_reached: 0
http.ingress.downstream_cx_overload_disable_keepalive: 0
http.ingress.downstream_cx_protocol_error: 0
http.ingress.downstream_cx_rx_bytes_buffered: 944712
http.ingress.downstream_cx_rx_bytes_total: 15527850
http.ingress.downstream_cx_ssl_active: 50
http.ingress.downstream_cx_ssl_total: 50
http.ingress.downstream_cx_total: 50
http.ingress.downstream_cx_tx_bytes_buffered: 0
http.ingress.downstream_cx_tx_bytes_total: 66974
http.ingress.downstream_cx_upgrades_active: 0
http.ingress.downstream_cx_upgrades_total: 0
http.ingress.downstream_flow_control_paused_reading_total: 0
http.ingress.downstream_flow_control_resumed_reading_total: 0
http.ingress.downstream_rq_1xx: 0
http.ingress.downstream_rq_2xx: 33
http.ingress.downstream_rq_3xx: 0
http.ingress.downstream_rq_4xx: 677
http.ingress.downstream_rq_5xx: 0
http.ingress.downstream_rq_active: 0
http.ingress.downstream_rq_completed: 710
http.ingress.downstream_rq_http1_total: 677
http.ingress.downstream_rq_http2_total: 33
http.ingress.downstream_rq_http3_total: 0
http.ingress.downstream_rq_idle_timeout: 0
http.ingress.downstream_rq_non_relative_path: 0
http.ingress.downstream_rq_overload_close: 0
http.ingress.downstream_rq_response_before_rq_complete: 0
http.ingress.downstream_rq_rx_reset: 0
http.ingress.downstream_rq_timeout: 0
http.ingress.downstream_rq_too_large: 0
http.ingress.downstream_rq_total: 710
http.ingress.downstream_rq_tx_reset: 0
http.ingress.downstream_rq_ws_on_non_ws_route: 0
http.ingress.ext_authz.denied: 0
http.ingress.ext_authz.error: 710
http.ingress.ext_authz.failure_mode_allowed: 0
http.ingress.ext_authz.ok: 0
http.ingress.no_cluster: 0
http.ingress.no_route: 0
http.ingress.rq_direct_response: 0
http.ingress.rq_redirect: 0
http.ingress.rq_reset_after_downstream_response_started: 0
http.ingress.rq_retry_skipped_request_not_complete: 0
http.ingress.rq_total: 0
http.ingress.rs_too_large: 0
http.ingress.tracing.client_enabled: 0
http.ingress.tracing.health_check: 0
http.ingress.tracing.not_traceable: 0
http.ingress.tracing.random_sampling: 0
http.ingress.tracing.service_forced: 0
http1.metadata_not_supported_error: 0
http2.header_overflow: 0
http2.headers_cb_no_stream: 0
http2.inbound_empty_frames_flood: 0
http2.inbound_priority_frames_flood: 0
http2.inbound_window_update_frames_flood: 0
http2.outbound_control_flood: 0
http2.outbound_flood: 0
http2.rx_messaging_error: 0
http2.rx_reset: 0
http2.too_many_header_frames: 0
http2.trailers: 0
http2.tx_reset: 0
listener.0.0.0.0_10002.downstream_cx_active: 50
listener.0.0.0.0_10002.downstream_cx_destroy: 0
listener.0.0.0.0_10002.downstream_cx_total: 50
listener.0.0.0.0_10002.downstream_pre_cx_active: 0
listener.0.0.0.0_10002.downstream_pre_cx_timeout: 0
listener.0.0.0.0_10002.http.ingress.downstream_rq_1xx: 0
listener.0.0.0.0_10002.http.ingress.downstream_rq_2xx: 33
listener.0.0.0.0_10002.http.ingress.downstream_rq_3xx: 0
listener.0.0.0.0_10002.http.ingress.downstream_rq_4xx: 677
listener.0.0.0.0_10002.http.ingress.downstream_rq_5xx: 0
listener.0.0.0.0_10002.http.ingress.downstream_rq_completed: 710
listener.0.0.0.0_10002.no_filter_chain_match: 0
listener.0.0.0.0_10002.server_ssl_socket_factory.downstream_context_secrets_not_ready: 0
listener.0.0.0.0_10002.server_ssl_socket_factory.ssl_context_update_by_sds: 0
listener.0.0.0.0_10002.server_ssl_socket_factory.upstream_context_secrets_not_ready: 0
listener.0.0.0.0_10002.ssl.ciphers.TLS_AES_128_GCM_SHA256: 50
listener.0.0.0.0_10002.ssl.connection_error: 0
listener.0.0.0.0_10002.ssl.curves.X25519: 50
listener.0.0.0.0_10002.ssl.fail_verify_cert_hash: 0
listener.0.0.0.0_10002.ssl.fail_verify_error: 0
listener.0.0.0.0_10002.ssl.fail_verify_no_cert: 0
listener.0.0.0.0_10002.ssl.fail_verify_san: 0
listener.0.0.0.0_10002.ssl.handshake: 50
listener.0.0.0.0_10002.ssl.no_certificate: 50
listener.0.0.0.0_10002.ssl.session_reused: 0
listener.0.0.0.0_10002.ssl.versions.TLSv1.3: 50
listener.0.0.0.0_10002.worker_0.downstream_cx_active: 8
listener.0.0.0.0_10002.worker_0.downstream_cx_total: 8
listener.0.0.0.0_10002.worker_1.downstream_cx_active: 11
listener.0.0.0.0_10002.worker_1.downstream_cx_total: 11
listener.0.0.0.0_10002.worker_2.downstream_cx_active: 8
listener.0.0.0.0_10002.worker_2.downstream_cx_total: 8
listener.0.0.0.0_10002.worker_3.downstream_cx_active: 8
listener.0.0.0.0_10002.worker_3.downstream_cx_total: 8
listener.0.0.0.0_10002.worker_4.downstream_cx_active: 11
listener.0.0.0.0_10002.worker_4.downstream_cx_total: 11
listener.0.0.0.0_10002.worker_5.downstream_cx_active: 4
listener.0.0.0.0_10002.worker_5.downstream_cx_total: 4
listener.admin.downstream_cx_active: 2
listener.admin.downstream_cx_destroy: 1
listener.admin.downstream_cx_total: 3
listener.admin.downstream_pre_cx_active: 0
listener.admin.downstream_pre_cx_timeout: 0
listener.admin.http.admin.downstream_rq_1xx: 0
listener.admin.http.admin.downstream_rq_2xx: 2
listener.admin.http.admin.downstream_rq_3xx: 0
listener.admin.http.admin.downstream_rq_4xx: 0
listener.admin.http.admin.downstream_rq_5xx: 0
listener.admin.http.admin.downstream_rq_completed: 2
listener.admin.main_thread.downstream_cx_active: 2
listener.admin.main_thread.downstream_cx_total: 3
listener.admin.no_filter_chain_match: 0
listener_manager.listener_added: 1
listener_manager.listener_create_failure: 0
listener_manager.listener_create_success: 6
listener_manager.listener_modified: 0
listener_manager.listener_removed: 0
listener_manager.listener_stopped: 0
listener_manager.total_listeners_active: 1
listener_manager.total_listeners_draining: 0
listener_manager.total_listeners_warming: 0
runtime.admin_overrides_active: 0
runtime.deprecated_feature_use: 1
runtime.load_error: 0
runtime.load_success: 1
runtime.num_keys: 0
runtime.num_layers: 2
runtime.override_dir_exists: 0
runtime.override_dir_not_exists: 1
server.concurrency: 6
server.days_until_first_cert_expiring: 728
server.debug_assertion_failures: 0
server.dynamic_unknown_fields: 0
server.hot_restart_epoch: 0
server.live: 1
server.main_thread.watchdog_mega_miss: 0
server.main_thread.watchdog_miss: 0
server.memory_allocated: 5171184
server.memory_heap_size: 14680064
server.parent_connections: 0
server.state: 0
server.static_unknown_fields: 0
server.stats_recent_lookups: 0
server.total_connections: 50
server.uptime: 30
server.version: 9381141
server.watchdog_mega_miss: 0
server.watchdog_miss: 0
server.worker_0.watchdog_mega_miss: 0
server.worker_0.watchdog_miss: 0
server.worker_1.watchdog_mega_miss: 0
server.worker_1.watchdog_miss: 0
server.worker_2.watchdog_mega_miss: 0
server.worker_2.watchdog_miss: 0
server.worker_3.watchdog_mega_miss: 0
server.worker_3.watchdog_miss: 0
server.worker_4.watchdog_mega_miss: 0
server.worker_4.watchdog_miss: 0
server.worker_5.watchdog_mega_miss: 0
server.worker_5.watchdog_miss: 0
cluster.backend_cluster.upstream_cx_connect_ms: No recorded values
cluster.backend_cluster.upstream_cx_length_ms: No recorded values
cluster.extauthz_cluster.upstream_cx_connect_ms: P0(0,0) P25(0,0) P50(1.075,1.08304) P75(3.01429,3.05493) P90(3.09429,5.0708) P95(4.07333,8.00318) P99(5.096,13.035) P99.5(8.044,18.5175) P99.9(8.0888,21.7035) P100(8.1,22)
cluster.extauthz_cluster.upstream_cx_length_ms: P0(0,0) P25(1.05588,1.06193) P50(3.01667,3.01508) P75(5.04444,5.07944) P90(7.02545,9.0388) P95(7.07636,11.4833) P99(9.06267,19.07) P99.5(9.08133,20.6783) P99.9(9.09627,32.407) P100(9.1,33)
http.admin.downstream_cx_length_ms: No recorded values
http.admin.downstream_rq_time: P0(nan,0) P25(nan,0) P50(nan,0) P75(nan,0) P90(nan,0) P95(nan,0) P99(nan,0) P99.5(nan,0) P99.9(nan,0) P100(nan,0)
http.ingress.downstream_cx_length_ms: No recorded values
http.ingress.downstream_rq_time: P0(1,1) P25(1.08485,2.00907) P50(3.08333,3.09493) P75(6.08333,6.09946) P90(8.04667,10.18) P95(9.08,12.6214) P99(11.94,20.0233) P99.5(13.44,21.035) P99.9(13.888,22.7035) P100(14,23)
listener.0.0.0.0_10002.downstream_cx_length_ms: No recorded values
listener.admin.downstream_cx_length_ms: No recorded values
server.initialization_time_ms: P0(nan,37) P25(nan,37.25) P50(nan,37.5) P75(nan,37.75) P90(nan,37.9) P95(nan,37.95) P99(nan,37.99) P99.5(nan,37.995) P99.9(nan,37.999) P100(nan,38)
/server_info : Click to expand! 
{
 "version": "8f2515a19bdcc75bea0bfd7016231a7661d0be6e/1.12.2/Clean/RELEASE/BoringSSL",
 "state": "LIVE",
 "hot_restart_version": "11.104",
 "command_line_options": {
  "base_id": "0",
  "concurrency": 6,
  "config_path": "/etc/envoy/envoy.yaml",
  "config_yaml": "",
  "allow_unknown_static_fields": false,
  "reject_unknown_dynamic_fields": false,
  "admin_address_path": "",
  "local_address_ip_version": "v4",
  "log_level": "info",
  "component_log_level": "",
  "log_format": "[%Y-%m-%d %T.%e][%t][%l][%n] %v",
  "log_path": "",
  "service_cluster": "",
  "service_node": "",
  "service_zone": "",
  "mode": "Serve",
  "max_stats": "0",
  "max_obj_name_len": "0",
  "disable_hot_restart": false,
  "enable_mutex_tracing": false,
  "restart_epoch": 0,
  "cpuset_threads": false,
  "file_flush_interval": "10s",
  "drain_time": "600s",
  "parent_shutdown_time": "900s"
 },
 "uptime_current_epoch": "67s",
 "uptime_all_epochs": "67s"
}
/config_dump : Click to expand! 
{
 "configs": [
  {
   "@type": "type.googleapis.com/envoy.admin.v2alpha.BootstrapConfigDump",
   "bootstrap": {
    "node": {
     "build_version": "8f2515a19bdcc75bea0bfd7016231a7661d0be6e/1.12.2/Clean/RELEASE/BoringSSL"
    },
    "static_resources": {
     "listeners": [
      {
       "address": {
        "socket_address": {
         "address": "0.0.0.0",
         "port_value": 10002
        }
       },
       "filter_chains": [
        {
         "filters": [
          {
           "name": "envoy.http_connection_manager",
           "typed_config": {
            "@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager",
            "route_config": {
             "virtual_hosts": [
              {
               "routes": [
                {
                 "route": {
                  "cluster": "backend_cluster"
                 },
                 "match": {
                  "prefix": "/"
                 }
                }
               ],
               "name": "service",
               "domains": [
                "*"
               ]
              }
             ],
             "name": "satellite_route"
            },
            "http_filters": [
             {
              "config": {
               "clear_route_cache": true,
               "grpc_service": {
                "envoy_grpc": {
                 "cluster_name": "extauthz_cluster"
                }
               },
               "with_request_body": {
                "allow_partial_message": false,
                "max_request_bytes": 5000000
               }
              },
              "name": "envoy.ext_authz"
             },
             {
              "typed_config": {
               "@type": "type.googleapis.com/envoy.config.filter.http.router.v2.Router",
               "start_child_span": true
              },
              "name": "envoy.router"
             }
            ],
            "access_log": [
             {
              "typed_config": {
               "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog",
               "path": "/dev/stdout"
              },
              "name": "envoy.file_access_log"
             }
            ],
            "stat_prefix": "ingress",
            "codec_type": "AUTO"
           }
          }
         ],
         "transport_socket": {
          "name": "envoy.transport_sockets.tls",
          "typed_config": {
           "@type": "type.googleapis.com/envoy.api.v2.auth.DownstreamTlsContext",
           "common_tls_context": {
            "tls_certificates": [
             {
              "private_key": {
               "filename": "certs/example.key.pem"
              },
              "certificate_chain": {
               "filename": "certs/example.bundle.pem"
              }
             }
            ]
           }
          }
         }
        }
       ]
      }
     ],
     "clusters": [
      {
       "name": "extauthz_cluster",
       "type": "STRICT_DNS",
       "connect_timeout": "15s",
       "hosts": [
        {
         "socket_address": {
          "address": "host.docker.internal",
          "port_value": 9123
         }
        }
       ]
      },
      {
       "name": "backend_cluster",
       "type": "STRICT_DNS",
       "connect_timeout": "15s",
       "hosts": [
        {
         "socket_address": {
          "address": "host.docker.internal",
          "port_value": 9998
         }
        }
       ],
       "http2_protocol_options": {}
      }
     ]
    },
    "admin": {
     "access_log_path": "/dev/stdout",
     "address": {
      "socket_address": {
       "address": "0.0.0.0",
       "port_value": 9901
      }
     }
    }
   },
   "last_updated": "2020-01-24T17:28:38.540Z"
  },
  {
   "@type": "type.googleapis.com/envoy.admin.v2alpha.ClustersConfigDump",
   "static_clusters": [
    {
     "cluster": {
      "name": "backend_cluster",
      "type": "STRICT_DNS",
      "connect_timeout": "15s",
      "hosts": [
       {
        "socket_address": {
         "address": "host.docker.internal",
         "port_value": 9998
        }
       }
      ],
      "http2_protocol_options": {}
     },
     "last_updated": "2020-01-24T17:28:38.546Z"
    },
    {
     "cluster": {
      "name": "extauthz_cluster",
      "type": "STRICT_DNS",
      "connect_timeout": "15s",
      "hosts": [
       {
        "socket_address": {
         "address": "host.docker.internal",
         "port_value": 9123
        }
       }
      ]
     },
     "last_updated": "2020-01-24T17:28:38.544Z"
    }
   ]
  },
  {
   "@type": "type.googleapis.com/envoy.admin.v2alpha.ListenersConfigDump",
   "static_listeners": [
    {
     "listener": {
      "address": {
       "socket_address": {
        "address": "0.0.0.0",
        "port_value": 10002
       }
      },
      "filter_chains": [
       {
        "filters": [
         {
          "name": "envoy.http_connection_manager",
          "typed_config": {
           "@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager",
           "route_config": {
            "virtual_hosts": [
             {
              "routes": [
               {
                "route": {
                 "cluster": "backend_cluster"
                },
                "match": {
                 "prefix": "/"
                }
               }
              ],
              "name": "service",
              "domains": [
               "*"
              ]
             }
            ],
            "name": "satellite_route"
           },
           "http_filters": [
            {
             "config": {
              "clear_route_cache": true,
              "grpc_service": {
               "envoy_grpc": {
                "cluster_name": "extauthz_cluster"
               }
              },
              "with_request_body": {
               "allow_partial_message": false,
               "max_request_bytes": 5000000
              }
             },
             "name": "envoy.ext_authz"
            },
            {
             "typed_config": {
              "@type": "type.googleapis.com/envoy.config.filter.http.router.v2.Router",
              "start_child_span": true
             },
             "name": "envoy.router"
            }
           ],
           "access_log": [
            {
             "typed_config": {
              "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog",
              "path": "/dev/stdout"
             },
             "name": "envoy.file_access_log"
            }
           ],
           "stat_prefix": "ingress",
           "codec_type": "AUTO"
          }
         }
        ],
        "transport_socket": {
         "name": "envoy.transport_sockets.tls",
         "typed_config": {
          "@type": "type.googleapis.com/envoy.api.v2.auth.DownstreamTlsContext",
          "common_tls_context": {
           "tls_certificates": [
            {
             "private_key": {
              "filename": "certs/example.key.pem"
             },
             "certificate_chain": {
              "filename": "certs/example.bundle.pem"
             }
            }
           ]
          }
         }
        }
       }
      ]
     },
     "last_updated": "2020-01-24T17:28:38.569Z"
    }
   ]
  },
  {
   "@type": "type.googleapis.com/envoy.admin.v2alpha.ScopedRoutesConfigDump"
  },
  {
   "@type": "type.googleapis.com/envoy.admin.v2alpha.RoutesConfigDump",
   "static_route_configs": [
    {
     "route_config": {
      "name": "satellite_route",
      "virtual_hosts": [
       {
        "name": "service",
        "domains": [
         "*"
        ],
        "routes": [
         {
          "match": {
           "prefix": "/"
          },
          "route": {
           "cluster": "backend_cluster"
          }
         }
        ]
       }
      ]
     },
     "last_updated": "2020-01-24T17:28:38.566Z"
    }
   ]
  },
  {
   "@type": "type.googleapis.com/envoy.admin.v2alpha.SecretsConfigDump"
  }
 ]
}
Logs : Click to expand! 
[2020-01-24 16:52:39.908][1][info][main] [source/server/server.cc:249] initializing epoch 0 (hot restart version=11.104)
[2020-01-24 16:52:39.908][1][info][main] [source/server/server.cc:251] statically linked extensions:
[2020-01-24 16:52:39.908][1][info][main] [source/server/server.cc:253]   access_loggers: envoy.file_access_log,envoy.http_grpc_access_log,envoy.tcp_grpc_access_log
[2020-01-24 16:52:39.908][1][info][main] [source/server/server.cc:256]   filters.http: envoy.buffer,envoy.cors,envoy.csrf,envoy.ext_authz,envoy.fault,envoy.filters.http.adaptive_concurrency,envoy.filters.http.dynamic_forward_proxy,envoy.filters.http.grpc_http1_reverse_bridge,envoy.filters.http.grpc_stats,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.original_src,envoy.filters.http.rbac,envoy.filters.http.tap,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash
[2020-01-24 16:52:39.908][1][info][main] [source/server/server.cc:259]   filters.listener: envoy.listener.http_inspector,envoy.listener.original_dst,envoy.listener.original_src,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2020-01-24 16:52:39.908][1][info][main] [source/server/server.cc:262]   filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.dubbo_proxy,envoy.filters.network.mysql_proxy,envoy.filters.network.rbac,envoy.filters.network.sni_cluster,envoy.filters.network.thrift_proxy,envoy.filters.network.zookeeper_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy
[2020-01-24 16:52:39.908][1][info][main] [source/server/server.cc:264]   stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd
[2020-01-24 16:52:39.908][1][info][main] [source/server/server.cc:266]   tracers: envoy.dynamic.ot,envoy.lightstep,envoy.tracers.datadog,envoy.tracers.opencensus,envoy.tracers.xray,envoy.zipkin
[2020-01-24 16:52:39.908][1][info][main] [source/server/server.cc:269]   transport_sockets.downstream: envoy.transport_sockets.alts,envoy.transport_sockets.raw_buffer,envoy.transport_sockets.tap,envoy.transport_sockets.tls,raw_buffer,tls
[2020-01-24 16:52:39.908][1][info][main] [source/server/server.cc:272]   transport_sockets.upstream: envoy.transport_sockets.alts,envoy.transport_sockets.raw_buffer,envoy.transport_sockets.tap,envoy.transport_sockets.tls,raw_buffer,tls
[2020-01-24 16:52:39.908][1][info][main] [source/server/server.cc:278] buffer implementation: new
[2020-01-24 16:52:39.925][1][info][main] [source/server/server.cc:344] admin address: 0.0.0.0:9901
[2020-01-24 16:52:39.926][1][info][main] [source/server/server.cc:458] runtime: layers:
  - name: base
    static_layer:
      {}
  - name: admin
    admin_layer:
      {}
[2020-01-24 16:52:39.926][1][info][config] [source/server/configuration_impl.cc:62] loading 0 static secret(s)
[2020-01-24 16:52:39.926][1][info][config] [source/server/configuration_impl.cc:68] loading 2 cluster(s)
[2020-01-24 16:52:39.928][1][info][config] [source/server/configuration_impl.cc:72] loading 1 listener(s)
[2020-01-24 16:52:39.941][1][warning][misc] [source/common/protobuf/utility.cc:282] Using deprecated option 'envoy.config.filter.network.http_connection_manager.v2.HttpFilter.config' from file http_connection_manager.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2020-01-24 16:52:39.946][1][info][config] [source/server/configuration_impl.cc:97] loading tracing configuration
[2020-01-24 16:52:39.946][1][info][config] [source/server/configuration_impl.cc:117] loading stats sink configuration
[2020-01-24 16:52:39.947][1][info][main] [source/server/server.cc:549] starting main dispatch loop
[2020-01-24 16:52:39.948][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:161] cm init: all clusters initialized
[2020-01-24 16:52:39.948][1][info][main] [source/server/server.cc:528] all clusters initialized. initializing init manager
[2020-01-24 16:52:39.948][1][info][config] [source/server/listener_manager_impl.cc:578] all dependencies initialized. starting workers
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/wire_format_lite.cc:584] String field 'envoy.service.auth.v2.AttributeContext.HttpRequest.body' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/wire_format_lite.cc:584] String field 'envoy.service.auth.v2.AttributeContext.HttpRequest.body' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/wire_format_lite.cc:584] String field 'envoy.service.auth.v2.AttributeContext.HttpRequest.body' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/wire_format_lite.cc:584] String field 'envoy.service.auth.v2.AttributeContext.HttpRequest.body' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/wire_format_lite.cc:584] String field 'envoy.service.auth.v2.AttributeContext.HttpRequest.body' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/wire_format_lite.cc:584] String field 'envoy.service.auth.v2.AttributeContext.HttpRequest.body' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/wire_format_lite.cc:584] String field 'envoy.service.auth.v2.AttributeContext.HttpRequest.body' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/wire_format_lite.cc:584] String field 'envoy.service.auth.v2.AttributeContext.HttpRequest.body' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 20 (14 by maintainers)

Commits related to this issue

Most upvoted comments

Yes this is a problem and we should fix it. However, it will require a deprecation cycle as we can’t just change the type on the current field. Marking help wanted.

+4
mattklein123 on Jan 24, 2020

This is a problem for us as well. Another thought is that we add a content-type filter to the ext_authz that will not forward the body if the content-type doesn’t match. We could probably set a flag indicating that it didn’t forward the body – possible the same flag that we use for partial data.

E.g. if I only care about json data, I could add the following condition:

HEADER["content-type"]: oneof("application/json", ...)

It’d also be nice if these filters were per-route, since different endpoints are likely to have different content types.

+2
klarose on Jul 6, 2020

Workaround is to base64 your payload. However if that’s not an option, let’s try to add a new field in v3.

+1
dio on Apr 9, 2020
Read more comments on GitHub
← envoy: `ext_proc` HTTP filter does not properly end the gRPC request stream
envoy: FileSystemHTTPCache crash →