envoy: gRPC Json Transcoder + Options Request from Web Browsers causes 404 for envoy config without a "/" as a matching path

If you are reporting any crash or any potential security issue, do not open an issue in this repo. Please report the issue via emailing envoy-security@googlegroups.com where the issue will be triaged appropriately.

Title: One line description gRPC Json Transcoder + Options Request from Web Browsers causes 404 for envoy config without a “/” as a matching path

Description:

What issue is being seen? Describe what should be happening instead of the bug, for example: Envoy should not crash, the expected value isn’t returned, etc.

CORS + gRPC-JSON transcoder do not translate requests downstream when “/” not specified as a route. OPTIONS returns 404. When multiple clusters are configures - one cannot use “/” as a route, which previously filed bugs have used as a “dummy” route to overcome this issue.

Config:

Include the config used to configure Envoy.

admin:
  address:
    socket_address: { address: 0.0.0.0, port_value: 9901 }
layered_runtime:
  layers:
  - name: disable_apple_dns
    static_layer:
      envoy.restart_features.use_apple_api_for_dns_lookups: false

static_resources:
  listeners:
  - name: listener1
    address:
      socket_address: { address: 0.0.0.0, port_value: 5000 }
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: grpc_json
          access_log:
          - name: envoy.access_loggers.stdout
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
          codec_type: AUTO
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              cors:
                allow_origin_string_match:
                - prefix: "*"
                allow_methods: GET, POST, OPTIONS
                allow_headers: authorization,keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web
                expose_headers: grpc-status,grpc-message,x-envoy-upstream-service-time
              routes:
              # NOTE: by default, matching happens based on the gRPC route, and not on the incoming request path.
              # Reference: https://www.envoyproxy.io/docs/envoy/latest/configuration/http_filters/grpc_json_transcoder_filter#route-configs-for-transcoded-requests
              - match: { prefix: "/commerce.catalog" }
                route: { cluster: rust, timeout: 60s }
              - match: { prefix: "/fs" }
                route: { cluster: rust, timeout: 60s }
              - match: { prefix: "/stream" }
                route: { cluster: rust, timeout: 60s }
              - match: { prefix: "/core.profile" }
                route: { cluster: rust, timeout: 60s }
              - match: { prefix: "/commerce.order" }
                route: { cluster: netty, timeout: 60s }
              - match: { prefix: "/commerce.session" }
                route: { cluster: netty, timeout: 60s }
              - match: { prefix: "/core.auth.AuthenticationApi" }
                route: { cluster: netty, timeout: 60s }
              - match: { prefix: "/" }
                route: { cluster: rust, timeout: 60s }
          http_filters:
          - name: envoy.filters.http.cors
          - name: envoy.filters.http.grpc_json_transcoder
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.grpc_json_transcoder.v3.GrpcJsonTranscoder
              proto_descriptor: "/tmp/envoy/proto.pb"
              services: ["commerce.catalog.CatalogApi", "commerce.order.OrderApi", "commerce.session.SessionApi", "stream.StreamApi", "core.profile.ProfileApi", "core.auth.AuthenticationApi", "fs.FileApi", "fs.StorageApi"]
              print_options:
                add_whitespace: true
                always_print_primitive_fields: true
                always_print_enums_as_ints: true
                preserve_proto_field_names: true
          - name: envoy.filters.http.router

  clusters:
  - name: rust
    connect_timeout: 30.00s
    type: LOGICAL_DNS
    lb_policy: ROUND_ROBIN
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http2_protocol_options:
            # Configure an HTTP/2 keep-alive to detect connection issues and reconnect
            # to the admin server if the connection is no longer responsive.
            connection_keepalive:
              interval: 30s
              timeout: 5s
    load_assignment:
      cluster_name: rust
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                # WARNING: "docker.for.mac.localhost" has been deprecated from Docker v18.03.0.
                # If you're running an older version of Docker, please use "docker.for.mac.localhost" instead.
                # Reference: https://docs.docker.com/docker-for-mac/release-notes/#docker-community-edition-18030-ce-mac59-2018-03-26
                address: localhost # 172.0.0.1 #host.docker.internal
                port_value: 50051
  - name: netty
    connect_timeout: 30.00s
    type: LOGICAL_DNS
    lb_policy: ROUND_ROBIN
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http2_protocol_options:
            # Configure an HTTP/2 keep-alive to detect connection issues and reconnect
            # to the admin server if the connection is no longer responsive.
            connection_keepalive:
              interval: 30s
              timeout: 5s
    load_assignment:
      cluster_name: netty
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                # WARNING: "docker.for.mac.localhost" has been deprecated from Docker v18.03.0.
                # If you're running an older version of Docker, please use "docker.for.mac.localhost" instead.
                # Reference: https://docs.docker.com/docker-for-mac/release-notes/#docker-community-edition-18030-ce-mac59-2018-03-26
                address: localhost # 172.0.0.1 #host.docker.internal
                port_value: 6565

Logs:

Include the access logs and the Envoy logs.

Note: If there are privacy concerns, sanitize the data prior to sharing.

Call Stack:

If the Envoy binary is crashing, a call stack is required. Please refer to the Bazel Stack trace documentation.

About this issue

Original URL
State: open
Created 3 years ago
Reactions: 2
Comments: 18 (2 by maintainers)

Most upvoted comments

This is still an issue in the latest version

FinnRG on Apr 29, 2022

I think this happens because there usually is no mapping for CORS preflight requests (HTTP method OPTIONS) in proto files. Thus grpc_json_transcoder filter will not match any routes and router will return a 404. In this case CORS filter is most probably bypassed.

You should be able to prevent this by adding additional bindings for HTTP OPTIONS method to each RPC in your proto file (not tested yet):

syntax = "proto3";
package bookstore;

import "google/api/annotations.proto";

service Bookstore {
  // Creates a new shelf in the bookstore.
  rpc CreateShelf(CreateShelfRequest) returns (Shelf) {
    option (google.api.http) = {
      post: "/shelf"
      body: "shelf"
      additional_bindings [{
        custom {
          kind: "OPTIONS"
          path: "/shelf"
        }
      }]
    };
  }
  // Returns a specific bookstore shelf.
  rpc GetShelf(GetShelfRequest) returns (Shelf) {
    option (google.api.http) = {
      get: "/shelves/{shelf}"
      additional_bindings [{
        custom {
          kind: "OPTIONS"
          path: "/shelves/{shelf}"
        }
      }]
    };
  }
  // Returns a list of all shelves in the bookstore.
  rpc ListShelves(google.protobuf.Empty) returns (ListShelvesResponse) {
    option (google.api.http) = {
      get: "/shelves"
      additional_bindings [{
        custom {
          kind: "OPTIONS"
          path: "/shelves"
        }
      }]
    };
  }
}

You then need to make sure that the upstream properly handles those requests. It should at least reply with status code 200 I think.

Another workaround is to add a separate route for OPTIONS requests to envoy configuration (note that this CORS configuration is insecure!):

admin:
  address:
    socket_address: {address: 0.0.0.0, port_value: 9901}

static_resources:
  listeners:
  - name: listener_1
    address:
      socket_address: {address: 0.0.0.0, port_value: 20000}
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: grpc_json
          codec_type: AUTO
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              typed_per_filter_config:
                envoy.filters.http.cors:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.cors.v3.CorsPolicy
                  allow_origin_string_match:
                  - safe_regex:
                      regex: ".*"
                  allow_methods: "GET, POST, PUT, PATCH, DELETE, OPTIONS"
                  allow_headers: "*, Authorization"
                  allow_credentials: true
                  allow_private_network_access: true
                  max_age: "86400"
              routes:
              # CORS preflight requests (HTTP method OPTIONS) do not work as expected when using gRPC-JSON transcoder
              # because there is no mapping for OPTIONS requests in proto files. A 404 Not Found will be returned instead.
              # This fallback route which matches any OPTIONS request works around this.
              - match:
                  safe_regex:
                    regex: ".*"
                  headers:
                  - name: ":method"
                    string_match:
                      exact: "OPTIONS"
                direct_response:
                  status: 204
                response_headers_to_add:
                - append_action: ADD_IF_ABSENT
                  header:
                    key: "access-control-allow-origin"
                    value: "%REQ(ORIGIN?:AUTHORITY)%"
                - append_action: ADD_IF_ABSENT
                  header:
                    key: "access-control-allow-methods"
                    value: "GET, POST, PUT, PATCH, DELETE, OPTIONS"
                - append_action: ADD_IF_ABSENT
                  header:
                    key: "access-control-allow-headers"
                    value: "*, Authorization"
                - append_action: ADD_IF_ABSENT
                  header:
                    key: "access-control-allow-credentials"
                    value: "true"
                - append_action: ADD_IF_ABSENT
                  header:
                    key: "access-control-max-age"
                    value: "86400"
              - match: {prefix: '/dev.hello'}
                route: {cluster: cluster-grpc-upstream, timeout: {seconds: 60}}
          http_filters:
          - name: envoy.filters.http.grpc_json_transcoder
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.grpc_json_transcoder.v3.GrpcJsonTranscoder
              proto_descriptor: "/etc/envoy/descriptor_set.pb"
              services: ["dev.hello.GreeterService", "dev.hello.FooService"]
              ignore_unknown_query_parameters: true
              convert_grpc_status: true
              print_options:
                add_whitespace: true
                always_print_primitive_fields: false
                always_print_enums_as_ints: false
                preserve_proto_field_names: true
                stream_newline_delimited: true
          - name: envoy.filters.http.cors
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.cors.v3.Cors
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

  clusters:
  - name: cluster-grpc-upstream
    type: strict_dns
    dns_lookup_family: V4_ONLY
    lb_policy: round_robin
    connect_timeout: 3s
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http2_protocol_options: {}
    load_assignment:
      cluster_name: grpc
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address: {address: hello-service, port_value: 10000}

Please note that according to my testing CORS filter should come after grpc_json_transcoder if you want it to add access-control-allow-* headers not only to preflight (OPTIONS) requests but also to the “real” (GET, PUT, POST, …) requests.

This issue relates to #6608 (with a cleaner fix + https://github.com/envoyproxy/envoy/issues/6608#issuecomment-1827354781) and #7833 I think.

Also see #8262 for why direct_response bypasses CORS filter. That’s the reason why I added all those response_headers_to_add to the configuration. When using a route like route: {cluster: cluster-grpc-upstream} instead of direct_response, this is not needed.

n0v1 on Feb 2, 2024