istio: ext_authz gRPC service doesn't work with the cluster name generated by pilot

Bug description When using ext_authz filter with gRPC service, the pilot generated cluster name includes invalid character (|) making it unable to be used in the ext_authz configuration.

Expected behavior

Steps to reproduce the bug

  1. Assume the user deploys their external authorization server inside the mesh with the following config:
apiVersion: v1
kind: Service
metadata:
  name: ext-authz-server
  namespace: "foo"
  labels:
    app: ext-authz-server
spec:
  ports:
  - name: grpc
    port: 50051
    targetPort: 50051
  selector:
    app: ext-authz-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ext-authz-server
  namespace: "foo"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ext-authz-server
  template:
    metadata:
      labels:
        app: ext-authz-server
      annotations:
        sidecar.istio.io/inject: "true"
    spec:
      containers:
      - image: gcr.io/ymzhu-istio/ext_authz_server:0.1
        imagePullPolicy: IfNotPresent
        name: ext-authz-server
        ports:
        - containerPort: 50051
  1. Pilot will generate the cluster name as outbound|50051||ext-authz-server.foo.svc.cluster.local.

  2. Apply the following EnvoyFilter to insert the ext_authz to ingress gateway. The ext_authz filter is configured to send the request to outbound|50051||ext-authz-server.foo.svc.cluster.local:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ext-authz-filter
  namespace: istio-system
spec:
  workloadLabels:
    app: istio-ingressgateway
  filters:
  - listenerMatch:
      listenerType: GATEWAY
      listenerProtocol: HTTP 
    insertPosition:
      index: FIRST           
    filterName: envoy.ext_authz
    filterType: HTTP
    filterConfig:
      grpc_service:
        envoy_grpc:
          cluster_name: outbound|50051||ext-authz-server.foo.svc.cluster.local
  1. The configuration will not work because when ext_authz filter is making the gRPC request, the client side Envoy gRPC code will set the host header to the cluster name (see https://github.com/envoyproxy/envoy/blob/6da5308d780381ed112fb231baee0c3b7225b6d8/source/common/grpc/common.cc#L229 for the code) and the character | is invalid in the HTTP host header.

  2. On the server side (the Envoy in front of the ext-authz-server), the gRPC request will be rejected because the host header includes invalid character | with the following logging invalid frame: Invalid HTTP header field was received on stream 1 (see https://github.com/envoyproxy/envoy/issues/7061 for the background):

[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][debug][filter] [external/envoy/source/extensions/filters/listener/original_dst/original_dst.cc:18] original_dst: New connection accepted
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][debug][filter] [external/envoy/source/extensions/filters/listener/tls_inspector/tls_inspector.cc:72] tls inspector: new connection accepted
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][filter] [external/envoy/source/extensions/filters/listener/tls_inspector/tls_inspector.cc:165] tls inspector: recv: 803
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:466] [C106] socket event: 3
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:554] [C106] write ready
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:504] [C106] read ready
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][connection] [external/envoy/source/common/network/raw_buffer_socket.cc:24] [C106] read returns: 803
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][connection] [external/envoy/source/common/network/raw_buffer_socket.cc:38] [C106] read error: Resource temporarily unavailable
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][debug][http2] [external/envoy/source/common/http/http2/codec_impl.cc:900] [C106] setting stream-level initial window size to 268435456
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][debug][http2] [external/envoy/source/common/http/http2/codec_impl.cc:922] [C106] updating connection-level initial window size to 268435456
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:405] [C106] dispatching 803 bytes
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:468] [C106] about to recv frame type=4, flags=0
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:1185] [C106] track inbound frame type=4 flags=0 length=18 padding_length=0
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:483] [C106] recv frame type=4
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:468] [C106] about to recv frame type=8, flags=0
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:1185] [C106] track inbound frame type=8 flags=0 length=4 padding_length=0
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:483] [C106] recv frame type=8
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:468] [C106] about to recv frame type=1, flags=4
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:1185] [C106] track inbound frame type=1 flags=4 length=282 padding_length=0
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:263] [C106] new stream
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][debug][http2] [external/envoy/source/common/http/http2/codec_impl.cc:633] [C106] invalid frame: Invalid HTTP header field was received on stream 1
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:278] [C106] dispatch error: The user callback function failed
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1854] [C106][S1891935447965057206] stream reset
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][debug][connection] [external/envoy/source/common/network/connection_impl.cc:101] [C106] closing data_to_write=0 type=2
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][debug][connection] [external/envoy/source/common/network/connection_impl_base.cc:30] [C106] setting delayed close timer with timeout 1000 ms

Workaround

  1. A quick workaround is to apply another EnvoyFilter to change the cluster name to not to include the | character on ingress gateway:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ext-authz-filter
  namespace: istio-system
spec:
  workloadLabels:
    app: istio-ingressgateway
  filters:
  - listenerMatch:
      listenerType: GATEWAY
      listenerProtocol: HTTP 
    insertPosition:
      index: FIRST           
    filterName: envoy.ext_authz
    filterType: HTTP
    filterConfig:
      grpc_service:
        envoy_grpc:
          cluster_name: patched.ext-authz-server.foo.svc.cluster.local
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ext-authz-filter-cluster-patch
  namespace: istio-system
spec:
  workloadLabels:
    app: istio-ingressgateway
  configPatches:
  - applyTo: CLUSTER
    match:
      cluster:
        service: ext-authz-server.foo.svc.cluster.local
    patch:
      operation: MERGE
      value:
        name: "patched.ext-authz-server.foo.svc.cluster.local"
  1. After applied the above config, everything works. We can even enable mTLS and authorization policy on the ext-authz-server to further protect it from unauthorized access.

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm) 1.5

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 18 (8 by maintainers)

Most upvoted comments

here’s an end-toend example but for istio 1.5: https://github.com/salrashid123/istio_external_authorization_server

configPatches:
  - applyTo: CLUSTER
    match:
      cluster:
        name:  outbound|50051||ext-authz-server.foo.svc.cluster.local

@yangminzhu I think this is more appropriate, especially if multiple ports are exposed.