istio: ext_authz gRPC service doesn't work with the cluster name generated by pilot
Bug description
When using ext_authz
filter with gRPC service, the pilot generated cluster name includes invalid character (|
) making it unable to be used in the ext_authz configuration.
Expected behavior
Steps to reproduce the bug
- Assume the user deploys their external authorization server inside the mesh with the following config:
apiVersion: v1
kind: Service
metadata:
name: ext-authz-server
namespace: "foo"
labels:
app: ext-authz-server
spec:
ports:
- name: grpc
port: 50051
targetPort: 50051
selector:
app: ext-authz-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ext-authz-server
namespace: "foo"
spec:
replicas: 1
selector:
matchLabels:
app: ext-authz-server
template:
metadata:
labels:
app: ext-authz-server
annotations:
sidecar.istio.io/inject: "true"
spec:
containers:
- image: gcr.io/ymzhu-istio/ext_authz_server:0.1
imagePullPolicy: IfNotPresent
name: ext-authz-server
ports:
- containerPort: 50051
-
Pilot will generate the cluster name as
outbound|50051||ext-authz-server.foo.svc.cluster.local
. -
Apply the following EnvoyFilter to insert the ext_authz to ingress gateway. The
ext_authz
filter is configured to send the request tooutbound|50051||ext-authz-server.foo.svc.cluster.local
:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: ext-authz-filter
namespace: istio-system
spec:
workloadLabels:
app: istio-ingressgateway
filters:
- listenerMatch:
listenerType: GATEWAY
listenerProtocol: HTTP
insertPosition:
index: FIRST
filterName: envoy.ext_authz
filterType: HTTP
filterConfig:
grpc_service:
envoy_grpc:
cluster_name: outbound|50051||ext-authz-server.foo.svc.cluster.local
-
The configuration will not work because when ext_authz filter is making the gRPC request, the client side Envoy gRPC code will set the
host
header to the cluster name (see https://github.com/envoyproxy/envoy/blob/6da5308d780381ed112fb231baee0c3b7225b6d8/source/common/grpc/common.cc#L229 for the code) and the character|
is invalid in the HTTP host header. -
On the server side (the Envoy in front of the ext-authz-server), the gRPC request will be rejected because the host header includes invalid character
|
with the following logginginvalid frame: Invalid HTTP header field was received on stream 1
(see https://github.com/envoyproxy/envoy/issues/7061 for the background):
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][debug][filter] [external/envoy/source/extensions/filters/listener/original_dst/original_dst.cc:18] original_dst: New connection accepted
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][debug][filter] [external/envoy/source/extensions/filters/listener/tls_inspector/tls_inspector.cc:72] tls inspector: new connection accepted
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][filter] [external/envoy/source/extensions/filters/listener/tls_inspector/tls_inspector.cc:165] tls inspector: recv: 803
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:466] [C106] socket event: 3
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:554] [C106] write ready
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:504] [C106] read ready
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][connection] [external/envoy/source/common/network/raw_buffer_socket.cc:24] [C106] read returns: 803
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][connection] [external/envoy/source/common/network/raw_buffer_socket.cc:38] [C106] read error: Resource temporarily unavailable
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][debug][http2] [external/envoy/source/common/http/http2/codec_impl.cc:900] [C106] setting stream-level initial window size to 268435456
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][debug][http2] [external/envoy/source/common/http/http2/codec_impl.cc:922] [C106] updating connection-level initial window size to 268435456
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:405] [C106] dispatching 803 bytes
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:468] [C106] about to recv frame type=4, flags=0
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:1185] [C106] track inbound frame type=4 flags=0 length=18 padding_length=0
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:483] [C106] recv frame type=4
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:468] [C106] about to recv frame type=8, flags=0
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:1185] [C106] track inbound frame type=8 flags=0 length=4 padding_length=0
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:483] [C106] recv frame type=8
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:468] [C106] about to recv frame type=1, flags=4
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:1185] [C106] track inbound frame type=1 flags=4 length=282 padding_length=0
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:263] [C106] new stream
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][debug][http2] [external/envoy/source/common/http/http2/codec_impl.cc:633] [C106] invalid frame: Invalid HTTP header field was received on stream 1
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:278] [C106] dispatch error: The user callback function failed
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1854] [C106][S1891935447965057206] stream reset
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][debug][connection] [external/envoy/source/common/network/connection_impl.cc:101] [C106] closing data_to_write=0 type=2
[Envoy (Epoch 0)] [2020-03-05 01:26:53.606][28][debug][connection] [external/envoy/source/common/network/connection_impl_base.cc:30] [C106] setting delayed close timer with timeout 1000 ms
Workaround
- A quick workaround is to apply another EnvoyFilter to change the cluster name to not to include the
|
character on ingress gateway:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: ext-authz-filter
namespace: istio-system
spec:
workloadLabels:
app: istio-ingressgateway
filters:
- listenerMatch:
listenerType: GATEWAY
listenerProtocol: HTTP
insertPosition:
index: FIRST
filterName: envoy.ext_authz
filterType: HTTP
filterConfig:
grpc_service:
envoy_grpc:
cluster_name: patched.ext-authz-server.foo.svc.cluster.local
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: ext-authz-filter-cluster-patch
namespace: istio-system
spec:
workloadLabels:
app: istio-ingressgateway
configPatches:
- applyTo: CLUSTER
match:
cluster:
service: ext-authz-server.foo.svc.cluster.local
patch:
operation: MERGE
value:
name: "patched.ext-authz-server.foo.svc.cluster.local"
- After applied the above config, everything works. We can even enable mTLS and authorization policy on the ext-authz-server to further protect it from unauthorized access.
Version (include the output of istioctl version --remote
and kubectl version
and helm version
if you used Helm)
1.5
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 18 (8 by maintainers)
here’s an end-toend example but for istio 1.5: https://github.com/salrashid123/istio_external_authorization_server
@yangminzhu I think this is more appropriate, especially if multiple ports are exposed.