istio: Wildcard hosts fail to match when port is included in host header

Bug description

Hi,

I am using istio ingress gateway (simplified yaml are shown below) to expose tensorflow serving. I encounter similar issue where I need to explicitly set the host name in :authority from GRPC client to make GRPC call works. The setup for HTTP is working fine.

The GRPC issue is similar with what’s described by where it seems istio tries to match $host:$port for the hosts in virtual service. e.g. if I have GRPC endpoint in ms-123.byom.abc.com

• if explicitly set :authority to ms-123.byom.abc.com, GRPC works fine
• if I don’t set :authority explicitly. GRPC will return Unimplemented. and I see the access log in ingress-gateway pod that host = ms-123.byom.abc.com:443 (note that there’s additional port).

[2020-07-09T01:00:06.361Z] "POST /tensorflow.serving.PredictionService/GetModelMetadata HTTP/2" 200 NR "-" "-" 0 0 0 - "100.96.1.1" "grpc-java-netty/1.29.0" "8ae3780a-725f-4aa8-899d-ac222968268d" "ms-9177c71a-874d-4821-b65c-9e58ca423fa3.byom.abc.com:443" "-" - - 100.96.0.27:8443 100.96.1.1:63486 ms-9177c71a-874d-4821-b65c-9e58ca423fa3.byom.abc.com -

Comparing with log of successful HTTP call, that log doesn’t have 443 port.

[2020-07-09T01:17:56.496Z] "GET /v1/models/inception-v3-08 HTTP/2" 200 - "-" "-" 0 154 53 53 "100.96.1.1" "curl/7.64.1" "f9ca8da5-cad2-4d56-8d53-6b9b1a0b1374" "ms-9177c71a-874d-4821-b65c-9e58ca423fa3.byom.abc.com" "100.96.0.41:53545" outbound|53545||mlf-inference-server.byom.svc.cluster.local 100.96.0.27:38586 100.96.0.27:8443 100.96.1.1:64506 ms-9177c71a-874d-4821-b65c-9e58ca423fa3.byom.abc.com -

Can anyone help me find what’s missing? We can’t force our clients to change their codes to explicitly set :authority in their GRPC requests.

Since @andraxylia mentioned that this is supported by the gateway. I thought there should be a way to make it work in ingress gateway.

[ ] Configuration Infrastructure [ ] Docs [ ] Installation [ x] Networking [ ] Performance and Scalability [ ] Policies and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure

Expected behavior GRPC client doesn’t need to explicitly specify host in authority metadata.

Steps to reproduce the bug

• deploy GRPC server (i.e. inception model running on tensorflow serving 1.14)
• these are gateway and related virtual service (domain name is modified to byom.abc.com)

tensorflow serving 1.14 expose both GRPC and HTTP. I remove HTTP configuration for brevity, because HTTP call is working fine.

curl -v https://$ENDPOINT/v1/models/inception -H "Authorization: Bearer $TOKEN"

for GRPC, I use grpcurl.

note that here I need to specify -authority $ENDPOINT explicitly to make it work. Otherwise, I will get unimplemented.

grpcurl -v \
-authority $ENDPOINT \
-import-path  $TEST_DATA_DIR/proto -proto $TEST_DATA_DIR/proto/tensorflow_serving/apis/model_service.proto \
-d '{"model_spec": {"name": "inception"}}' \
-H "Authorization: Bearer $TOKEN" \
$ENDPOINT:443 tensorflow.serving.ModelService/GetModelStatus

kind: VirtualService
metadata:
  name: mlf-inference-server
  namespace: byom
spec:
  gateways:
  - istio-system/byom-gateway
  hosts:
  - '*.byom.abc.com'
  http:
  - match:
    - uri:
        prefix: /tensorflow.serving.PredictionService
    - uri:
        prefix: /tensorflow.serving.ModelService
    - uri:
        prefix: /tensorflow.serving.SessionService
    retries:
      attempts: 3
      perTryTimeout: 30s
    route:
    - destination:
        host: mlf-inference-server.byom.svc.cluster.local
        port:
          number: 53547
    timeout: 10m

gateway

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: byom-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*.byom.abc.com'
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: byom-cert
      mode: SIMPLE
  - hosts:
    - '*.byom.abc.com'
    port:
      name: http
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: true

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

(global-3.7.2) i341814@ariefhidayat ~ % istioctl version --remote
client version: 1.6.3
cluster-local-gateway version: 
pilot version: 1.6.3
data plane version: 1.6.3 (8 proxies)

(global-3.7.2) i341814@ariefhidayat ~ % kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-14T04:24:34Z", GoVersion:"go1.12.13", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.10", GitCommit:"f3add640dbcd4f3c33a7749f38baaac0b3fe810d", GitTreeState:"clean", BuildDate:"2020-05-20T13:51:56Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed? via istioctl

Environment where bug was observed (cloud vendor, OS, etc) AWS gardener cluster. Haven’t tried in other cluster.