istio: http2 requests fail with "protocol error: The user callback function failed"

Describe the bug Using the daily-build/master-20180615-09-15, http2 requests to an HTTPS service fail in the sidecar with the following message:

[2018-06-20 13:37:29.830][30][info][client] external/envoy/source/common/http/codec_client.cc:117] [C6] protocol error: The user callback function failed
[2018-06-20T13:37:29.825Z] "GET / HTTP/2" 503 UC 0 57 5 - "10.132.0.8" "curl/7.60.0" "5c2bfba2-ea0d-932b-b937-dbe03cdda2df" "httpbin.mydomain.xyz" "127.0.0.1:80"

http1.1 requests are successful:

$ curl -I --http1.1 https://httpbin.mydomain.xyz
HTTP/1.1 200 OK
server: envoy
date: Wed, 20 Jun 2018 13:41:01 GMT
content-type: text/html; charset=utf-8
content-length: 7814
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 111

But http2 requests fail:

$ curl -I --http2 https://httpbin.mydomain.xyz
HTTP/2 503
content-length: 57
content-type: text/plain
date: Wed, 20 Jun 2018 13:46:03 GMT
server: envoy
x-envoy-upstream-service-time: 88

Expected behavior http2 requests should succeed.

Steps to reproduce the bug

Deploy Istio into a fresh GKE 1.9.7-gke.3 cluster, with cert-manager installed and a certificate created for the *.mydomain.xyz domain.

Edit the istio config map to add istio-system namespace to the statsdUdpAddress: statsdUdpAddress: istio-statsd-prom-bridge.istio-system:9125

$ helm install install/kubernetes/helm/istio --name istio --namespace istio-system \
        --set global.mtls.enabled=true \
        --set global.controlPlaneSecurityEnabled=true \
        --set global.proxy.includeIPRanges="10.28.0.0/14\,10.31.240.0/20" \
        --set global.proxy.resources.limits.cpu=100m \
        --set global.proxy.resources.limits.memory=128Mi \
        --set ingressgateway.enabled=true \
        --set ingress.enabled=false \
        --set egressgateway.enabled=false \
        --set prometheus.enabled=false

Deploy httpbin service. Update the gateway host to something available.

$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Namespace
metadata:
  name: httpbin
  labels:
    name: httpbin
    istio-injection: enabled
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: gateway
  namespace: httpbin
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - '*.mydomain.xyz'
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
  namespace: httpbin
spec:
  hosts:
  - httpbin.mydomain.xyz
  gateways:
  - gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: httpbin.httpbin.svc.cluster.local
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  namespace: httpbin
  labels:
    app: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: httpbin
  namespace: httpbin
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      containers:
      - image: kennethreitz/httpbin
        imagePullPolicy: Always
        name: httpbin
        ports:
        - containerPort: 80
EOF

Version What version of istio and Kubernetes are you using? Use istioctl version and kubectl version

Istio release https://gcsweb.istio.io/gcs/istio-prerelease/daily-build/master-20180615-09-15/

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.6", GitCommit:"6260bb08c46c31eea6cb538b34a9ceb3e406689c", GitTreeState:"clean", BuildDate:"2017-12-21T06:34:11Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.7-gke.3", GitCommit:"9b5b719c5f295c99de68ffb5b63101b0e0175376", GitTreeState:"clean", BuildDate:"2018-05-31T18:32:23Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

Is Istio Auth enabled or not? Enabled, command line above.

Environment GKE

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 27 (18 by maintainers)

Commits related to this issue

Most upvoted comments

I suddenly get it too now…

I tried deploying your yaml on IBM’s cluster with the slight change below and it works correctly with master. I get HTTP/2 200 when accessing the https ingress gateway with curl --http2.

The change I’ve made is to have:

hosts:
  - '*'

in both VS and GW. I used my own self-signed key/cert for the ingressgateway tls secret.

I wonder if it’s something specific to that daily or specific to GKE?