istio: Exposing jager via ingress gateway gives strange errors with http2 - Envoy breaks http1.1 <-> http2 ALPN

Bug description

I want to expose the standard jaeger service deployed with Istio via ingress gateway. When hitting the url with a http2 request I get 503 - Upstream connect error or disconnect/reset before headers. reset reason: connection termination

# curl --http1.1  -I https://jaeger.foo.com/
HTTP/1.1 200 OK
content-type: text/html; charset=utf-8
date: Sat, 16 Nov 2019 17:10:56 GMT
x-envoy-upstream-service-time: 0
server: istio-envoy
transfer-encoding: chunked

# curl --http2  -I https://jaeger.foo.com/
HTTP/2 503 
content-length: 95
content-type: text/plain
date: Sat, 16 Nov 2019 17:11:01 GMT
server: istio-envoy

Not sure how to answer this question - seems to be an envoy bug.

Expected behavior

200s are returned with http2

Steps to reproduce the bug

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: jaeger-gateway
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "jaeger.foo.com"
    tls:
      httpsRedirect: true # sends 301 redirect for http requests
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - "jaeger.foo.com"
    tls:
      mode: SIMPLE
      # these keys have to exist in a secret called 'istio-ingressgateway-certs' in the istio-system namespace.
      privateKey: /etc/istio/ingressgateway-certs/tls.key
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt


apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: jaeger-query
spec:
  hosts:
  - "jaeger.foo.com"
  gateways:
  - jaeger-gateway
  http:
  - match:
    - uri:
        prefix: /
    - uri:
        prefix: /ping
    route:
    - destination:
        port:
          number: 16686
        host: jaeger-query-ext


apiVersion: v1
kind: Service
metadata:
  name: jaeger-query-ext
  namespace: istio-system
  annotations:
  labels:
    app: jaeger
    jaeger-infra: jaeger-service
    chart: tracing
    heritage: Tiller
    release: istio
spec:
  ports:
    - name: query-http
      port: 16686
      protocol: TCP
      targetPort: 16686
  selector:
    app: jaeger

Confirm that its working properly on http1.1

curl --http1.1  -I https://jaeger.foo.com/

Then hit it with a http2 call

curl --http2  -I https://jaeger.foo.com/

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

$ istioctl version --remote
client version: 1.4.0-beta.5
control plane version: 1.4.0
data plane version: 1.4.0-beta.5 (2 proxies), 1.3.5 (1 proxies), 1.4.0 (1 proxies)
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.11-dispatcher", GitCommit:"2e298c7e992f83f47af60cf4830b11c7370f6668", GitTreeState:"clean", BuildDate:"2019-09-19T22:26:40Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.11-gke.14", GitCommit:"56d89863d1033f9668ddd6e1c1aea81cd846ef88", GitTreeState:"clean", BuildDate:"2019-11-07T19:12:22Z", GoVersion:"go1.12.11b4", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed? helm template install/kubernetes/helm/istio --name istio --namespace istio-system --set grafana.enabled=false --set kiali.enabled=true --set prometheus.enabled=false --set tracing.enabled=true --set tracing.ingress.enabled=true --set pilot.traceSampling=100

Environment where bug was observed (cloud vendor, OS, etc)

GKE

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 26 (8 by maintainers)

Most upvoted comments

@objectiser I think this should stay open. The envoy peeps don’t actually seem very keen on fixing it and I don’t think that using nginx is really a valid workaround. Could we see if its getting any +1s in the next month and then let the stale bot kill it?

fyi. adding http- in front of the service port name fixed the issue if you weren’t doing that already. We found the issue by finding this thread and confirmed it with 2 methods:

  1. curl with and without http2 curl --http1.1 -I https://api...com/your-path/ and curl --http2 -I https://api...com/your-path/ if the http1.1 works and the http2 doesn’t, you are in this issue. But the other tests.

  2. chrome with http2 turned off. Our sites came back on and worked.

These 2 tests provided the clear definition and then all we did was update the service to get things working again.

@mooperd Could you create a feature request in the Jaeger repo?