istio: Exposing jager via ingress gateway gives strange errors with http2 - Envoy breaks http1.1 <-> http2 ALPN
Bug description
I want to expose the standard jaeger service deployed with Istio via ingress gateway. When hitting the url with a http2 request I get 503 - Upstream connect error or disconnect/reset before headers. reset reason: connection termination
# curl --http1.1 -I https://jaeger.foo.com/
HTTP/1.1 200 OK
content-type: text/html; charset=utf-8
date: Sat, 16 Nov 2019 17:10:56 GMT
x-envoy-upstream-service-time: 0
server: istio-envoy
transfer-encoding: chunked
# curl --http2 -I https://jaeger.foo.com/
HTTP/2 503
content-length: 95
content-type: text/plain
date: Sat, 16 Nov 2019 17:11:01 GMT
server: istio-envoy
Not sure how to answer this question - seems to be an envoy bug.
Expected behavior
200s are returned with http2
Steps to reproduce the bug
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: jaeger-gateway
spec:
selector:
istio: ingressgateway # use Istio default gateway implementation
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "jaeger.foo.com"
tls:
httpsRedirect: true # sends 301 redirect for http requests
- port:
number: 443
name: https
protocol: HTTPS
hosts:
- "jaeger.foo.com"
tls:
mode: SIMPLE
# these keys have to exist in a secret called 'istio-ingressgateway-certs' in the istio-system namespace.
privateKey: /etc/istio/ingressgateway-certs/tls.key
serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: jaeger-query
spec:
hosts:
- "jaeger.foo.com"
gateways:
- jaeger-gateway
http:
- match:
- uri:
prefix: /
- uri:
prefix: /ping
route:
- destination:
port:
number: 16686
host: jaeger-query-ext
apiVersion: v1
kind: Service
metadata:
name: jaeger-query-ext
namespace: istio-system
annotations:
labels:
app: jaeger
jaeger-infra: jaeger-service
chart: tracing
heritage: Tiller
release: istio
spec:
ports:
- name: query-http
port: 16686
protocol: TCP
targetPort: 16686
selector:
app: jaeger
Confirm that its working properly on http1.1
curl --http1.1 -I https://jaeger.foo.com/
Then hit it with a http2 call
curl --http2 -I https://jaeger.foo.com/
Version (include the output of istioctl version --remote
and kubectl version
and helm version
if you used Helm)
$ istioctl version --remote
client version: 1.4.0-beta.5
control plane version: 1.4.0
data plane version: 1.4.0-beta.5 (2 proxies), 1.3.5 (1 proxies), 1.4.0 (1 proxies)
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.11-dispatcher", GitCommit:"2e298c7e992f83f47af60cf4830b11c7370f6668", GitTreeState:"clean", BuildDate:"2019-09-19T22:26:40Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.11-gke.14", GitCommit:"56d89863d1033f9668ddd6e1c1aea81cd846ef88", GitTreeState:"clean", BuildDate:"2019-11-07T19:12:22Z", GoVersion:"go1.12.11b4", Compiler:"gc", Platform:"linux/amd64"}
How was Istio installed?
helm template install/kubernetes/helm/istio --name istio --namespace istio-system --set grafana.enabled=false --set kiali.enabled=true --set prometheus.enabled=false --set tracing.enabled=true --set tracing.ingress.enabled=true --set pilot.traceSampling=100
Environment where bug was observed (cloud vendor, OS, etc)
GKE
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 26 (8 by maintainers)
@objectiser I think this should stay open. The envoy peeps don’t actually seem very keen on fixing it and I don’t think that using nginx is really a valid workaround. Could we see if its getting any +1s in the next month and then let the stale bot kill it?
fyi. adding
http-
in front of the service port name fixed the issue if you weren’t doing that already. We found the issue by finding this thread and confirmed it with 2 methods:curl with and without http2
curl --http1.1 -I https://api...com/your-path/
andcurl --http2 -I https://api...com/your-path/
if the http1.1 works and the http2 doesn’t, you are in this issue. But the other tests.chrome with http2 turned off. Our sites came back on and worked.
These 2 tests provided the clear definition and then all we did was update the service to get things working again.
@mooperd Could you create a feature request in the Jaeger repo?