istio: Istio does not adhere to HTTP/2 RFC 7540
Bug description
Istio does not properly send a 421 response when a connection is reused and accident sent to a server that is not the correct origin. This can occur when there are two gateways, one with a wildcard certificate (*.example.com) and one with a different non-wildcard certificate (b.example.com) routing to two different apps (a.example.com and b.example.com) where a http/2 connection is first established to the wildcard gateway (on host a.example.com with *.example.com) then a resource is requested from an application on the non-wildcard gateway (b.example.com with certificate b.example.com).
Because of http/2 connection reuse it’s possible for traffic destined for the second app (b.example.com) to end up being routed on the existing connection for (a.example.com) due to the RFC definition of connection re-use in section 9.1.1 (https://tools.ietf.org/html/rfc7540#section-9.1.1, e.g., because the certificate can authoritatively handle the request, and the IP address is the same as they are on the same ingressgateway).
When that happens, according to section 9.1.2 istio should respond with a 421 indicating that the wrong connection was used and the origin was not found. This would instruct the browser to retry on a new connection, thus renegotiating TLS and presenting SNI and thus going down the non-wildcard certificate route and to a different gateway/virtual service and the correct service.
Expected behavior
Should return a 421 and the browser should re-connect and successfully find the resource. Instead a 404 is returned.
Steps to reproduce the bug
- Create istio 1.1.1+ instance with one ingress gateway.
- Create a DNS record a.example.com, and b.example.com both point to the ingress gateway.
- Create a gateway named “a” for a.example.com that uses a.example.com server host, and has a wildcard certificate for *.example.com.
- Create a gateway named “b” for b.example.com that uses b.example.com server host and has a specific certificate b.example.com.
- Create an app that hosts a static website with two files index.html and foobar.png. The index.html file should have an image tag that refers to an image
https://b.example.com/foobar.png(e.g.,<img src="https://b.example.com/foobar.png">) - Deploy the app twice to Kubernetes and attach virtual services for a.example.com to go to the app and b.example.com to go to the app (effectively a.example.com and b.example.com are both hosting the app with different certificates on the same IP address, where a is on a wildcard cert and b is not).
- Visit https://a.example.com/, notice that you receive a 404 in Chrome and Firefox but not safari or opera.
Version (include the output of istioctl version --remote and kubectl version)
kubectl vclient version: version.BuildInfo{Version:"1.1.1", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989", User:"root", Host:"7077232d-4c6c-11e9-813c-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.0-17-g2b13318"}
apps-private-ingressgateway version: version.BuildInfo{Version:"", GitRevision:"", User:"", Host:"", GolangVersion:"", DockerHub:"", BuildStatus:"", GitTag:""}
apps-public-ingressgateway version: version.BuildInfo{Version:"", GitRevision:"", User:"", Host:"", GolangVersion:"", DockerHub:"", BuildStatus:"", GitTag:""}
citadel version: version.BuildInfo{Version:"1.1.2", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989-dirty", User:"root", Host:"35adf5bb-5570-11e9-b00d-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.1"}
galley version: version.BuildInfo{Version:"1.1.2", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989-dirty", User:"root", Host:"35adf5bb-5570-11e9-b00d-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.1"}
pilot version: version.BuildInfo{Version:"1.1.2", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989-dirty", User:"root", Host:"35adf5bb-5570-11e9-b00d-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.1"}
policy version: version.BuildInfo{Version:"1.1.2", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989-dirty", User:"root", Host:"35adf5bb-5570-11e9-b00d-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.1"}
sidecar-injector version: version.BuildInfo{Version:"1.1.2", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989-dirty", User:"root", Host:"35adf5bb-5570-11e9-b00d-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.1"}
telemetry version: version.BuildInfo{Version:"1.1.2", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989-dirty", User:"root", Host:"35adf5bb-5570-11e9-b00d-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.1"}
sites-private-ingressgateway version: version.BuildInfo{Version:"", GitRevision:"", User:"", Host:"", GolangVersion:"", DockerHub:"", BuildStatus:"", GitTag:""}
sites-public-ingressgateway version: version.BuildInfo{Version:"", GitRevision:"", User:"", Host:"", GolangVersion:"", DockerHub:"", BuildStatus:"", GitTag:""}
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-18T14:14:00Z", GoVersion:"go1.9.7", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.5", GitCommit:"753b2dbc622f5cc417845f0ff8a77f539a4213ea", GitTreeState:"clean", BuildDate:"2018-11-26T14:31:35Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
How was Istio installed?
Helm
Environment where bug was observed (cloud vendor, OS, etc)
AWS, with istio installed and running with an NLB ingress or as a nodeport terminating the TLS.
About this issue
- Original URL
- State: open
- Created 8 years ago
- Reactions: 22
- Comments: 35 (11 by maintainers)
Thanks for the excellent writeup. I think we’re seeing the same issue with a slightly different setup. (Please tell me if I’m wrong. I’m happy to open a separate case.)
foo.example.local,bar.example.local,baz.example.local.NR no_route_foundfrom the gateway for routes that should exist. (Restarting the browser, or using a different browser, shows that they do exist.)Unrelated differences (I think?)
HttpRoutefor our routing. But the routes all get sent to the istio gateway-controller, so I think we’re ending up in the same code path(s).We also noticed that we can not reproduce this behavior in the latest Firefox. Maybe it’s being more conservative and choosing not to re-use http/2 connections across hostnames? (because of just these kinds of server-side errors? Or because it’s simpler not to? 🤷)
This gave us a ton of headaches because we started experiencing this when we were adding and testing CORS headers to allow requests across those domains. Whether that would work really depended on the particular access pattern that might trigger the bad behavior. And we couldn’t see anything wrong in our apps because the gateway was just rejecting the routes before even sending them on to our pods.
I estimate we wasted a cumulative week of developer time (across multiple developers) if not more because of this issue.
Is there nothing to be done that can at least avoid this issue until there’s a fix in envoy? (For example: Maybe the gateway could refuse to reuse HTTP/2 connections on certs that are wildcards and/or serve multiple hostnames? at least until the upstream fix is available?)
Our workaround for now was to create separate certs for each hostname. This is fine in development, but isn’t really tenable (as people say above) if you have a large number of hostnames to deal with, and can’t act as your own CA.
The easiest quick and dirty solution would be to allow users to disable ALPN for HTTP/2 on the gateway, because right now it’s hard-coded to be on with no settings to force disable it.
I had the same problem, when using two services, one which is part of istio-servicemesh and one which is not. Both with the same wildcard-certificate.
For me it seems to be working fine when creating an additional catch-all virtual service with wildcard-host on the related gateway. The virtual service is catching all hosts which are not defined in other virtual services. Then it is serving a 421 status code, which tells the browser to try it again with a new connection.
I found the needed information here: https://istio.io/v1.10/docs/ops/best-practices/security/#explicitly-disable-all-the-sensitive-http-host-under-relaxed-sni-host-matching
I would appreciate any updates on this topic too as we are hit in our development environment by this issue. This problem appeared after we migrated away from traefik. Having a workaround for this would be very welcome.
There are three choices to work around the issue. These focus on making sure the browser doesn’t try to reuse your connections:
The browser reuses connections if the IP:Port is the same as a previous connection and the previous connection’s cert is valid for the new hostname. See the HTTP spec for details. If you can affect one of those 3 items (IP, Port, Cert), you can avoid the connection reuse, which works around Istio’s limitation.