istio: httpsRedirect at Gateway causes redirect loop when used in AWS EKS with ACM cert

Bug description When used in AWS EKS, the release version 1.2.5 of istio (installed using helm), causes a continuous HTTPS redirect loop if the value of tls.httpsRedirect is set to true at the Gateway level.

Affected product area (please put an X in all that apply)

[ ] Configuration Infrastructure [ ] Docs [ ] Installation [x] Networking [ ] Performance and Scalability [ ] Policies and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure

Expected behavior Should not cause HTTPS redirect loop

Steps to reproduce the bug Start an AWS EKS cluster and deploy istio 1.2.5

Version (include the output of istioctl version --remote and kubectl version)

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.8-eks-a977ba", GitCommit:"a977bab148535ec195f12edc8720913c7b943f9c", GitTreeState:"clean", BuildDate:"2019-07-29T20:47:04Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
$ istioctl version --remote
client version: 1.2.5
citadel version: 1.2.5
egressgateway version: 1.2.5
galley version: 1.2.5
ingressgateway version: 1.2.5
pilot version: 1.2.5
policy version: 1.2.5
sidecar-injector version: 1.2.5
telemetry version: 1.2.5

How was Istio installed? Using helm

Environment where bug was observed (cloud vendor, OS, etc) AWS EKS

Relevant information

This is the export of the gateway:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: my-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - '*'
      port:
        name: http
        number: 80
        protocol: HTTP
      tls:
        httpsRedirect: true
    - hosts:
        - '*.staging.some.domain'
      port:
        name: https
        number: 443
        protocol: HTTP

And this is the export of the virtualservice:

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: image-service
  namespace: sip
spec:
  gateways:
  - my-gateway.istio-system.svc.cluster.local
  hosts:
  - img.staging.some.domain
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: image-service.sip.svc.cluster.local
        port:
          number: 80

I have checked all logs within the custom service and the requests do not reach the container, meaning that the redirect loop happens somewhere within the istio system.

Sample ->

$ curl -iL https://img.staging.some.domain/
HTTP/1.1 301 Moved Permanently
location: https://img.staging.some.domain/
date: Tue, 03 Sep 2019 18:27:30 GMT
server: istio-envoy
content-length: 0

HTTP/1.1 301 Moved Permanently
location: https://img.staging.some.domain/
date: Tue, 03 Sep 2019 18:27:30 GMT
server: istio-envoy
content-length: 0

HTTP/1.1 301 Moved Permanently
location: https://img.staging.some.domain/
date: Tue, 03 Sep 2019 18:27:30 GMT
server: istio-envoy
content-length: 0

HTTP/1.1 301 Moved Permanently
location: https://img.staging.some.domain/
date: Tue, 03 Sep 2019 18:27:31 GMT
server: istio-envoy
content-length: 0

HTTP/1.1 301 Moved Permanently
location: https://img.staging.some.domain/
date: Tue, 03 Sep 2019 18:27:31 GMT
server: istio-envoy
content-length: 0

Furthermore, if the definition of tls.httpsRedirect: true is removed from the Gateway, the service acts and behaves normally (there’s just no redirect from HTTP to HTTPS), meaning that both http://img.staging.some.domain/ and ttps://img.staging.some.domain/ simply reply to the request.

Any thoughts on this ?

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 17 (7 by maintainers)

Most upvoted comments

We are also having the same issue: Adding this to the virtual service fixed the issue. We are also on EKS with ACM. Istio 1.3.0

  http:
  - route:
    - destination:
        host: aaaaaaaa
        port:
          number: 8080
    headers:
      request:
        set:
          x-forwarded-proto: https
          x-forwarded-port: "443"

crhuber on Oct 4, 2019

@howardjohn I think I found out the reason why.

When you are using an AWS ELB with AWS Certificate Manager support that is set up at ELB level, COMBINED with protocol: HTTP for the istio gateway, it ALWAYS pushes the header x-forwarded-proto as http, causing a redirect loop because it’s continuously trying to update it to https, while keeping it as http after the request comes in 😃

IN SHORT, if the following conditions apply:

Gateway is set up as following for https:

    - hosts:
        - '*.staging.sip.onl'
      port:
        name: https
        number: 443
        protocol: HTTP

<AND> 2) Istio was deployed using this config for the *gateway* resource in helm:

gateways:
  istio-ingressgateway:
    serviceAnnotations:
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "SOME_CERTIFI_CATE_HERE"
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: 443

THEN THE FOLLOWING HAPPENS: The header x-forwarded-proto is always applied for each request, even if tls. httpsRedirect is set to true.

POSSIBLE SOLUTION:

IF config for tls.httpsRedirect is set to true, THEN first make sure that the x-forwarded-proto header is not already https 😃

PS: My logic may be stupid, but you get the idea…

mariusmitrofan on Sep 16, 2019