kubernetes: ELB L7 listeners not valid for mixed http/https

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.): No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Everything ‘ELB’ for past few months

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report

Kubernetes version (use kubectl version): 1.4.6

Environment:

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): Jessie
• Kernel (e.g. uname -a): 4.4.26-k8s
• Install tools: kops 1.4.1
• Others:

What happened:

My goal was to expose a k8s Service using an AWS ELB, using both HTTP and HTTPS. HTTPS should use an AWS cert, passing through both HTTP and HTTPS to an nginx container.

Scenario A:

With these three options set: service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xxxxxx service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https

The listeners created were: TCP/80 -> TCP/(nodeport for container’s 80) HTTPS/443 -> HTTPS/(nodeport for container’s 443), using AWS cert

While this does move traffic on both 80 and 443, the source IP is lost on port 80. 443 works properly.

Scenario B:

Backend changed to http, otherwise the same: service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http

The listeners created this time were: HTTP/80 -> HTTP/(nodeport for container’s 80) HTTPS/443 -> HTTP/(nodeport for container’s 443), using AWS cert

In this instance, 80 works properly, but 443 is completely broken as it’s attempting to deliver HTTP traffic to the container over port 443.

What you expected to happen:

In scenario A (backend-protocol: https), I expected: HTTP/80 -> HTTP/(nodeport for container’s 80) [or alternatively, -> HTTPS/(nodeport for 443)] HTTPS/443 -> HTTPS/(nodeport for container’s 443)

In short, I expected HTTP on 80, not TCP, which would enable proper L7 on both ports.

In scenario B (backend-protocol: http), I expected: HTTP/80 -> HTTP/(nodeport for container’s 80) HTTPS/443 -> HTTP/(nodeport for container’s 80)

Here I expected the backend HTTP to be directed to the NodePort that forwards to container’s port 80 in both cases.

In my mind, in the presence of aws-load-balancer-ssl-cert, k8s already assumes traffic should now be L7. I expect L7 (and the benefits of the X-Forwarded-* set of headers) to happen on both 80 and 443, not just 443.

Further, I expect the Instance Protocol and the NodePort assigned to Instance Port to match, and not get HTTP traffic directed to a NodePort for HTTPS, or vice versa.

How to reproduce it (as minimally and precisely as possible): I installed Kubernetes via Kops 1.4.1, targeting K8s 1.4.6. Then I built a simple nginx container (that exposes 80 + 443) and used the following Service definition:

apiVersion: v1
kind: Service
metadata:
  name: web
  labels:
    app: web
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xxxxxx
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
      # or http
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    protocol: TCP
  - name: https
    port: 443
    protocol: TCP
  selector:
    app: web

Anything else do we need to know:

Ideally, I’d like backend-protocol of both http and https to be working, as we do run some sensitive services that require https. If only https worked, we’d accept the internal slowdown on less critical services.