ingress-nginx: Websockets not working on AWS w/ ingress-nginx helm package

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

NGINX Ingress controller version: 0.9.0-beta15 (what comes with the latest stable/nginx-ingress helm package as of 2017-12-12)

Kubernetes version (use kubectl version): 1.8.5

Environment:

  • Cloud provider or hardware configuration: AWS
  • OS (e.g. from /etc/os-release): CoreOS 1576.4.0
  • Kernel (e.g. uname -a): 4.13.16
  • Install tools: kops
  • Others: helm

Steps to reproduce:

  1. Create k8s cluster on AWS
    1. kops create cluster --cloud=aws --state=s3://state-bucket --node-count=3 --kubernetes-version=1.8.5 --zones=us-west-2a,us-west-2b,us-west-2c --master-zones=us-west-2a,us-west-2b,us-west-2c --dns-zone=example.com --node-size=m4.2xlarge --master-size=t2.large --ssh-public-key=~/.ssh/key.pub --image=coreos.com/CoreOS-stable-1576.4.0-hvm k8s.example.com --yes
    2. Wait until kops validate cluster says it’s ready
  2. Install nginx-ingress helm package
    1. helm init
    2. Setup basic SSL and timeout params for ingress controller
      echo '
      controller:
        config:
          ssl-redirect: "false"
          proxy-read-timeout: "3600"
          proxy-send-timeout: "3600"
        replicaCount: 3
        service:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:my-acm-cert"
            service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
            service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
      ' > ingress-values.yaml
      
    3. Wait until tiller is ready
    4. helm install stable/nginx-ingress --values ingress-values.yaml --name my-ingress-controller
  3. Deploy example websocket echo service
    1. Use this example
      1. …but don’t forget to change the websocket.uswest2-01.rocket-science.io host param to something pointing at your AWS ELB
  4. Browse to http://www.websocket.org/echo.html and attempt to connect to any of ws://your-elb-pointing-host.example.com/echo, ws://your-elb-pointing-host.example.com/, wss://your-elb-pointing-host.example.com/echo, or wss://your-elb-pointing-host.example.com.

Expected behavior:

The echo client would connect successfully, the Send Text button would become active, clicking it would send the Message string to the example websocket service, and the Message string would be echoed back by the example websocket service.

Observed behavior:

The echo client connects, the Send Text button remains inactive, the browser console reports a 400 error, and the connection is closed a few seconds later. Logs from http://demos.kaazing.com/echo/:

CONNECT: ws://admin-api-k8s.turbovote.org/echo
CLOSED: (1006)
CONNECT: ws://admin-api-k8s.turbovote.org/
CLOSED: (1006)
CONNECT: wss://admin-api-k8s.turbovote.org/
CLOSED: (1006)
CONNECT: wss://admin-api-k8s.turbovote.org/echo
CLOSED: (1006)

Example log messages from nginx ingress controllers:

  • 172.20.71.55 - [172.20.71.55] - - [12/Dec/2017:19:12:17 +0000] "GET /?.kr=xs HTTP/1.1" 400 271 "http://demos.kaazing.com/echo/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7" 493 0.000 [] - - - -

  • 172.20.112.61 - [172.20.112.61] - - [12/Dec/2017:19:17:53 +0000] "GET /?.kl=Y HTTP/1.1" 400 673 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36" 587 0.000 [] - - - -

  • 172.20.37.239 - [172.20.37.239] - - [12/Dec/2017:19:17:07 +0000] "GET /echo?.kl=Y HTTP/1.1" 400 673 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36" 611 0.000 [] - - - -

No output is seen in the websocket pods’ logs.

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 16 (9 by maintainers)

Most upvoted comments

For others looking to solve their WebSockets/Ingress issues, I created a checklist here: https://gist.github.com/jsdevtom/7045c03c021ce46b08cb3f41db0d76da#file-ingress-service-yaml

@0verc1ocker Nope, had to ditch websockets over https/ssl

@aledbf Thanks for that detail. Although it doesn’t work w/o TLS either, so there’s still something amiss here.

In the wss:// case I need to terminate TLS in the ELB because I’m using an ACM certificate (so I don’t have access to it).

So now I’m wondering if…

  1. The behavior I’m seeing with the ws:// protocol is still a bug.
  2. This can turn into a feature request for supporting ELB-terminated TLS for wss:// connections.
  3. This should be noted somewhere in the docs for ingress-nginx because this is a pretty big exception to “Support for websockets is provided by NGINX out of the box. No special configuration required.”