ingress-nginx: Websockets not working on AWS w/ ingress-nginx helm package
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT
NGINX Ingress controller version: 0.9.0-beta15 (what comes with the latest stable/nginx-ingress helm package as of 2017-12-12)
Kubernetes version (use kubectl version): 1.8.5
Environment:
- Cloud provider or hardware configuration: AWS
- OS (e.g. from /etc/os-release): CoreOS 1576.4.0
- Kernel (e.g.
uname -a): 4.13.16 - Install tools: kops
- Others: helm
Steps to reproduce:
- Create k8s cluster on AWS
kops create cluster --cloud=aws --state=s3://state-bucket --node-count=3 --kubernetes-version=1.8.5 --zones=us-west-2a,us-west-2b,us-west-2c --master-zones=us-west-2a,us-west-2b,us-west-2c --dns-zone=example.com --node-size=m4.2xlarge --master-size=t2.large --ssh-public-key=~/.ssh/key.pub --image=coreos.com/CoreOS-stable-1576.4.0-hvm k8s.example.com --yes- Wait until
kops validate clustersays it’s ready
- Install nginx-ingress helm package
helm init- Setup basic SSL and timeout params for ingress controller
echo ' controller: config: ssl-redirect: "false" proxy-read-timeout: "3600" proxy-send-timeout: "3600" replicaCount: 3 service: annotations: service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:my-acm-cert" service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443" service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600" ' > ingress-values.yaml - Wait until tiller is ready
helm install stable/nginx-ingress --values ingress-values.yaml --name my-ingress-controller
- Deploy example websocket echo service
- Use this example
- …but don’t forget to change the
websocket.uswest2-01.rocket-science.iohost param to something pointing at your AWS ELB
- …but don’t forget to change the
- Use this example
- Browse to
http://www.websocket.org/echo.htmland attempt to connect to any ofws://your-elb-pointing-host.example.com/echo,ws://your-elb-pointing-host.example.com/,wss://your-elb-pointing-host.example.com/echo, orwss://your-elb-pointing-host.example.com.
Expected behavior:
The echo client would connect successfully, the Send Text button would become active, clicking it would send the Message string to the example websocket service, and the Message string would be echoed back by the example websocket service.
Observed behavior:
The echo client connects, the Send Text button remains inactive, the browser console reports a 400 error, and the connection is closed a few seconds later.
Logs from http://demos.kaazing.com/echo/:
CONNECT: ws://admin-api-k8s.turbovote.org/echo
CLOSED: (1006)
CONNECT: ws://admin-api-k8s.turbovote.org/
CLOSED: (1006)
CONNECT: wss://admin-api-k8s.turbovote.org/
CLOSED: (1006)
CONNECT: wss://admin-api-k8s.turbovote.org/echo
CLOSED: (1006)
Example log messages from nginx ingress controllers:
-
172.20.71.55 - [172.20.71.55] - - [12/Dec/2017:19:12:17 +0000] "GET /?.kr=xs HTTP/1.1" 400 271 "http://demos.kaazing.com/echo/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7" 493 0.000 [] - - - - -
172.20.112.61 - [172.20.112.61] - - [12/Dec/2017:19:17:53 +0000] "GET /?.kl=Y HTTP/1.1" 400 673 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36" 587 0.000 [] - - - - -
172.20.37.239 - [172.20.37.239] - - [12/Dec/2017:19:17:07 +0000] "GET /echo?.kl=Y HTTP/1.1" 400 673 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36" 611 0.000 [] - - - -
No output is seen in the websocket pods’ logs.
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 16 (9 by maintainers)
For others looking to solve their WebSockets/Ingress issues, I created a checklist here: https://gist.github.com/jsdevtom/7045c03c021ce46b08cb3f41db0d76da#file-ingress-service-yaml
@0verc1ocker Nope, had to ditch websockets over https/ssl
@aledbf Thanks for that detail. Although it doesn’t work w/o TLS either, so there’s still something amiss here.
In the
wss://case I need to terminate TLS in the ELB because I’m using an ACM certificate (so I don’t have access to it).So now I’m wondering if…
ws://protocol is still a bug.wss://connections.