cloudflared: vnc failing to connect with "websocket: bad handshake"

I think the error is a result of incorrect ingress rule parsing.

The server cloudflared is using config.yaml:

tunnel: redacted
credentials-file: /etc/cloudflared/redacted.json
origincert: /etc/cloudflared/cert.pem

ingress:
  - hostname: redacted.net
    path: /vnc/5901
    service: tcp://localhost:5901
... # multiple `/vnc/<port>` per user, there are 9 skipped here
  - hostname: redacted.net
    service: ssh://localhost:22
  - service: http_status:404

The server cloudflared is using the wrong ingress rule. It should be using ingress rule 0 but it is using 10:

{"level":"debug","time":"2021-04-30T17:15:59Z","message":"CF-RAY: 64826a4e3bb0057d-LAX GET http://localhost:8080/favicon.ico HTTP/1.1"}
{"level":"debug","time":"2021-04-30T17:15:59Z","message":"CF-RAY: 64826a4e3bb0057d-LAX Request Headers map[Accept:[image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8] Accept-Encoding:[gzip] Accept-Language:[en-US,en;q=0.9] Cdn-Loop:[cloudflare] Cf-Access-Authenticated-User-Email:[redacted] Cf-Access-Jwt-Assertion:[redacted] Cf-Cloudflared-Proxy-Tunnel-Hostname:[redacted] Cf-Connecting-Ip:[redacted] Cf-Ipcountry:[US] Cf-Ray:[64826a4e3bb0057d-LAX] Cf-Request-Id:[09c560c4e40000057d473f8000000001] Cf-Visitor:[{\"scheme\":\"https\"}] Cf-Warp-Tag-Id:[redacted] Cookie:[redacted] Sec-Ch-Ua:[\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\"] Sec-Ch-Ua-Mobile:[?0] Sec-Fetch-Dest:[image] Sec-Fetch-Mode:[no-cors] Sec-Fetch-Site:[same-origin] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36] X-Forwarded-For:[redacted] X-Forwarded-Proto:[https]]"}
{"level":"debug","time":"2021-04-30T17:15:59Z","message":"CF-RAY: 64826a4e3bb0057d-LAX Serving with ingress rule 10"}
{"level":"debug","time":"2021-04-30T17:15:59Z","message":"CF-RAY: 64826a4e3bb0057d-LAX Request content length 0"}
{"level":"error","time":"2021-04-30T17:15:59Z","message":"localhost:22 is not a http service"}
{"level":"error","time":"2021-04-30T17:15:59Z","message":"CF-RAY: 64826a4e3bb0057d-LAX Proxying to ingress 10 error: Not a http service"}

I am using the following on the client to open a VNC client with the respective error messages:

$ cloudflared access tcp --hostname redacted.net/vnc/5901 --url localhost:5901
A browser window should have opened at the following URL:

https://redacted.net/cdn-cgi/access/cli?...

If the browser failed to open, please visit the URL above directly in your browser.
2021-04-30T17:16:01Z ERR failed to connect to origin error="websocket: bad handshake" originURL=https://redacted.net/vnc/5901

$ vncviewer localhost:5901

---

The verbose logs are also available on support ticket #2138135.