cloudflared: Failed to serve quic connection

not sure if related but in docker container suddenly cloudflared says Serve tunnel error error="failed to accept QUIC stream: Application error 0x0 (remote)

We’re seeing this too. Though it’s potentially related to this: https://www.cloudflarestatus.com/incidents/s1hkh315y9s9

Edit: Seems to be working again as of resolution of this incident.

2022-11-22T18:55:12Z INF Starting tunnel tunnelID=83e89690-936f-4d28-a2d0-8abcc4473e71
2022-11-22T18:55:12Z INF Version 2022.10.3
2022-11-22T18:55:12Z INF GOOS: linux, GOVersion: go1.18.6, GoArch: amd64
2022-11-22T18:55:12Z INF Settings: map[config:.cloudflared/config.yml cred-file:/home/x/.cloudflared/83e89690-936f-4d28-a2d0-8abcc4473e71.json credentials-file:/home/x/.cloudflared/83e89690-936f-4d28-a2d0-8abcc4473e71.json url:http://127.0.0.1:80]
2022-11-22T18:55:12Z INF cloudflared will not automatically update if installed by a package manager.
2022-11-22T18:55:12Z INF Generated Connector ID: 1453508b-213a-4109-8116-a8d2d4f9dfb1
2022-11-22T18:55:12Z INF Initial protocol quic
2022-11-22T18:55:12Z INF ICMP proxy will use 192.168.0.x as source for IPv4
2022-11-22T18:55:12Z INF ICMP proxy will use fe80::96de:x:x:x in zone enp2s0 as source for IPv6
2022-11-22T18:55:12Z INF Starting metrics server on 127.0.0.1:46867/metrics
2022/11/22 18:55:12 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
2022-11-22T18:55:12Z WRN Your version 2022.10.3 is outdated. We recommend upgrading it to 2022.11.0
2022-11-22T18:55:22Z ERR Failed to serve quic connection error="Application error 0x0" connIndex=0 ip=198.41.x.x
2022-11-22T18:55:22Z ERR Serve tunnel error error="Application error 0x0" connIndex=0 ip=198.41.x.x
2022-11-22T18:55:22Z INF Retrying connection in up to 2s connIndex=0 ip=198.41.x.x
2022-11-22T18:55:24Z INF Tunnel server stopped
2022-11-22T18:55:24Z ERR Initiating shutdown error="Application error 0x0"
2022-11-22T18:55:24Z ERR icmp router terminated error="context canceled"
2022-11-22T18:55:24Z INF Metrics server stopped
Application error 0x0

I’m having this issue rn. I literally changed nothing.

Same issue 2023-11-14T04:39:53Z ERR Failed to serve quic connection error=“timeout: no recent network activity” connIndex=0 event=0

We’ll close the issue for now. We’d appreciate a new one being open with new evidence, should it happen, so we do not conflate a lot of confusing evidence in the same thread over a long period of time.

We have done some work/fixes for QUIC protocol transport in both cloudflared and our edge. Would you mind letting us know how this goes with latest cloudflared version 2022.2.2?

I can confirm that cloudflared version 2022.2.0 (built 2022-02-04-1113 UTC) also has this problem. I encountered the same issue a few days ago.

Adding a bit more context here… I think this is due to when a backend issues a TCP RST (or hangs too long)… In my deployment, it is like so:

Cloudflared <-> Contour Envoy <-> Backend

When the backend takes too long to reply, Envoy breaks the backend connection (TCP RST), but Cloudflared doesn’t realize it and then the entire flow breaks as well as clients cannot connect (origin timeouts). I even created multiple origin tunnels (Cloudflare Load Balancer) and still, even one connection having this issue will break all origin tunnels.

PSA: If this happens on http2, the Cloudflare Edge terminates the origin connection, which is quite annoying if you have a shell to your server, because that is destroyed as well.

in my case this happens after i sent many http requests (like seeking through an hls video feed quickly) in a short amount of time then it will give 524. if i switch my network (client device) from wifi to mobile data for example then it starts responding again and if i switched back to the same wifi then it gives me 524 again. so it seems that 524 is tied to one type of connection.