cloudflared: Failed to create quic connection and cause cloudflared container failed to run with 2022.4.0 on Docker

Thanks for those. I may have found something interesting, and we’ll pursue it internally.

For now, the gist is that cloudflared connects to 2 data-centers (for reliability, 2 connections in each). It looks like your cloudflared is unable to connect with QUIC to a specific data-center only. Since the 12th of April I see many successful QUIC connections to various data-centers, but I see HTTP2 connections only to that one specific data-center.

So, it seems like you’re using QUIC in 2 of the 4 connections. The logs that you see for falling back will generally be for connection index 0 and 2 (which connect to one anycast region), whereas connection index 1 and 3 will successfully connect with QUIC (to another anycast region).

Well. I was right.

@DCCInterstellar Publish Argument tunnel --protocol http2 --no-autoupdate run --token xxx

Thanks, it works!

I live in China and use cloudflare tunnel to hide my traffic amount other cloudflare users. QUIC completely fails this purpose as no other user connect to cloudflare in QUIC, so my connection was interrupted multiple times a day and I had to manually restart the container. By changing QUIC to http2, the problem is completely solved.

Edit in 23 Mar:

The same problem arises again recently and this time both http2 and quic failed. It seems like the traffic to cloudflare data center in USA was completely banned by GFW. For anyone who has the same problem, I had switched to use frp on my own server.

@DCCInterstellar Publish Argument tunnel --protocol http2 --no-autoupdate run --token xxx

I absolutely understand the frustration @darth-pika-hu

I can guarantee this is a problem with your network not allowing egress to 7844 UDP. E.g., this docker run --rm -it docker.io/cloudflare/cloudflared:latest tunnel --hello-world runs just fine from my infrastructure, and we can see thousand of other users that are doing the same just fine.

As noted above, you can force your Tunnel to run with http2 even though it is managed in the UI (and the UI does not yet allow to control that). Check out https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/remote-management/ for the details

Let me also reiterate on the reasoning behind this: we’re “forcing” quic protocol because we (Cloudflare) believe it is a big part of the future of the Internet. But many networks still block UDP. We must force admins behind those networks to feel that “pain” in some way, so that people are aware and begin allowing UDP egress. E.g., our Private DNS resolution, which uses UDP, only works with QUIC protocol. So it is frustrating for users to spin up Tunnels defaulting to http2 (that does not support UDP proxying) and not have Private DNS resolution working (see https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/private-net/private-hostnames-ips/#update-cloudflared)

We’ve uncovered that a small number of data-centers was indeed having this problem where they would not take in QUIC connections. We’ll fix that and post back, at which point all your connections (and not just half of them) should work with QUIC fine.

We’ll likely make a new release of cloudflared that fallsback to http2 from quic when this scenario happens. This is already the case normally when the quic protocol is picked automatically (and not configured by the user). We will make it so for Tunnels managed by the UI as well.

In practice we’ll want to promote quic usage, but this likely will need some tool to help troubleshoot this sort of scenarios, which are time consuming, and for which we do not currently have bandwidth to attack.

Here is my offer: What if I set up a virtual machine for you and let you do whatever you need to do? Let me know the best way to privatly contact you.

We still do not have any details about what your environment is like. There isn’t much to go on here really except empirical description.

Begin with a cloudflared Docker container on a Linux server, followed by a cloudflared installation file on a Windows 10 virtual machine and a Windows 11 virtual machine. Docker on the Linux server utilizes an AMD CPU, whereas the Windows 10 VM uses an INTEL CPU and Windows 11 uses an AMD CPU.

Edited on 04/11/2022: @sudarshan-reddy @nmldiegues Today is Monday, I’m at work, and I just used wireshark’s “udp.port==7844” filter to check the openvpn connection between the VM and the server. It is UDP and uses port 7844, as seen below:

Frame 26931: 1347 bytes on wire (10776 bits), 1347 bytes captured (10776 bits) on interface \Device\NPF_{4CA37862-9848-4728-9D2B-98465194344F}, id 0
    Interface id: 0 (\Device\NPF_{4CA37862-9848-4728-9D2B-98465194344F})
    Encapsulation type: Ethernet (1)
    Arrival Time: Apr 11, 2022 11:09:38.246542000 Pacific Daylight Time
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1649700578.246542000 seconds
    [Time delta from previous captured frame: 0.000127000 seconds]
    [Time delta from previous displayed frame: 0.000127000 seconds]
    [Time since reference or first frame: 410.840630000 seconds]
    Frame Number: 26931
    Frame Length: 1347 bytes (10776 bits)
    Capture Length: 1347 bytes (10776 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ip:udp:openvpn]
    [Coloring Rule Name: UDP]
    [Coloring Rule String: udp]
Ethernet II, Src: MS-NLB-PhysServer-17_32:22:20:d7 (02:11:32:22:20:d7), Dst: eero_3f:55:12 (c4:f1:74:3f:55:12)
    Destination: eero_3f:55:12 (c4:f1:74:3f:55:12)
    Source: MS-NLB-PhysServer-17_32:22:20:d7 (02:11:32:22:20:d7)
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 10.0.1.213, Dst: xxxxxxxxxxx
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 1333
    Identification: 0x6a30 (27184)
    Flags: 0x00
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 128
    Protocol: UDP (17)
    Header Checksum: 0x0000 [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 10.0.1.213
    Destination Address: xxxxxxxx
User Datagram Protocol, Src Port: 1194, Dst Port: 7844
    Source Port: 1194
    Destination Port: 7844
    Length: 1313
    Checksum: 0x8e2f [unverified]
    [Checksum Status: Unverified]
    [Stream index: 0]
    [Timestamps]
    UDP payload (1305 bytes)
OpenVPN Protocol
    Type: 0x48 [opcode/key_id]
    Peer ID: 0
    Data (1301 bytes)

And here’s the log for cloudflared on the 7844 port:

Frame 4170: 1294 bytes on wire (10352 bits), 1294 bytes captured (10352 bits) on interface \Device\NPF_{4CA37862-9848-4728-9D2B-98465194344F}, id 0
    Interface id: 0 (\Device\NPF_{4CA37862-9848-4728-9D2B-98465194344F})
    Encapsulation type: Ethernet (1)
    Arrival Time: Apr 11, 2022 11:21:40.311984000 Pacific Daylight Time
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1649701300.311984000 seconds
    [Time delta from previous captured frame: 0.000149000 seconds]
    [Time delta from previous displayed frame: 0.000149000 seconds]
    [Time since reference or first frame: 81.330803000 seconds]
    Frame Number: 4170
    Frame Length: 1294 bytes (10352 bits)
    Capture Length: 1294 bytes (10352 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ip:udp:quic:tls]
    [Coloring Rule Name: UDP]
    [Coloring Rule String: udp]
Ethernet II, Src: MS-NLB-PhysServer-17_32:22:20:d7 (02:11:32:22:20:d7), Dst: eero_3f:55:12 (c4:f1:74:3f:55:12)
    Destination: eero_3f:55:12 (c4:f1:74:3f:55:12)
    Source: MS-NLB-PhysServer-17_32:22:20:d7 (02:11:32:22:20:d7)
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 10.0.1.213, Dst: 198.41.200.193
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 1280
    Identification: 0x2f95 (12181)
    Flags: 0x40, Don't fragment
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 128
    Protocol: UDP (17)
    Header Checksum: 0x0000 [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 10.0.1.213
    Destination Address: 198.41.200.193
User Datagram Protocol, Src Port: 50246, Dst Port: 7844
    Source Port: 50246
    Destination Port: 7844
    Length: 1260
    Checksum: 0x9fbd [unverified]
    [Checksum Status: Unverified]
    [Stream index: 50]
    [Timestamps]
        [Time since first frame: 3.014733000 seconds]
        [Time since previous frame: 0.000149000 seconds]
    UDP payload (1252 bytes)
QUIC IETF
    QUIC Connection information
    [Packet Length: 1252]
    1... .... = Header Form: Long Header (1)
    .1.. .... = Fixed Bit: True
    ..00 .... = Packet Type: Initial (0)
    .... 00.. = Reserved: 0
    .... ..01 = Packet Number Length: 2 bytes (1)
    Version: 1 (0x00000001)
    Destination Connection ID Length: 11
    Destination Connection ID: 2f40a59f8028404f1afd5d
    Source Connection ID Length: 0
    Token Length: 0
    Length: 1231
    Packet Number: 8
    Payload: 01dafabdc667ed6f3c2c4f518da1dc9ae247c4b0ab9899c9e7a693fdeef9bbb40a8ecd2c…
    PADDING Length: 862
        Frame Type: PADDING (0x0000000000000000)
        [Padding Length: 862]
    CRYPTO
        Frame Type: CRYPTO (0x0000000000000006)
        Offset: 0
        Length: 347
        Crypto Data
        TLSv1.3 Record Layer: Handshake Protocol: Client Hello
            Handshake Protocol: Client Hello

Please advice

Same problem with 2023.05 version.

Here are my debug logs:

2023-05-13T13:53:12Z INF ICMP proxy will use 10.0.0.47 as source for IPv4

2023-05-13T13:53:12Z INF ICMP proxy will use fe80::ec90:adff:feb5:4a94 in zone eth0 as source for IPv6

2023-05-13T13:53:13Z DBG edge discovery: looking up edge SRV record domain=_v2-origintunneld._tcp.argotunnel.com event=0

2023-05-13T13:53:13Z DBG edge discovery: resolved edge addresses addresses=["198.41.192.227","198.41.192.7","198.41.192.27","198.41.192.107","198.41.192.47","198.41.192.57","198.41.192.37","198.41.192.67","198.41.192.167","198.41.192.77","2606:4700:a0::9","2606:4700:a0::1","2606:4700:a0::3","2606:4700:a0::4","2606:4700:a0::6","2606:4700:a0::5","2606:4700:a0::10","2606:4700:a0::7","2606:4700:a0::8","2606:4700:a0::2"] event=0

2023-05-13T13:53:13Z DBG edge discovery: resolved edge addresses addresses=["198.41.200.233","198.41.200.33","198.41.200.113","198.41.200.23","198.41.200.13","198.41.200.43","198.41.200.53","198.41.200.63","198.41.200.193","198.41.200.73","2606:4700:a8::4","2606:4700:a8::6","2606:4700:a8::9","2606:4700:a8::1","2606:4700:a8::5","2606:4700:a8::7","2606:4700:a8::10","2606:4700:a8::8","2606:4700:a8::3","2606:4700:a8::2"] event=0

2023-05-13T13:53:17Z INF Starting metrics server on [::]:2000/metrics

2023-05-13T13:53:17Z DBG edge discovery: looking up edge SRV record domain=_v2-origintunneld._tcp.argotunnel.com event=0

2023-05-13T13:53:18Z DBG edge discovery: resolved edge addresses addresses=["198.41.192.67","198.41.192.227","198.41.192.37","198.41.192.47","198.41.192.107","198.41.192.77","198.41.192.167","198.41.192.27","198.41.192.7","198.41.192.57","2606:4700:a0::5","2606:4700:a0::6","2606:4700:a0::4","2606:4700:a0::2","2606:4700:a0::8","2606:4700:a0::9","2606:4700:a0::7","2606:4700:a0::10","2606:4700:a0::3","2606:4700:a0::1"] event=0

2023-05-13T13:53:18Z DBG edge discovery: resolved edge addresses addresses=["198.41.200.193","198.41.200.73","198.41.200.33","198.41.200.13","198.41.200.63","198.41.200.233","198.41.200.53","198.41.200.23","198.41.200.113","198.41.200.43","2606:4700:a8::4","2606:4700:a8::1","2606:4700:a8::9","2606:4700:a8::2","2606:4700:a8::6","2606:4700:a8::5","2606:4700:a8::8","2606:4700:a8::3","2606:4700:a8::7","2606:4700:a8::10"] event=0

2023-05-13T13:53:18Z DBG edge discovery: giving new address to connection connIndex=0 event=0 ip=198.41.200.113

2023/05/13 13:53:18 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.

2023-05-13T13:53:24Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.113

2023-05-13T13:53:24Z DBG edge discovery: giving new address to connection available=19 connIndex=0 event=0 ip=198.41.200.233

2023-05-13T13:53:24Z INF Retrying connection in up to 2s connIndex=0 event=0 ip=198.41.200.113

2023-05-13T13:53:24Z DBG edge discovery: returning same edge address back to pool connIndex=0 event=0 ip=198.41.200.233

2023-05-13T13:53:29Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.233

2023-05-13T13:53:29Z DBG edge discovery: giving new address to connection available=19 connIndex=0 event=0 ip=198.41.200.53

2023-05-13T13:53:29Z INF Retrying connection in up to 4s connIndex=0 event=0 ip=198.41.200.233

2023-05-13T13:53:31Z DBG edge discovery: returning same edge address back to pool connIndex=0 event=0 ip=198.41.200.5

Failed to serve quic connection

What is quic connection? Quic is a new encrypted transport layer network protocol that makes HTTP traffic more secure, efficient, and faster. However, there are certain ISP’s do not support this protocol, which can interfere with your Cloudflare Tunneling.

There is a HIGH possibility that your ISP *doesn’t support this and your tunnel will continue to have these errors. To fix this error, there are two options.

OPTION 1: If your Cloudflared Template is using the latest Repository like below.

cloudflare/cloudflared:latest

Please, add another Variable. Add a name of your liking for the Variable.

Key: TUNNEL_TRANSPORT_PROTOCOL

There are four different values you can add, “auto, http2, h2mux, and quic.” (choose one)

• The “auto” value will automatically configure the quic protocol. If cloudflared is unable to establish connections with quic connection, it will fallback to using the http2 protocol. However, if you use auto as a value you will still get the Failed to serve quic connection error. This is a default function in cloudflared docker app. Use this value if your ISP supports quic connection. This runs on UDP (User Datagram Protocol).
• The “http2” value is the standard protocol that all ISP’s uses. It runs on TCP.
• The “h2mux” value is an unknown protocol that was used by cloudflared. Outdated.

In our case, we will be using the “http2” protocol to fix the quic connection error. ​

Once has been added. Restart the Container. The quic connection error should be resolved.

OPTION 2: In Cloudflared Template, change the Repository to:

cloudflare/cloudflared:2022.3.4

By changing this you will downgrade the Cloudflared Docker Container to 2022.3.4 version. It will use the http2 protocol but won’t have the latest security patches for the tunnel. Recommend using the latest version.

I have same problem with QUIC, I need it for route internal private network, fallback to https works but I can’t use it for previus motivation. This my log: 2023-02-24T19:21:08Z INF Starting metrics server on 127.0.0.1:49927/metrics 2023-02-24T19:21:13Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 ip=198.41.192.37 2023-02-24T19:21:13Z INF Retrying connection in up to 2s connIndex=0 ip=198.41.192.37 2023-02-24T19:21:19Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 ip=198.41.192.107 2023-02-24T19:21:19Z INF Retrying connection in up to 4s connIndex=0 ip=198.41.192.107 2023-02-24T19:21:27Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 ip=198.41.192.227 2023-02-24T19:21:27Z INF Retrying connection in up to 8s connIndex=0 ip=198.41.192.227 2023-02-24T19:21:32Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 ip=198.41.200.73 2023-02-24T19:21:32Z INF Retrying connection in up to 16s connIndex=0 ip=198.41.200.73 2023-02-24T19:21:43Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 ip=198.41.192.167 2023-02-24T19:21:43Z INF Retrying connection in up to 32s connIndex=0 ip=198.41.192.167 2023-02-24T19:21:58Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 ip=198.41.192.37 2023-02-24T19:21:58Z INF Retrying connection in up to 1m4s connIndex=0 ip=198.41.192.37 2023-02-24T19:22:23Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 ip=198.41.200.233 2023-02-24T19:22:23Z INF Retrying connection in up to 1m4s connIndex=0 ip=198.41.200.233 2023-02-24T19:22:41Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 ip=198.41.200.53 2023-02-24T19:22:41Z INF Retrying connection in up to 1m4s connIndex=0 ip=198.41.200.53 2023-02-24T19:23:01Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 ip=198.41.200.73 2023-02-24T19:23:01Z INF Retrying connection in up to 1m4s connIndex=0 ip=198.41.200.73 2023-02-24T19:23:26Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with quicprotocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/ If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not work unless your cloudflared can connect with Cloudflare Network withquic. connIndex=0 ip=198.41.200.73 2023-02-24T19:23:26Z INF Switching to fallback protocol http2 connIndex=0 ip=198.41.200.73 2023-02-24T19:23:27Z INF Connection 0052f011-91d6-4a8b-a1e2-96b174ae8269 registered with protocol: http2 connIndex=0 ip=198.41.200.13 location=FCO 2023-02-24T19:23:27Z INF Connection 7243036b-689e-4774-9347-df7ccc4ac6db registered with protocol: http2 connIndex=1 ip=198.41.192.7 location=MXP 2023-02-24T19:23:27Z INF Warp-routing is enabled 2023-02-24T19:23:27Z INF Updated to new configuration config="{\"warp-routing\":{\"enabled\":true}}" version=1 2023-02-24T19:23:28Z INF Connection b88a9007-b4f4-4a75-9c07-1fb8a28d679a registered with protocol: http2 connIndex=2 ip=198.41.200.193 location=FCO 2023-02-24T19:23:30Z INF Connection 45bf6fc2-2a7d-4cb6-94bc-10ae8cb3044c registered with protocol: http2 connIndex=3 ip=198.41.192.27 location=MXP

@nmldiegues and @sudarshan-reddy Want to give you guys a heads up. QUIC starts working suddenly. Update: stopped working again.

Well, we certainly haven’t done anything over the weekend.

@nmldiegues Thank you for providing an update. Now I finally realized we were just white mice to you guys. All the changes you guys made are just for your goal or “the future” not for current users. The solution to the problem? Sorry, we are too busy and don’t care.

@sudarshan-reddy Here is the tcpdump log generated while openvpn client on the Windows Virtual Machine connected to the server:

listening on \Device\{4CA37862-9848-4728-9D2B-98465194344F}, link-type EN10MB (Ethernet), capture size 262144 bytes
15:06:05.655120 IP 10.0.1.213.1194 > xxxx.7844: UDP, length 77
15:06:05.998815 IP 10.0.1.213.1194 > xxxx.7844: UDP, length 77
15:06:06.188366 IP xxxx.7844 > 10.0.1.213.1194: UDP, length 77
15:06:06.188366 IP 10.0.1.213.1194 > xxxx.7844: UDP, length 65
15:06:06.201661 IP 10.0.1.213.1194 > xxxx.7844: UDP, length 346
15:06:06.389166 IP xxxx.7844 > 10.0.1.213.1194: UDP, length 65
15:06:06.398070 IP xxxx.7844 > 10.0.1.213.1194: UDP, length 1422
15:06:06.398070 IP xxxx.7844 > 10.0.1.213.1194: UDP, length 983
15:06:06.398070 IP 10.0.1.213.1194 > xxxx.7844: UDP, length 65

The following is the tcpdump log generated while cloudflared attempted to connect through QUIC:

listening on \Device\{4CA37862-9848-4728-9D2B-98465194344F}, link-type EN10MB (Ethernet), capture size 262144 bytes
11:48:25.201412 IP 10.0.1.213.61706 > 198.41.200.113.7844: UDP, length 1252
11:48:25.410162 IP 10.0.1.213.61706 > 198.41.200.113.7844: UDP, length 1252
11:48:25.410162 IP 10.0.1.213.61706 > 198.41.200.113.7844: UDP, length 1252
11:48:25.813411 IP 10.0.1.213.61706 > 198.41.200.113.7844: UDP, length 1252
11:48:25.813411 IP 10.0.1.213.61706 > 198.41.200.113.7844: UDP, length 1252
11:48:26.617403 IP 10.0.1.213.61706 > 198.41.200.113.7844: UDP, length 1252
11:48:26.617403 IP 10.0.1.213.61706 > 198.41.200.113.7844: UDP, length 1252
11:48:28.217673 IP 10.0.1.213.61706 > 198.41.200.113.7844: UDP, length 1252
11:48:28.217673 IP 10.0.1.213.61706 > 198.41.200.113.7844: UDP, length 1252

PS: configuring tcpdump on Windows is a hassle.

Please let me know if you are interested in my proposal:

What if I set up a virtual machine for you and let you do whatever you need to do? Let me know the best way to privatly contact you.