cloudflared: šIntermittent outages
Describe the bug I have a Cloudflare tunnel set up. Recently (maybe related to updating to 2022.7.1?) the tunnel disconnects from all four connections, preventing me from connecting at all. If I restart Cloudflare it works again, but goes down after a random amount of time. I suspect itās related to
- quic
- china firewall (the server is in China)
To Reproduce Steps to reproduce the behavior:
- Very basic Cloudflare tunnel config (the same config works on 2 other machines outside of China with no issue)
- Verify ports are fine as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/ (all tests report no errors)
- Run Cloudflared as a system service, tunnel works upon start / restart
- After a few hours, the tunnel goes down
- I get these errors in the log file (for all 4 active connections):
ERR Failed to create new quit connection error="failed to dial to edge with quick: timeout: no recent network activity" connIndex=2
ERR Connection terminated error="failed to dial to edge with quick: timeout: no recent network activity" connIndex=2
Expected behavior I expect the tunnel to go up and stay up barring exceptional circumstances
Environment and versions
- OS: Ubuntu 20.04
- Architecture: x64
- Version: 2022.07.1
Logs and errors Logs included above.
Additional context As I said, server is in China so it might be a GFW issue. However, the tunnel was quite stable up until this month, so it might be related to 2022.7.1. As a bandaid, Iām trying high values for Connection Timeout and Idle Connection Expiration Time, but doubling those to 60 and 60 didnāt resolve the issue.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 22 (10 by maintainers)
Oh. That does sound very promising! Iāll build from that commit over the weekend and do some testing on one of the origin machines that has been running into the issue consistently (due to the Internet upstream on that box going down for up to an hour at a time quite frequently)
Thanks for checking @cwegener ! Itās a real delight when we have members of the community not only helpfully reporting problems but willing to beta test them. ā¤ļø
https://github.com/cloudflare/cloudflared/commit/e3390fcb15c52a8e2a33fd9002680852b12e2f67 is the fix Weāll make a release early next week. In the meantime if you want to test this or want this feature, feel free to build from source.
@abelinkinbio Yup. Will do. Iāll try and do some tests that can produce the issue at will (since I normally have to wait for my upstream Internet provider to have an outage in order for the issue to occur, but I think I can simulate the upstream outage myself.)
You should have likely had this bug with
2022.7.0. We fixed the fall back on2022.7.1.Glad to hear your problem is fixed. Feel free to open an issue if you have further issues. (And perhaps give QUIC a try too since itās going to be our stable protocol going forwards!)