caddy: HTTP 520 Origin Error on Headscale behind Caddy behind Cloudflare

Hello there, I’m posting this issue on Caddy’s GitHub project page because it shows up only when deploying a specific version of this software, and never showed up before; however I do understand that after further investigation this discussion may be moved to the CLoudflare’s forum or even to Headscale’s GitHub page. That being said, I’ll do my best to explain the situation.

Minimal working example

The following Caddyfile, served through the Caddy 2.6.2 container image, works perfectly with Cloudflare’s strict TLS and Origin certificate enabled in front of it, and Headscale 0.20.0 behind it:

*.mydomain.stuff {
    tls  {
        dns cloudflare {env.CLOUDFLARE_API_TOKEN}
        client_auth {
            mode require_and_verify
            trusted_ca_cert_file /cert/origin-pull-ca.pem
        }
    }

    @headscale-app host vpn.mydomain.stuff

    handle @headscale-app {
        encode gzip
        reverse_proxy http://localhost:8080
    }
}

Reproduce the issue

Among these three components, upon changing the Caddy container to version 2.6.3 (also tested with 2.6.4) the following happens:

• on the Headscale server, the following error is logged INF The HTTP2 server was closed error=EOF
• on the Tailscale clients (everyone of them) the following error is logged: Received error: PollNetMap: Post "https://vpn.mydomain.stuff/machine/map": all connection attempts failed (HTTP: unexpected HTTP response: 301 Moved Permanently, HTTPS: unexpected HTTP response: 520 Origin Error)

By stopping Caddy, and restarting it through the 2.6.2 container these errors disappear without the need of restarting the server nor the clients.

Caddy debug logs

During the aforementioned unexpected behaviour on the clients and the server, using the debug level on the Caddy log only highlights the following lines, which do not ever show up if running the same minimal working example with Caddy <= 2.6.2:

2023-02-24T10:24:16.593275567+01:00 stderr F {"level":"debug","ts":1677230656.5929484,"logger":"http.stdlib","msg":"http: response.WriteHeader on hijacked connection from github.com/caddyserver/caddy/v2/modules/caddyhttp.(*responseRecorder).WriteHeader (responsewriter.go:191)"}

Impact

New Headscale clients are unable to join the network, nor a left client event will be notified to

Only with Headscale?!

Apparently this issue with HTTP error 520 affects only the Headscale application: I also tried the Caddy versions 2.6.3 and 2.6.4 with Nextcloud, Vaultwarden, Grocy and Jellyfin, and all of them are working as usual, at least from what I could see from the log files and by functionally testing each specific application in the browser.

Moreover, by pausing Cloudflare altogether for the site mydomain.stuff and then restarting Caddy 2.6.4, Headscale works again. At that point is even possible to turn back on the Cloudflare features for the site and no more HTTP 520 errors appear on the Tailscale clients logs.

Note: by “pausing” Cloudflare I am referring to the option named “Pause Cloudflare for this site” in the site overview page, and not the option to just disable proxying for that site: I also tried to just disable Cloudflare’s proxying for *.mydomain.stuff, but that does not solve the issue.