caddy: reverse_proxy hangs on wrong URL in browsers

If you have a way to repro a curl problem, I will certainly be interested in knowing exactly how so that we can fix it!

Okie dokie, I took a few minutes to try this out. And it works fine in my browser (which uses HTTP/2 and HTTP/3):

2022/11/09 05:02:47.505	INFO	http.log.access	handled request	{"request": {"remote_ip": "127.0.0.1", "remote_port": "40376", "proto": "HTTP/2.0", "method": "GET", "host": "keystonenetwork.localhost", "uri": "/", "headers": {"Cache-Control": ["no-cache"], "Te": ["trailers"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"], "Accept-Language": ["en-US,en;q=0.5"], "Accept-Encoding": ["gzip, deflate, br"], "Upgrade-Insecure-Requests": ["1"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-User": ["?1"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/106.0"], "Dnt": ["1"], "Sec-Fetch-Dest": ["document"], "Sec-Fetch-Site": ["none"], "Pragma": ["no-cache"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "keystonenetwork.localhost"}}, "user_id": "", "duration": 0.022696275, "size": 5216, "status": 200, "resp_headers": {"Accept-Ranges": ["bytes"], "Vary": ["Accept-Encoding,x-wf-forwarded-proto"], "Content-Length": ["5216"], "Server": ["Caddy", "openresty"], "X-Served-By": ["cache-iad-kiad7000101-IAD, cache-den8246-DEN"], "Content-Encoding": ["gzip"], "X-Timer": ["S1667970167.489611,VS0,VE0"], "Alt-Svc": ["h3=\":443\"; ma=2592000"], "Via": ["1.1 varnish, 1.1 varnish"], "Date": ["Wed, 09 Nov 2022 05:02:47 GMT"], "Content-Type": ["text/html"], "Age": ["203"], "X-Cache-Hits": ["1, 4"], "Content-Security-Policy": ["frame-ancestors 'self' https://*.webflow.com http://*.webflow.com http://*.webflow.io http://webflow.com https://webflow.com"], "X-Cache": ["HIT, HIT"]}}

Curl also works.

So, I’m not entirely sure what the problem is, but I don’t think it’s in Caddy? Maybe something in your network is dropping or modifying packets.

@mholt, I built curl with http/3 support and got something interesting.

Run: curl3 --http3 -v "https://keystonenetwork.localhost/sito/wp-includes/wlwmanifest.xml"

output:

*   Trying 127.0.0.1:443...
* Connected to keystonenetwork.localhost (127.0.0.1) port 443 (#0)
* h2h3 [:method: GET]
Not enough setting.
* h2h3 [:path: /sito/wp-includes/wlwmanifest.xml]
* h2h3 [:scheme: https]
* h2h3 [:authority: keystonenetwork.localhost]
* h2h3 [user-agent: curl/7.86.0]
* h2h3 [accept: */*]
> GET /sito/wp-includes/wlwmanifest.xml HTTP/3
> Host: keystonenetwork.localhost
> user-agent: curl/7.86.0
> accept: */*
> 
* request aborted
* Connection #0 to host keystonenetwork.localhost left intact
curl: (95) request aborted

Caddy’s log:

2022/11/07 19:00:04.307	DEBUG	events	event	{"name": "tls_get_certificate", "id": "4d2cba26-d81a-4d0a-8d6c-a2a95db52241", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4866,4867,4865,255],"ServerName":"keystonenetwork.localhost","SupportedCurves":[29,23,30,25,24],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537],"SupportedProtos":["h3","h3-29"],"SupportedVersions":[772],"Conn":{}}}}
2022/11/07 19:00:04.309	DEBUG	tls.handshake	choosing certificate	{"identifier": "keystonenetwork.localhost", "num_choices": 1}
2022/11/07 19:00:04.309	DEBUG	tls.handshake	default certificate selection results	{"identifier": "keystonenetwork.localhost", "subjects": ["keystonenetwork.localhost"], "managed": true, "issuer_key": "local", "hash": "46a82935921296123aeeda22d45b865351954ce33fe9f6c277b2e9ac700a50e0"}
2022/11/07 19:00:04.310	DEBUG	tls.handshake	matched certificate in cache	{"remote_ip": "::1", "remote_port": "56809", "subjects": ["keystonenetwork.localhost"], "managed": true, "expiration": "2022/11/08 01:46:35.000", "hash": "46a82935921296123aeeda22d45b865351954ce33fe9f6c277b2e9ac700a50e0"}
2022/11/07 19:00:04.339	DEBUG	http.handlers.reverse_proxy	selected upstream	{"dial": "35.208.143.249:443", "total_upstreams": 1}
2022/11/07 19:00:06.482	DEBUG	http.handlers.reverse_proxy	upstream roundtrip	{"upstream": "35.208.143.249:443", "duration": 2.142127939, "request": {"remote_ip": "::1", "remote_port": "56809", "proto": "HTTP/3.0", "method": "GET", "host": "www.keystonenetwork.com", "uri": "/sito/wp-includes/wlwmanifest.xml", "headers": {"X-Forwarded-For": ["::1"], "X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["keystonenetwork.localhost"], "User-Agent": ["curl/7.86.0"], "Accept": ["*/*"]}, "tls": {"resumed": false, "version": 0, "cipher_suite": 0, "proto": "", "server_name": ""}}, "headers": {"Date": ["Mon, 07 Nov 2022 19:01:59 GMT"], "Vary": ["Accept-Encoding"], "Host-Header": ["6b7412fb82ca5edfd0917e3957f05d89"], "X-Content-Type-Options": ["nosniff"], "X-Xss-Protection": ["1; mode=block"], "Link": ["<https://www.keystonenetwork.com/wp-json/>; rel=\"https://api.w.org/\""], "X-Proxy-Cache": ["MISS"], "Server": ["nginx"], "Expires": ["Wed, 11 Jan 1984 05:00:00 GMT"], "Cache-Control": ["no-cache, must-revalidate, max-age=0"], "X-Httpd-Modphp": ["1"], "X-Proxy-Cache-Info": ["0 NC:000000 UP:SKIP_CACHE_NO_CACHE"], "Content-Type": ["text/html; charset=UTF-8"], "X-Cache-Enabled": ["True"]}, "status": 404}
2022/11/07 19:00:11.487	ERROR	http.handlers.reverse_proxy	aborting with incomplete response	{"error": "timeout: no recent network activity"}

curl version

curl 7.86.0 (x86_64-pc-linux-gnu) libcurl/7.86.0 OpenSSL/1.1.1q zlib/1.2.13 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.3 libpsl/0.21.1 (+libidn2/2.3.0) libssh2/1.10.0 nghttp2/1.50.0
Release-Date: 2022-10-26
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

Caddy’s log when a request hangs in browser:

2022/11/07 18:04:10.844	DEBUG	events	event	{"name": "tls_get_certificate", "id": "cd3c37a3-0898-4dae-b2b4-5975c0e4deac", "origin": "tls", "data": {"client_hello":{"CipherSuites":[43690,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"keystonenetwork.localhost","SupportedCurves":[64250,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[51914,772,771],"Conn":{}}}}
2022/11/07 18:04:10.845	DEBUG	tls.handshake	choosing certificate	{"identifier": "keystonenetwork.localhost", "num_choices": 1}
2022/11/07 18:04:10.845	DEBUG	tls.handshake	default certificate selection results	{"identifier": "keystonenetwork.localhost", "subjects": ["keystonenetwork.localhost"], "managed": true, "issuer_key": "local", "hash": "46a82935921296123aeeda22d45b865351954ce33fe9f6c277b2e9ac700a50e0"}
2022/11/07 18:04:10.845	DEBUG	tls.handshake	matched certificate in cache	{"remote_ip": "::1", "remote_port": "48252", "subjects": ["keystonenetwork.localhost"], "managed": true, "expiration": "2022/11/08 01:46:35.000", "hash": "46a82935921296123aeeda22d45b865351954ce33fe9f6c277b2e9ac700a50e0"}
2022/11/07 18:04:10.850	DEBUG	http.handlers.reverse_proxy	selected upstream	{"dial": "35.208.143.249:443", "total_upstreams": 1}

@mholt, but with curl this bug doesn’t reproduce. I have this problem only in browser (Brave 1.45.118 Chromium: 107.0.5304.91). I think, the problem is with http/2 or http/3 only. I ready to provide more info if needed. But which info will be really helpful?

Thanks for opening an issue! We’ll look into this.

It’s not immediately clear to me what is going on, so I’ll need your help to understand it better. Specifically, logs and a relevant curl -v command with its output.

Ideally, we need to be able to reproduce the bug in the most minimal way possible. This allows us to write regression tests to verify the fix is working. If we can’t reproduce it, then you’ll have to test our changes for us until it’s fixed – and then we can’t add test cases, either.

I’ve attached a template below that will help make this easier and faster! This will require some effort on your part – please understand that we will be dedicating time to fix the bug you are reporting if you can just help us understand it and reproduce it easily.

This template will ask for some information you’ve already provided; that’s OK, just fill it out the best you can. 👍 I’ve also included some helpful tips below the template. Feel free to let me know if you have any questions!

Thank you again for your report, we look forward to resolving it!

Template

## 1. Environment

### 1a. Operating system and version

```
paste here
```


### 1b. Caddy version (run `caddy version` or paste commit SHA)

```
paste here
```


### 1c. Go version (if building Caddy from source; run `go version`)

```
paste here
```


## 2. Description

### 2a. What happens (briefly explain what is wrong)




### 2b. Why it's a bug (if it's not obvious)




### 2c. Log output

```
paste terminal output or logs here
```



### 2d. Workaround(s)




### 2e. Relevant links




## 3. Tutorial (minimal steps to reproduce the bug)

Instructions – please heed otherwise we cannot help you (help us help you!)

1. Environment: Please fill out your OS and Caddy versions, even if you don’t think they are relevant. (They are always relevant.) If you built Caddy from source, provide the commit SHA and specify your exact Go version.

2. Description: Describe at a high level what the bug is. What happens? Why is it a bug? Not all bugs are obvious, so convince readers that it’s actually a bug.

   • 2c) Log output: Paste terminal output and/or complete logs in a code block. DO NOT REDACT INFORMATION except for credentials. Please enable debug and access logs.
   • 2d) Workaround: What are you doing to work around the problem in the meantime? This can help others who encounter the same problem, until we implement a fix.
   • 2e) Relevant links: Please link to any related issues, pull requests, docs, and/or discussion. This can add crucial context to your report.

3. Tutorial: What are the minimum required specific steps someone needs to take in order to experience the same bug? Your goal here is to make sure that anyone else can have the same experience with the bug as you do. You are writing a tutorial, so make sure to carry it out yourself before posting it. Please:

   • Start with an empty config. Add only the lines/parameters that are absolutely required to reproduce the bug.
   • Do not run Caddy inside containers.
   • Run Caddy manually in your terminal; do not use systemd or other init systems.
   • If making HTTP requests, avoid web browsers. Use a simpler HTTP client instead, like curl.
   • Do not redact any information from your config (except credentials). Domain names are public knowledge and often necessary for quick resolution of an issue!
   • Note that ignoring this advice may result in delays, or even in your issue being closed. 😞 Only actionable issues are kept open, and if there is not enough information or clarity to reproduce the bug, then the report is not actionable.

Example of a tutorial:

Create a config file:

{ ... }

Open terminal and run Caddy:

$ caddy ...

Make an HTTP request:

$ curl ...

Notice that the result is ___ but it should be ___.