caddy: v2&v1: HTTP/3 breaks support of getting remote IP and port

The version of caddy is 1.0.4 and 2.0beta14.

Config of caddy 2:

{
    "admin": {
        "disabled": true,
        "listen": "",
        "enforce_origin": false,
        "origins": [""],
        "config": {
            "persist": false
        }
    },
    "apps": {
        "http": {
            "servers": {
                "h3-proxy": {
                    "listen": [":2"],
                    "routes": [{
                        "match": [{
                            "host": ["qwq.ren", "*.qwq.ren", "imvictor.tech", "*.imvictor.tech"]
                        }],
                        "handle": [{
                            "handler": "subroute",
                            "routes": [{
                                "handle": [{
                                    "encodings": {
                                        "brotli": {},
                                        "gzip": {}
                                    },
                                    "handler": "encode"
                                }, {
                                    "handler": "reverse_proxy",
                                    "headers": {
                                        "request": {
                                            "set": {
                                                "Host": ["{http.request.host}"],
                                                "X-Forwarded-For": ["{http.request.remote.host}"],
                                                "X-Forwarded-Proto": ["{http.request.scheme}"],
                                                "X-Real-IP": ["{http.request.remote.host}"],
                                                "X-Victors-Test1": ["passed"]
                                            }
                                        }
                                    },
                                    "upstreams": [{
                                        "dial": "127.0.0.1:80"
                                    }]
                                }]
                            }]
                        }],
                        "terminal": true
                    }],
                    "tls_connection_policies": [{
                        "match": {
                            "sni": ["*.imvictor.tech", "qwq.ren", "*.qwq.ren", "imvictor.tech"]
                        },
                        "certificate_selection": {
                            "policy": "custom",
                            "tag": "main-cert"
                        }
                    }, {}],
                    "experimental_http3": true,
                    "automatic_https": {
                        "disable": true
                    }
                }
            }
        },
        "tls": {
            "certificates": {
                "load_files": [{
                    "certificate": "[HIDDEN].cer",
                    "key": "[HIDDEN].key",
                    "tags": ["main-cert"]
                }]
            }
        }
    }
}

Config of caddy 1.0.4:

https://qwq.ren:2 https://*.qwq.ren:2 https://imvictor.tech:2 https://*.imvictor.tech:2 {
    gzip
    proxy / http://127.0.0.1:80 {
        header_upstream Host {host}
        header_upstream X-Forwarded-Proto {scheme}
        header_upstream X-Forwarded-Port {port}
        header_upstream X-Forwarded-For {remote}
        header_upstream X-Real-IP {remote}
        header_upstream X-Victors-Test1 "passed"
        insecure_skip_verify
    }
    tls [HIDDEN].cer [HIDDEN].key
}

Test HTTP/3 with curl 7.69.0-dev, it shows that some headers like X-Forwarded-For or X-Real-IP cannot be passed to the backend,

Victor@Victor-Mac:/tmp » curl-h3 -6 --http3 https://dns-main.imvictor.tech:2/__test/dump -H 'X-Victors-Test0: test'
Port: 80
Remote Addr: 127.0.0.1
Proxy Protocol Addr: 
X-Forwarded-For Header: 
X-Real-IP Header: 
X-Forwarded-Proto Header: https
X-Forwarded-Port Header: 
X-Victors-Test0 Header: test
X-Victors-Test1 Header: passed

while HTTP/2 and HTTP/1.1 both work:

Victor@Victor-Mac:/tmp » curl-h3 -6 --http2 https://dns-main.imvictor.tech:2/__test/dump -H 'X-Victors-Test0: test'
Port: 80
Remote Addr: 2409:8a3c:5b7c:2300:[HIDDEN]
Proxy Protocol Addr: 
X-Forwarded-For Header: 2409:8a3c:5b7c:2300:[HIDDEN]
X-Real-IP Header: 2409:8a3c:5b7c:2300:[HIDDEN]
X-Forwarded-Proto Header: https
X-Forwarded-Port Header: 54881
X-Victors-Test0 Header: test
X-Victors-Test1 Header: passed
Victor@Victor-Mac:/tmp » curl-h3 -6 --http1.1 https://dns-main.imvictor.tech:2/__test/dump -H 'X-Victors-Test0: test'
Port: 80
Remote Addr: 2409:8a3c:5b7c:2300:[HIDDEN]
Proxy Protocol Addr: 
X-Forwarded-For Header: 2409:8a3c:5b7c:2300:[HIDDEN]
X-Real-IP Header: 2409:8a3c:5b7c:2300:[HIDDEN]
X-Forwarded-Proto Header: https
X-Forwarded-Port Header: 55544
X-Victors-Test0 Header: test
X-Victors-Test1 Header: passed

Now I have no idea to make a workaround.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 24 (16 by maintainers)

Most upvoted comments

I’m very glad to see the problem fixed. Thank you all for your hard work.

+4
qwqVictor on Aug 24, 2020

@mholt Sure, let me see what’s going on.

+3
marten-seemann on Feb 18, 2020

@mholt It doesn’t. The request gets passed to the handler just a few lines after the remote address is set: https://github.com/lucas-clemente/quic-go/blob/9899be3a06a168f36768cc4a89cc9df6077b3ac7/http3/server.go#L270

+2
marten-seemann on Feb 18, 2020

I’m going to confirm this. But as version goes, the structure of config seems to have changed. I have to find out the right way to configure it before I can confirm it.

+1
qwqVictor on Aug 24, 2020

FWIW, I’m currently convinced that the bug is upstream. Can’t prove it either way yet, so I might be wrong, but from what I know of the Caddy code base, that’s my current suspicion. I don’t have the time or funding to spend more time on this right now (especially since the HTTP/3 integration is still experimental), but feel free to drill down more if you want a quicker fix! It will help a lot.

+1
mholt on Feb 18, 2020

I think bug and v2 labels can be added to this issue. Looks like that it’s a strange bug, which needs more attention.

+1
qwqVictor on Feb 18, 2020

@marten-seemann That’s what I figured too. Just spitballing here.

@qwqVictor Thanks for checking!

Hmm. This is odd…

+1
mholt on Feb 18, 2020

@qwqVictor Does this happen with an IPv4 address as well?

@marten-seemann

The http.Request.RemoteAddr is properly set.

Seems to be the case… there’s gotta be some other factor we aren’t considering yet then.

We do call SetQuicHeaders(), but that couldn’t possibly affect the request, right?

https://github.com/caddyserver/caddy/blob/57c6f22684e74191814a30f9de83a05c11ac4b78/modules/caddyhttp/server.go#L120-L125

I’m out of ideas for right now…

As a matter of fact, it’s also broken with IPv4.

+1
qwqVictor on Feb 18, 2020

SetQuicHeaders() just takes the http.Header, and adds some values to it. It doesn’t even have access to the http.Request struct.

+1
marten-seemann on Feb 18, 2020

@qwqVictor Does this happen with an IPv4 address as well?

@marten-seemann

The http.Request.RemoteAddr is properly set.

Seems to be the case… there’s gotta be some other factor we aren’t considering yet then.

We do call SetQuicHeaders(), but that couldn’t possibly affect the request, right? https://github.com/caddyserver/caddy/blob/57c6f22684e74191814a30f9de83a05c11ac4b78/modules/caddyhttp/server.go#L120-L125

I’m out of ideas for right now…

+1
mholt on Feb 18, 2020

@qwqVictor Awesome, thanks!

@marten-seemann For reasons I can’t explain, the req.RemoteAddr field is empty on HTTP/3 requests. According to the logs, it is empty even at the very entry point of Caddy’s ServeHTTP. Do you think you could help us investigate when you have a chance?

Edit: I just noticed that the tls information is all empty, too. (Not sure if that’s related.)

To make sure if it’s related to HTTP/3, there is a piece of log dumped from a HTTP/1.1 request. All parameters except HTTP version are the same.

{
	"request": {
		"method": "GET",
		"uri": "/__test/dump",
		"proto": "HTTP/1.1",
		"remote_addr": "[2409:8a3c:5b7a:bb50:HIDDEN_FOR_SEC_REASON]:56631",
		"host": "qwq.ren:2",
		"headers": {
			"User-Agent": ["curl/7.69.0-DEV"],
			"Accept": ["*/*"]
		},
		"tls": {
			"resumed": false,
			"version": 772,
			"ciphersuite": 4867,
			"proto": "http/1.1",
			"proto_mutual": true,
			"server_name": "qwq.ren"
		}
	},
	"common_log": "2409:8a3c:5b7a:bb50:HIDDEN_FOR_SEC_REASON - -- [18/Feb/2020:10:43:51 +0800] \"GET /__test/dump HTTP/1.1\" 200 402",
	"latency": 0.00044381,
	"size": 402,
	"status": 200,
	"resp_headers": {
		"Server": ["Caddy", "nginx/1.17.8"],
		"Alt-Svc": ["h3-24=\":2\"; ma=2592000"],
		"Date": ["Tue, 18 Feb 2020 02:43:51 GMT"],
		"Content-Type": ["application/octet-stream", "text/plain"],
		"Content-Length": ["402"]
	}
}

Hope it could help.

+1
qwqVictor on Feb 18, 2020

@qwqVictor Awesome, thanks!

@marten-seemann For reasons I can’t explain, the req.RemoteAddr field is empty on HTTP/3 requests. According to the logs, it is empty even at the very entry point of Caddy’s ServeHTTP. Do you think you could help us investigate when you have a chance?

Edit: I just noticed that the tls information is all empty, too. (Not sure if that’s related.)

+1
mholt on Feb 18, 2020

I have got the log and formatted it. It suggests that Caddy is unable to get a remote_addr value. @mholt

{
	"request": {
		"method": "GET",
		"uri": "/__test/dump",
		"proto": "HTTP/3",
		"remote_addr": "",
		"host": "qwq.ren:2",
		"headers": {
			"User-Agent": ["curl/7.69.0-DEV"],
			"Accept": ["*/*"]
		},
		"tls": {
			"resumed": false,
			"version": 0,
			"ciphersuite": 0,
			"proto": "",
			"proto_mutual": false,
			"server_name": ""
		}
	},
	"common_log": "- - -- [18/Feb/2020:10:36:18 +0800] \"GET /__test/dump HTTP/3\" 200 294",
	"latency": 0.001040343,
	"size": 294,
	"status": 200,
	"resp_headers": {
		"Server": ["Caddy", "nginx/1.17.8"],
		"Alt-Svc": ["h3-24=\":2\"; ma=2592000"],
		"Date": ["Tue, 18 Feb 2020 02:36:18 GMT"],
		"Content-Type": ["application/octet-stream", "text/plain"],
		"Content-Length": ["294"]
	}
}
+1
qwqVictor on Feb 18, 2020

Cool, thanks for the link.

Is there anything else that needs to be set for this to work?

Nope, that should be all: https://github.com/caddyserver/caddy/blob/57c6f22684e74191814a30f9de83a05c11ac4b78/modules/caddyhttp/replacer.go#L84-L89

@qwqVictor Let’s try debugging this with Caddy 2. Would you mind enabling access logging in your JSON config? All you have to do is set "logs": {} in your server struct: https://caddyserver.com/docs/json/apps/http/servers/logs/

That should output details of HTTP requests to the console when you make requests, including the request header values.

You can also try replacing your reverse_proxy handler with a static_response, like:

{
    "handler": "static_response",
    "body": "Client remote: {http.request.remote} {http.request.remote.host}"
}

Then can you report back with a log and HTTP response?

+1
mholt on Feb 18, 2020

@mholt Yes, quic-go does that: https://github.com/lucas-clemente/quic-go/blob/9899be3a06a168f36768cc4a89cc9df6077b3ac7/http3/server.go#L242

Is there anything else that needs to be set for this to work?

+1
marten-seemann on Feb 18, 2020
Read more comments on GitHub
← caddy: v2.7.x: panic when TLS-SNI is received for an unknown domain
caddy: Websocket fails with rancher →