aspnetcore: "Connection: upgrade" causes 400 error that never reaches application code. Triggered by common nginx config.
Describe the bug
If a Kestrel receives a Connection: upgrade
header, in a POST request with a nonempty body, then Kestrel will never pass the request to application code. It will give a 400 “Bad Request” response with an empty body.
This gets triggered by a very common HTTPS reverse proxy configuration of nginx (see below).
To Reproduce
I have a Jellyfin 10.4.1 server set up at localhost:8096
. Jellyfin uses Kestrel as its web server in a pretty straightforward configuration.
This curl line triggers the bug:
# curl -iv --raw --data foo -H 'Connection: upgrade' http://localhost:8096/notfound
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8096 (#0)
> POST /notfound HTTP/1.1
> Host: localhost:8096
> User-Agent: curl/7.58.0
> Accept: */*
> Connection: upgrade
> Content-Length: 3
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 3 out of 3 bytes
< HTTP/1.1 400 Bad Request
HTTP/1.1 400 Bad Request
< Connection: close
Connection: close
< Date: Thu, 14 Nov 2019 05:25:52 GMT
Date: Thu, 14 Nov 2019 05:25:52 GMT
< Server: Kestrel
Server: Kestrel
< Content-Length: 0
Content-Length: 0
<
* Closing connection 0
Jellyfin logs all errors in its request handler, but no logs are ever output for this case. Application code never sees the request.
If I omit the header, or use a GET request, or an empty POST request with -X POST
, then the bug is not triggered. I get the expected 404 error, and Jellyfin prints an error message.
Further technical details
I’m very new to .NET so I’m not sure how to get the version of ASP.NET Core being used here. I am using the official 10.4.1 Ubuntu release of Jellyfin.
I have checked out the Jellyfin source and done enough debugging to verify that the error seems to be in Kestrel, and it doesn’t appear to be anywhere in Jellyfin handler code or in its Kestrel config.
Jellyfin configures its Kestrel server here: https://github.com/jellyfin/jellyfin/blob/v10.4.1/Emby.Server.Implementations/ApplicationHost.cs#L615
I’m using nginx as an HTTPS reverse proxy to Jellyfin. I’m using some standard nginx config for reverse proxying, which looks like this:
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
This is common in lots of nginx documentation. Here’s an example.
Connection: upgrade
is usually accompanied by Upgrade: websocket
or so, but my read of the RFC is that specifying a missing header name shouldn’t be grounds for a 400 response. Connection: upgrade
should just mean that the server shouldn’t forward the Upgrade
header, if it exists.
In any case, I would expect that if Kestrel will catch malformed requests and not forward them to application code at all, that Kestrel will at least include some message indicating what’s wrong. The empty response had me chasing my tail for days trying to figure out the problem.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 14
- Comments: 33 (20 by maintainers)
Commits related to this issue
- Fix 400 error See https://github.com/dotnet/aspnetcore/issues/17081 — committed to d3473r/nginx by d3473r 4 years ago
- Allow/Ignore upgrades with bodies #17081 — committed to Tratcher/aspnetcore by Tratcher 4 years ago
- Allow/ignore upgrades with bodies (#27921) * Allow/Ignore upgrades with bodies #17081 * Rename, test cleanup — committed to dotnet/aspnetcore by Tratcher 4 years ago
- Allow/ignore upgrades with bodies #17081 — committed to Tratcher/aspnetcore by Tratcher 4 years ago
- Allow/ignore upgrades with bodies #17081 — committed to Tratcher/aspnetcore by Tratcher 4 years ago
- Allow/ignore upgrades with bodies #17081 — committed to Tratcher/aspnetcore by Tratcher 4 years ago
- [2.1] Allow/ignore upgrades with bodies (#28908) * Allow/ignore upgrades with bodies #17081 * Update patchconfig.props — committed to dotnet/aspnetcore by Tratcher 3 years ago
- Allow/ignore upgrades with bodies #17081 (#28907) — committed to dotnet/aspnetcore by Tratcher 3 years ago
- Allow/ignore upgrades with bodies #17081 (#28896) — committed to dotnet/aspnetcore by Tratcher 3 years ago
FWIW, this nginx config is working for me as a workaround:
Not sure why this isn’t the default in shipped config and documentation.
These changes have been backported to 5.0, 3.1, and 2.1 for the February patches.
@sbrudenell I was having a similar issue where my original nginx config had
proxy_set_header Connection keep-alive;
which caused web sockets errors with blazor. Changing it toproxy_set_header Connection "upgrade";
fixed those WS errors but broke the app’s api endpoints which all returned 400 errors. Setting the connection header as specified on the blazor docs fixed the issue for me, now both WS and POST requests work fine:proxy_set_header Connection $http_connection;
https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/blazor/server?view=aspnetcore-3.1#linux-with-nginx
I used the following two options in Elastic Beanstalk nginx config and it resolved the bad request:
Configured nginx via: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/platforms-linux-extend.html
Kestrel doesn’t support Connection upgrades to h2c, it only supports ALPN based h2 upgrades. The HTTP/2 connection upgrade mechanic is not widely supported, that’s actually the first client I’ve seen attempt it.
However, we should make kestrel capable of ignoring such an upgrade request. The PUT request should be executed like a normal HTTP/1.1 request.
Just want to add for @delebru post, in my case in order to make
$http_connection
work, I have to declare it first. Otherwise, nginx will fail:My full nginx configuration is something like: