aspnetcore: "Connection: upgrade" causes 400 error that never reaches application code. Triggered by common nginx config.

Describe the bug

If a Kestrel receives a Connection: upgrade header, in a POST request with a nonempty body, then Kestrel will never pass the request to application code. It will give a 400 “Bad Request” response with an empty body.

This gets triggered by a very common HTTPS reverse proxy configuration of nginx (see below).

To Reproduce

I have a Jellyfin 10.4.1 server set up at localhost:8096. Jellyfin uses Kestrel as its web server in a pretty straightforward configuration.

This curl line triggers the bug:

# curl -iv --raw --data foo -H 'Connection: upgrade' http://localhost:8096/notfound
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8096 (#0)
> POST /notfound HTTP/1.1
> Host: localhost:8096
> User-Agent: curl/7.58.0
> Accept: */*
> Connection: upgrade
> Content-Length: 3
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 3 out of 3 bytes
< HTTP/1.1 400 Bad Request
HTTP/1.1 400 Bad Request
< Connection: close
Connection: close
< Date: Thu, 14 Nov 2019 05:25:52 GMT
Date: Thu, 14 Nov 2019 05:25:52 GMT
< Server: Kestrel
Server: Kestrel
< Content-Length: 0
Content-Length: 0

<
* Closing connection 0

Jellyfin logs all errors in its request handler, but no logs are ever output for this case. Application code never sees the request.

If I omit the header, or use a GET request, or an empty POST request with -X POST, then the bug is not triggered. I get the expected 404 error, and Jellyfin prints an error message.

Further technical details

I’m very new to .NET so I’m not sure how to get the version of ASP.NET Core being used here. I am using the official 10.4.1 Ubuntu release of Jellyfin.

I have checked out the Jellyfin source and done enough debugging to verify that the error seems to be in Kestrel, and it doesn’t appear to be anywhere in Jellyfin handler code or in its Kestrel config.

Jellyfin configures its Kestrel server here: https://github.com/jellyfin/jellyfin/blob/v10.4.1/Emby.Server.Implementations/ApplicationHost.cs#L615

I’m using nginx as an HTTPS reverse proxy to Jellyfin. I’m using some standard nginx config for reverse proxying, which looks like this:

    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";

This is common in lots of nginx documentation. Here’s an example.

Connection: upgrade is usually accompanied by Upgrade: websocket or so, but my read of the RFC is that specifying a missing header name shouldn’t be grounds for a 400 response. Connection: upgrade should just mean that the server shouldn’t forward the Upgrade header, if it exists.

In any case, I would expect that if Kestrel will catch malformed requests and not forward them to application code at all, that Kestrel will at least include some message indicating what’s wrong. The empty response had me chasing my tail for days trying to figure out the problem.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 14
  • Comments: 33 (20 by maintainers)

Commits related to this issue

Most upvoted comments

FWIW, this nginx config is working for me as a workaround:

    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $http_connection;

Not sure why this isn’t the default in shipped config and documentation.

These changes have been backported to 5.0, 3.1, and 2.1 for the February patches.

@sbrudenell I was having a similar issue where my original nginx config had proxy_set_header Connection keep-alive; which caused web sockets errors with blazor. Changing it to proxy_set_header Connection "upgrade"; fixed those WS errors but broke the app’s api endpoints which all returned 400 errors. Setting the connection header as specified on the blazor docs fixed the issue for me, now both WS and POST requests work fine: proxy_set_header Connection $http_connection;

https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/blazor/server?view=aspnetcore-3.1#linux-with-nginx

I’ve just hit this too, using AWS’s Elastic Beanstalk with default settings for running in a Docker container. When I try to do a simple POST to a login action all I get is an empty 400 response, never hitting the application code. Checking the debug logs revealed that Connection: Upgrade error in Kestrel. Beanstalk uses nginx as the default proxy and sure enough disabling it “fixes” the problem. It’s a managed platform, presumably using a stock nginx config, though it’s abstracted away from the user and it doesn’t look super straight-forward to be able to edit it.

I used the following two options in Elastic Beanstalk nginx config and it resolved the bad request:

    proxy_set_header    X-Forwarded-Host        $http_host;
    proxy_set_header    Connection              $http_connection;

Configured nginx via: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/platforms-linux-extend.html

Kestrel doesn’t support Connection upgrades to h2c, it only supports ALPN based h2 upgrades. The HTTP/2 connection upgrade mechanic is not widely supported, that’s actually the first client I’ve seen attempt it.

However, we should make kestrel capable of ignoring such an upgrade request. The PUT request should be executed like a normal HTTP/1.1 request.

Just want to add for @delebru post, in my case in order to make $http_connection work, I have to declare it first. Otherwise, nginx will fail:

Error while running nginx -c /etc/nginx/nginx.conf -t.

nginx: [emerg] unknown "connection_upgrade" variable
nginx: configuration file /etc/nginx/nginx.conf test failed

My full nginx configuration is something like:

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

server {
    listen        80;
    server_name   xxx;

    location / {
        proxy_pass         http://localhost:5000
        proxy_http_version 1.1;
        proxy_set_header   Upgrade \$http_upgrade;
        proxy_set_header   Connection \$connection_upgrade;
        proxy_set_header   Host \$host;
        proxy_cache_bypass \$http_upgrade;
        proxy_set_header   X-Forwarded-For \$proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto \$scheme;
        proxy_set_header   X-Real-IP \$remote_addr;
    }
}