aspnetcore: Kestrel hangs

Describe the bug

The kestrel server stops responding to requests a few seconds after it starts and being publicly available through nginx.

To Reproduce

I’m not sure how to create a minimalistic repro as this only happens to me in production environment where there are some users, But basically:

1. Run aspnetcore kestrel server on a linux machine inside docker
2. The container is working fine and I can see that on the machine that it is used, it is working as expected
3. I load up an nginx service to open the website publicly and the server stop handling requests

Exceptions (if any)

No exceptions as far as I can see in the logs.

Further technical details

Host (useful for support):
  Version: 5.0.9
  Commit:  208e377a53

.NET SDKs installed:
  No SDKs were found.

.NET runtimes installed:
  Microsoft.AspNetCore.App 5.0.9 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 5.0.9 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET runtimes or SDKs:
  https://aka.ms/dotnet-download

More info

The following is the dockerfile I use: https://github.com/IsraelHikingMap/Site/blob/9ec891d567ad219d15f957020bbabf3b1db47695/Dockerfile The startup is fairly standard: https://github.com/IsraelHikingMap/Site/blob/9ec891d567ad219d15f957020bbabf3b1db47695/IsraelHiking.Web/Startup.cs

I tired yesterday for an hour to make the container stay responsive to outside request but couldn’t make it work properly and couldn’t update the site’s version. The docker-compose that we use for our website is the following:

version: '3.4'
services:

  website:
    image: israelhikingmap/website:9.9.160
    env_file: .env
    environment:
      ASPNETCORE_ENVIRONMENT: Production
    volumes:
      - ./appsettings.json:/israelhiking/appsettings.json
      - some more folders
    ports:
      - 8080:80
    logging:
      options:
        max-size: "1g"
        max-file: "3"
    restart: "unless-stopped"

  nginx:
    image: "nginxinc/nginx-unprivileged"
    networks:
      - maptiles_tiles_conn
      - default
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./nginx:/etc/nginx/conf.d
      - ${HOME}/${CERT_DIR}:/etc/nginx/ssl
      - some more folders
    logging:
      options:
        max-size: "256m"
        max-file: "3"
    restart: "unless-stopped"

 more-irrelevant-services

When doing the following the site hangs:

1. change version to 9.9.166 (which is a version with minimal code changes)
2. save changes
3. docker-compose pull israelhikingmap/website
4. docker-compose stop nginx
5. docker-compose up -d website
6. When using the website internally on the machine that runs the docker it works fine and reports healthy to the health api
7. docker-compose up nginx
8. Wait a few seconds and the health api stops responding…

If there are logs I can add or something I can do on the container itself please let me know as this problem is critical now as I can’t update the version on my site. We’ve seen some outages lately and we’ve tried all kinds of fixes and changes to the site but now I just can’t upgrade to the next version which is problematic. If you have instructions on what needs to be done on the container itself from command line it would be great as the container only has the runtime version of .net core and not the SDK with all the tools etc.

It feels somewhat similar to the following issue which was closed a long time ago: https://github.com/aspnet/KestrelHttpServer/issues/1267 @halter73 I’ve seen that you were active back then, if you have some insights it would be great.

I’d be happy to provide any information that will help solving this issue. Also worth mentioning that we’ve used kestrel as front facing without nginx and it had some hangs, but I think it’s worse now with nginx as front facing. having said that we needed a proxy support for a tile server we have and aspnet proxy does not work well with http/2 which we wanted. More information on that can be found here: https://github.com/twitchax/AspNetCore.Proxy/issues/78 I’m probably missing a lot of details that came to cause the current situation, but I’ll be happy to share any information that might be helpful…