nginx-proxy: ERR_HTTP2_SERVER_REFUSED_STREAM with PHP web application on 0.9.0

I’m running Xibo (which is PHP/Apache) behind nginx-proxy with the LetsEncrypt helper.

Normally no issues, but today I did a new install and on about 30% of my page loads (mainly javascript files), Chrome 90 gives an error loading those resources ERR_HTTP2_SERVER_REFUSED_STREAM. Refreshing the page gets the error on a different number of those URLs. If I access the URLs individually they work as expected.

Dropping down to 0.8.0 fixes it, and going back to 0.9.0 causes it again, so I can only presume it’s something changing in nginx?

To replicate, you’d need a docker-compose setup like this:

version: "2.1"

services:
    cms-db:
        image: mysql:5.7
        volumes:
            - "./shared/db:/var/lib/mysql:Z"
        environment:
            - MYSQL_DATABASE=cms
            - MYSQL_USER=cms
            - MYSQL_RANDOM_ROOT_PASSWORD=yes
            - MYSQL_PASSWORD=password123
    cms-web:
        image: xibosignage/xibo-cms:release-2.3.10
        volumes:
            - "./shared/cms/custom:/var/www/cms/custom:Z"
            - "./shared/backup:/var/www/backup:Z"
            - "./shared/cms/web/theme/custom:/var/www/cms/web/theme/custom:Z"
            - "./shared/cms/library:/var/www/cms/library:Z"
            - "./shared/cms/web/userscripts:/var/www/cms/web/userscripts:Z"
            - "./shared/cms/ca-certs:/var/www/cms/ca-certs:Z"
        links:
            - cms-db:mysql
        environment:
            - XMR_HOST=cms-xmr
            - VIRTUAL_HOST=cms.example.org
            - LETSENCRYPT_HOST=cms.example.org
            - LETSENCRYPT_EMAIL=user@example.org
            - HTTPS_METHOD=noredirect
            - SERVER_TOKENS=off
            - MYSQL_PASSWORD=password123
    cms-proxy:
        image: jwilder/nginx-proxy:0.9.0
        ports:
            - 80:80
            - 443:443
        volumes:
            - ./shared/proxy/certs:/etc/nginx/certs:ro
            - /var/run/docker.sock:/tmp/docker.sock:ro
            - /usr/share/nginx/html
        labels:
            - com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy
        environment:
            - "DEFAULT_HOST=cms.example.org"
            - "DHPARAM_GENERATION=false"
    cms-letsencrypt:
        image: jrcs/letsencrypt-nginx-proxy-companion
        volumes_from:
            - cms-proxy
        volumes:
            - ./shared/proxy/certs:/etc/nginx/certs:rw
            - /var/run/docker.sock:/var/run/docker.sock:ro

Once the containers are started, log in at https://cms.example.org - username xibo_admin, password password and check the Chrome network access tab in developer tools.

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 26 (12 by maintainers)

Most upvoted comments

I solved my ERR_HTTP2_SERVER_REFUSED_STREAM by setting a higher value for keep-alive-requests in my ingress-nginx:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-mse-controller
  namespace: mse
data:
  enable-brotli: "true"
  use-http2: "true"
  keep-alive-requests: "9999"

The default value for keep-alive-requests is 1000.

http://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_requests

If you want to specify “unlimited” there is no such value; just set a high value. The higher the value the more memory your server will consume.

More info here: https://serverfault.com/a/425130/241371 and here: https://trac.nginx.org/nginx/ticket/2155

Changes with nginx 1.19.7 16 Feb 2021

*) Change: connections handling in HTTP/2 has been changed to better
   match HTTP/1.x; the "http2_recv_timeout", "http2_idle_timeout", and
   "http2_max_requests" directives have been removed, the
   "keepalive_timeout" and "keepalive_requests" directives should be
   used instead.

*) Change: the "http2_max_field_size" and "http2_max_header_size"
   directives have been removed, the "large_client_header_buffers"
   directive should be used instead.

*) Feature: now, if free worker connections are exhausted, nginx starts
   closing not only keepalive connections, but also connections in
   lingering close.

*) Bugfix: "zero size buf in output" alerts might appear in logs if an
   upstream server returned an incorrect response during unbuffered
   proxying; the bug had appeared in 1.19.1.

*) Bugfix: HEAD requests were handled incorrectly if the "return"
   directive was used with the "image_filter" or "xslt_stylesheet"
   directives.

*) Bugfix: in the "add_trailer" directive.

Changes with nginx 1.19.8 09 Mar 2021

*) Feature: flags in the "proxy_cookie_flags" directive can now contain
   variables.

*) Feature: the "proxy_protocol" parameter of the "listen" directive,
   the "proxy_protocol" and "set_real_ip_from" directives in mail proxy.

*) Bugfix: HTTP/2 connections were immediately closed when using
   "keepalive_timeout 0"; the bug had appeared in 1.19.7.

*) Bugfix: some errors were logged as unknown if nginx was built with
   glibc 2.32.

*) Bugfix: in the eventport method.

Changes with nginx 1.19.9 30 Mar 2021

*) Bugfix: nginx could not be built with the mail proxy module, but
   without the ngx_mail_ssl_module; the bug had appeared in 1.19.8.

*) Bugfix: "upstream sent response body larger than indicated content
   length" errors might occur when working with gRPC backends; the bug
   had appeared in 1.19.1.

*) Bugfix: nginx might not close a connection till keepalive timeout
   expiration if the connection was closed by the client while
   discarding the request body.

*) Bugfix: nginx might not detect that a connection was already closed
   by the client when waiting for auth_delay or limit_req delay, or when
   working with backends.

*) Bugfix: in the eventport method.

Changes with nginx 1.19.10 13 Apr 2021

*) Change: the default value of the "keepalive_requests" directive was
   changed to 1000.

*) Feature: the "keepalive_time" directive.

*) Feature: the $connection_time variable.

*) Workaround: "gzip filter failed to use preallocated memory" alerts
   appeared in logs when using zlib-ng.

Relevant bits:

In 1.19.8

Bugfix: HTTP/2 connections were immediately closed when using “keepalive_timeout 0”; the bug had appeared in 1.19.7.

In 1.19.9

Bugfix: nginx might not close a connection till keepalive timeout expiration if the connection was closed by the client while discarding the request body.

Bugfix: nginx might not detect that a connection was already closed by the client when waiting for auth_delay or limit_req delay, or when working with backends.

And maybe this in 1.19.10

Change: the default value of the “keepalive_requests” directive was changed to 1000.

Feature: the “keepalive_time” directive.

Feature: the $connection_time variable.