kubernetes-ingress-controller: http2 requests not supported yet and checking config status failed: %!w(*kong.APIError=&{500 An unexpected error occurred})

It looks like the effect is that this new feature is not operational:

The controller can now detect whether a Kong container has crashed and needs a configuration push.

https://github.com/Kong/kubernetes-ingress-controller/commit/39c385e9b31e3871c2e38b2ea8e12ab328e341ad#diff-d8590e8687e7807da0cf97910b88e0ddbfc670866e0973a15c56597b99df9244R59

workaround

These helm chart values seem to restore the broken feature and remove all the error logs:

# Temporary workaround: disable HTTP2 on admin endpoint
# https://github.com/Kong/kubernetes-ingress-controller/issues/2435
admin:
  tls:
    parameters: []

(It seems like KIC could be hitting /status on the status port instead of the admin port, as the status port already has http2 disabled)

I have set ADMIN_LISTEN environment variable to “127.0.0.1:8444 http2 ssl” and it was causing this error.

As a workaround, I had to set ADMIN_LISTEN to “127.0.0.1:8444 ssl” to get rid of these errors.

Blocked until https://github.com/Kong/kong/pull/8690 releases (2.8.1)?

The workaround (disabling HTTP/2 support on the admin API) is fine. It doesn’t meaningfully change the operation of the controller, which makes a separate request for each API call anyway rather than using HTTP/2 pipelining. It may technically degrade Manager performance, but I can’t think of pages that are making more than 2-3 admin API requests at once, so I doubt the impact would be noticeable.

https://github.com/Kong/kong/pull/8690 is the PR that’s planned to fix this/will allow you to turn HTTP/2 back on.

The status endpoint (as of 2.8) provides both generic NGINX status information (connection count and such) and whether or not configuration is present. We only care about the latter, we just can’t fetch it separately, and weren’t using the endpoint before.

It seems that https://github.com/Kong/kong/pull/8690 didn’t land in 2.8.1: https://github.com/Kong/kong/commit/c5fd7233ab3e0d910fef31bc3eecd8bdc25fe220.

In that case this will be available in 3.0 (it’s already included in 3.0 alpha.1 build: https://github.com/Kong/kong/releases/tag/3.0.0-alpha.1)

I’m also seeing this on a straightforward installation of the 2.8.0 helm chart.

I found a proxy PR to support http2 on the status endpoint:

• https://github.com/Kong/kong/pull/8528

This is an exact output from proxy:

2022/04/25 08:58:07 [error] 1110#0: *1656702 [lua] api_helpers.lua:511: handle_error(): /usr/local/share/lua/5.1/lapis/application.lua:424: /usr/local/share/lua/5.1/kong/api/routes/health.lua:45: http2 requests not supported yet
stack traceback:
        [C]: in function 'capture'
        /usr/local/share/lua/5.1/kong/api/routes/health.lua:45: in function 'fn'
        /usr/local/share/lua/5.1/kong/api/api_helpers.lua:285: in function 'fn'
        /usr/local/share/lua/5.1/kong/api/api_helpers.lua:285: in function </usr/local/share/lua/5.1/kong/api/api_helpers.lua:268>

stack traceback:
        [C]: in function 'error'
        /usr/local/share/lua/5.1/lapis/application.lua:424: in function 'handler'
        /usr/local/share/lua/5.1/lapis/application.lua:146: in function 'resolve'
        /usr/local/share/lua/5.1/lapis/application.lua:183: in function </usr/local/share/lua/5.1/lapis/application.lua:181>
        [C]: in function 'xpcall'
        /usr/local/share/lua/5.1/lapis/application.lua:189: in function 'dispatch'
        /usr/local/share/lua/5.1/lapis/nginx.lua:231: in function 'serve'
        /usr/local/share/lua/5.1/kong/init.lua:1423: in function 'admin_content'
        content_by_lua(nginx-kong.conf:360):2: in main chunk, client: 127.0.0.1, server: kong_admin, request: "GET /status HTTP/2.0", host: "localhost:8444"
127.0.0.1 - - [25/Apr/2022:08:58:07 +0000] "GET /status HTTP/2.0" 500 42 "-" "Go-http-client/2.0"

Kong overall is operating fine, but the log volume is annoying and it’s not clear if this issue has any further impact.