ingress-nginx: Says invalid fcgi value, but it isn't

What happened:

Upgrade to 4.9.0:

 main.go:146] "fcgi annotation error" err="fcgi contains invalid key or value" configmap="php-config-map" namespace="main" key="SCRIPT_FILENAME" value="/app/src/index.php"
E0929 21:35:47.438810       7 annotations.go:189] "ingress contains invalid annotation value" err="annotation fastcgi-params-configmap contains invalid value"
E0929 21:35:47.438839       7 main.go:96] "invalid ingress configuration" err="annotation fastcgi-params-configmap contains invalid value" ingress="main/php"

and now seeing this when trying to update an ingress. Changing any key to a hardcoded value (and not a variable) results in this error. For example, setting DOCUMENT_ROOT to "/app" results in the same error but for DOCUMENT_ROOT.

What you expected to happen:

To be able to hard-code variables sent to PHP.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.9.0
  Build:         4bd3d6b8a00b01b009f225a5593ce502cce5c26b
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version):

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:40:17Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.6+k3s1", GitCommit:"bd04941a294793ec92e8703d5e5da14107902e88", GitTreeState:"clean", BuildDate:"2023-09-20T23:05:58Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: Bare Metal

• OS (e.g. from /etc/os-release): Ubuntu 22.04.3 LTS

• Kernel (e.g. uname -a): Linux cameo 5.15.0-84-generic #93-Ubuntu SMP Tue Sep 5 17:16:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:

  • k3s

• Basic cluster related info:

  • kubectl version
  • kubectl get nodes -o wide

• How was the ingress-nginx-controller installed: helm

How to reproduce this issue:

1. Create a cluster + install nginx ingress however you want (pods aren’t really necessary because you can’t save the ingress)
2. Create a config map

apiVersion: v1
kind: ConfigMap
metadata:
  name: php-config-map
  namespace: default
data:
  CONTENT_LENGTH: $content_length
  CONTENT_TYPE: $content_type
  DOCUMENT_ROOT: $document_root
  DOCUMENT_URI: $document_uri
  HTTPS: $https
  HTTP_X_FORWARDED_FOR: $remote_addr
  HTTP_X_FORWARDED_HOST: $best_http_host
  HTTP_X_FORWARDED_PORT: $pass_port
  HTTP_X_FORWARDED_PROTO: $pass_access_scheme
  HTTP_X_FORWARDED_SCHEME: $pass_access_scheme
  HTTP_X_ORIGINAL_FORWARDED_FOR: $http_x_forwarded_for
  HTTP_X_REAL_IP: $remote_addr
  HTTP_X_REQUEST_ID: $req_id
  HTTP_X_SCHEME: $pass_access_scheme
  QUERY_STRING: $query_string
  REDIRECT_STATUS: '200'
  REMOTE_ADDR: $remote_addr
  REMOTE_PORT: $remote_port
  REQUEST_METHOD: $request_method
  REQUEST_SCHEME: $scheme
  REQUEST_URI: $request_uri
  SCRIPT_NAME: $fastcgi_script_name
  SCRIPT_FILENAME: /app/index.php # <--------------------- this is important
  SERVER_ADDR: $server_addr
  SERVER_NAME: $server_name
  SERVER_PORT: $server_port
  SERVER_PROTOCOL: $server_protocol

Now create an ingress that uses the config map:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: php
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: FCGI
    nginx.ingress.kubernetes.io/fastcgi-index: index.php
    nginx.ingress.kubernetes.io/fastcgi-params-configmap: default/php-config-map
spec:
  ingressClassName: nginx
  rules:
    - host: example.com
      http:
        paths:
          - path: /login/
            pathType: Prefix
            backend:
              service:
                name: php
                port:
                  number: 9000

See the error mentioned at the beginning.