ingress-nginx: Ingress rule with non-wildcard path does not work

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.2.0
  Build:         a2514768cd282c41f39ab06bda17efefc4bd233a
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.10

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-14T08:49:13Z", GoVersion:"go1.17.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"8211ae4d6757c3fedc53cd740d163ef65287276a", GitTreeState:"clean", BuildDate:"2022-03-31T20:28:03Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}

Environment: AKS

• Cloud provider or hardware configuration: AKS

• How was the ingress-nginx-controller installed:

  • If helm was used then please show output of helm ls -A | grep -i ingress

$ helm ls -A | grep -i ingress
ingress-nginx	ingress-nginx	1       	2022-04-27 12:32:39.5649468 -0700 PDT	deployed	ingress-nginx-4.1.0	1.2.0

• If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename>

$ helm -n ingress-nginx get values ingress-nginx
USER-SUPPLIED VALUES:
null

• Current State of the controller:
  • kubectl describe ingressclasses

$ kubectl describe ingressclasses
Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.2.0
              helm.sh/chart=ingress-nginx-4.1.0
Annotations:  meta.helm.sh/release-name: ingress-nginx
              meta.helm.sh/release-namespace: ingress-nginx
Controller:   k8s.io/ingress-nginx
Events:       <none>

• Nginx service:

apiVersion: v1
items:
- apiVersion: v1
  kind: Service
  metadata:
    annotations:
      meta.helm.sh/release-name: ingress-nginx
      meta.helm.sh/release-namespace: ingress-nginx
    creationTimestamp: "2022-04-27T19:32:42Z"
    finalizers:
    - service.kubernetes.io/load-balancer-cleanup
    labels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
      app.kubernetes.io/version: 1.2.0
      helm.sh/chart: ingress-nginx-4.1.0
    name: ingress-nginx-controller
    namespace: ingress-nginx
    resourceVersion: "36534"
    uid: af6861ae-b2ab-4dad-9d2a-491c0be8ea5a
  spec:
    allocateLoadBalancerNodePorts: true
    clusterIP: 10.0.198.145
    clusterIPs:
    - 10.0.198.145
    externalTrafficPolicy: Cluster
    internalTrafficPolicy: Cluster
    ipFamilies:
    - IPv4
    ipFamilyPolicy: SingleStack
    ports:
    - appProtocol: http
      name: http
      nodePort: 31583
      port: 80
      protocol: TCP
      targetPort: http
    - appProtocol: https
      name: https
      nodePort: 30268
      port: 443
      protocol: TCP
      targetPort: https
    selector:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/name: ingress-nginx
    sessionAffinity: None
    type: LoadBalancer

• Additional info

$ kubectl get svc,ing -n httpbin
NAME              TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)     AGE
service/httpbin   ClusterIP   10.0.61.27   <none>        14001/TCP   47h

NAME                                CLASS   HOSTS   ADDRESS         PORTS   AGE
ingress.networking.k8s.io/httpbin   nginx   *       20.85.153.119   80      42h

$ kubectl describe ingress -A
Name:             httpbin
Labels:           <none>
Namespace:        httpbin
Address:          20.85.153.119
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /get   httpbin:14001 (10.244.1.15:14001)
Annotations:  <none>
Events:
  Type    Reason  Age                 From                      Message
  ----    ------  ----                ----                      -------
  Normal  Sync    8m5s (x5 over 42h)  nginx-ingress-controller  Scheduled for sync


# Request from another container that is able to access the backend httpbin service at /get
$ kubectl exec -n curl -ti "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items[0].metadata.name}')" -c curl -- curl -I http://httpbin.httpbin:14001/get
HTTP/1.1 200 OK
Server: gunicorn/19.9.0
Date: Fri, 29 Apr 2022 16:48:15 GMT
Connection: keep-alive
Content-Type: application/json
Content-Length: 208
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

What happened: I am seeing an extremely odd behavior where paths other than the root / don’t work. My sample app httpbin serves requests on multiple paths such as /, /get, /status/200 etc. However, any path other than the root path / don’t seem to work, and the requests simply hang with no logs in the nginx ingress controller pod.

Working: path=/

Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  namespace: httpbin
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: httpbin
            port:
              number: 14001

Request:

$ curl -sI  http://"$nginx_ingress_host":"$nginx_ingress_port"/get
HTTP/1.1 200 OK
Date: Wed, 27 Apr 2022 22:11:39 GMT
Content-Type: application/json
Content-Length: 289
Connection: keep-alive
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

Nginx log for request:

10.224.0.4 - - [27/Apr/2022:22:11:39 +0000] "HEAD /get HTTP/1.1" 200 0 "-" "curl/7.68.0" 81 0.005 [httpbin-httpbin-14001] [] 10.244.2.10:14001 0 0.004 200 5140242512fcba8d377f33d7157f6d09

Nginx conf: nginx-widcard.txt

Failing: path=/get

Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  namespace: httpbin
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /get
        pathType: Prefix
        backend:
          service:
            name: httpbin
            port:
              number: 14001

Request hangs:

$ curl -sI  http://"$nginx_ingress_host":"$nginx_ingress_port"/get
^C

No logs in Nginx for this request.

Logs corresponding to Nginx config update when switching from path / to /get: nginx-path-update.txt

Nginx conf: nginx-specific.txt

What you expected to happen:

Request /get should work with path=/get

How to reproduce it:

1. Deploy httpbin application: https://raw.githubusercontent.com/openservicemesh/osm-docs/release-v1.1/manifests/samples/httpbin/httpbin.yaml
2. Apply Ingress resources shared above and test it