ingress-nginx: rewrite-target: /$2 causes incorrect Location response header for redirects

If I’m not mistaken, the issue here is that the service behind the nginx proxy is returning a relative redirect in the Location header, which the nginx proxy is ignoring. So a simple GET request would…

# Request from open internet to the proxy
GET https://foo.example.com/some-service/foo

# Proxy rewrites it to the internal pod
GET http://pod.local/foo

# Pod returns a 30x redirect with a relative Path
302 Found
Location: /bar

# Nginx passes it straight out
302 Found
Location: /bar

However, what we need is for nginx to reverse the rewrite in any Location headers

302 Found
Location: /some-service/bar

Is there some other way of accomplishing this?

Okay, I have created a minimal example and think I am halfway to solving it.

Create Dockerfile for Flask app:

FROM python:3.8-slim

RUN set -eux \
    && python3 -m pip install --no-cache-dir --no-color --no-input flask \
    && echo 'from flask import Flask, jsonify, redirect, url_for'    >> app.py \
    && echo 'app = Flask(__name__)'                                  >> app.py \
    && echo '@app.route("/")'                                        >> app.py \
    && echo 'def hello_world(): return "<p>Hello, World!</p>"'       >> app.py \
    && echo '@app.route("/api/v1")'                                  >> app.py \
    && echo 'def hello_api(): return jsonify({"hello": "world"})'    >> app.py \
    && echo '@app.route("/foo")'                                     >> app.py \
    && echo 'def redirect_to_bar(): return redirect(url_for("bar"))' >> app.py \
    && echo '@app.route("/foo2")'                                    >> app.py \
    && echo 'def redirect_to_bar2(): return redirect("/bar")'        >> app.py \
    && echo '@app.route("/bar")'                                     >> app.py \
    && echo 'def bar(): return "<p>Hello, Bar!</p>"'                 >> app.py \
    && echo '@app.route("/external")'                                >> app.py \
    && echo 'def exredir(): return redirect("https://google.com")'   >> app.py

ENV FLASK_APP=app
ENV FLASK_SKIP_DOTENV=1
CMD ["python3", "-m", "flask", "run", "--host", "0.0.0.0", "--port", "5000", "--no-reload", "--no-debugger", "--without-threads"]

Create Kubernetes manifests:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flask-deployment
  labels:
    app: flask
spec:
  replicas: 2
  selector:
    matchLabels:
      app: flask
  template:
    metadata:
      labels:
        app: flask
    spec:
      containers:
      - name: flask
        image: <registry>/hello-flask:latest
        ports:
        - containerPort: 5000
          name: http
      imagePullSecrets:
        - name: <secretname>

---

kind: Pod
apiVersion: v1
metadata:
  name: curler
spec:
  containers:
  - name: curler
    image: curlimages/curl
    command:
      - sleep
      - "100d"

---

apiVersion: v1
kind: Service
metadata:
  name: flask-service
spec:
  selector:
    app: flask
  ports:
    - protocol: TCP
      port: 80
      targetPort: http

Test requesting the service name from another pod (service is ClusterIP):

$ k exec -n my-namespace curler -- curl -i -fsSL http://flask-service/foo
HTTP/1.0 302 FOUND
Content-Type: text/html; charset=utf-8
Content-Length: 214
Location: http://flask-service/bar
Server: Werkzeug/2.0.2 Python/3.8.12
Date: Thu, 02 Dec 2021 16:12:17 GMT

HTTP/1.0 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 18
Server: Werkzeug/2.0.2 Python/3.8.12
Date: Thu, 02 Dec 2021 16:12:17 GMT

<p>Hello, Bar!</p>

$ k exec -n my-namespace curler -- curl -I -fsS http://flask-service/external
HTTP/1.0 302 FOUND
Content-Type: text/html; charset=utf-8
Content-Length: 242
Location: https://google.com
Server: Werkzeug/2.0.2 Python/3.8.12
Date: Thu, 02 Dec 2021 19:17:08 GMT

Okay - so at this point app “works” and service talks to it properly. Service is default ClusterIP and no Ingress has been introduced yet.

Next deploy ingress-nginx with ArgoCD/Helm:

argocd login --insecure --username admin \
  xxx-xx.zzzz.elb.amazonaws.com

argocd repo add --type helm --name ingress-nginx \
  https://kubernetes.github.io/ingress-nginx

argocd app create ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --dest-namespace ingress-nginx \
  --dest-server https://kubernetes.default.svc \
  --revision 4.0.12 \
  --helm-chart ingress-nginx \
  --helm-set 'controller.service.externalTrafficPolicy=Local' \
  --helm-set-string 'controller.service.loadBalancerSourceRanges[0]=<your-ip/32>'

argocd app sync ingress-nginx

Now add Ingress resource for our Flask app:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: simple
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - backend:
          service:
            name: flask-service
            port:
              number: 80
        path: /app1(/|$)(.*)
        pathType: Prefix

Deploy this manifest as part of the Flask app deployment.

First two routes work:

• Request http://xx-xx.zzzz.elb.amazonaws.com/app1/
• Response log zzz.zzz.14.153 - - [02/Dec/2021 17:33:11] “GET / HTTP/1.1” 200 -
• Request http://xx-xx.zzzz.elb.amazonaws.com/app1/api/v1
• Response logzzz.zzz.14.153 - - [02/Dec/2021 17:33:40] “GET /api/v1 HTTP/1.1” 200 -

But how about /app1/foo?

$ curl -I http://xx-xx.zzzz.elb.amazonaws.com/app1/foo
HTTP/1.1 302 FOUND
Date: Thu, 02 Dec 2021 17:35:35 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 214
Connection: keep-alive
Location: http://xx-xx.zzzz.elb.amazonaws.com/bar

This illustrates the central problem - http://xx-xx.zzzz.elb.amazonaws.com/bar will 404.

• Actual: http://xx-xx.zzzz.elb.amazonaws.com/bar
• Should be: http://xx-xx.zzzz.elb.amazonaws.com/app1/bar

Now for the half-solution. The course of action seems to be to use nginx.ingress.kubernetes.io/proxy-redirect-to and nginx.ingress.kubernetes.io/proxy-redirect-from. Unfortunately there appears to be a limitation in Nginx config (https://trac.nginx.org/nginx/ticket/2291, https://stackoverflow.com/q/70205048/7954504) that does not allow the precise notation required:

• must redirect only for $proxy_host and ignore other hosts
• must use regex for the URL path

does this issue break the /healthz check in nginx? mine seems too, to note this happens when i don’t specify a host

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: webserver
  namespace: namespace
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_input_headers "x-request-start: t=$msec";
spec:
  rules:
    - http:
        paths:
          - path: /webserver(/|$)(.*)
            pathType: Prefix
            backend:
              service:
                name: webserver
                port:
                  number: 8000

 curl -Is https://cluster-dns.name/healthz
HTTP/1.1 404 Not Found

when i remove this rule i get a 200

You need to configure defaultBackend: true on your nginx controller to resolve that issue: https://kubernetes.github.io/ingress-nginx/user-guide/default-backend/

@brsolomon-deloitte , based on your last comment, is this still a open issue

Yes.