ingress-nginx: URLs with special characters don't work in the presence of URL rewriting

NGINX Ingress controller version: 0.26.1

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.10", GitCommit:"7a578febe155a7366767abce40d8a16795a96371", GitTreeState:"clean", BuildDate:"2019-05-01T04:14:38Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.10", GitCommit:"7a578febe155a7366767abce40d8a16795a96371", GitTreeState:"clean", BuildDate:"2019-05-01T04:05:01Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): Debian 10 (quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1)
• Kernel (e.g. uname -a): Linux redacted 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64 GNU/Linux (via kops)
• Install tools: kops 1.11.1
• Others:

What happened:

If urls are being rewritten for the given ingress, requests to urls with some special characters, such as | (%7c), get decoded before being passed upstream, producing an invalid path.

What you expected to happen:

I expected to be able to have any (correctly encoded) characters in urls.

How to reproduce it:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /site/$2
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: example-service
          servicePort: 8080
        path: /(/*|$)(.*)

Accessing https://example.com/whatever|whatever produces a proxied request to the decoded url /site/whatever|whatever, which is invalid and ends in a 400 response.

It appears there are only two ways out of it now (and possibly, as a result of this bug report, a third one in the future):

1. Create a custom nginx config.
2. Do the rewriting in lua.
3. Modify ingress-nginx to allow overriding the proxy_pass argument.

I haven’t tried the first solution, but I don’t want to do that anyway - maintaining the custom config is not something I want to do.

I will be trying the second option now (any hints would be appreciated!), hopefully this will work, but it would still be better to do it statically/declaratively in the generated config, than running custom lua code per request. If this does turn out to be a fix, I would suggest including it in official documentation.

The third option would be an annotation like

some.annotation.for.proxy_pass: http://upstream_balancer/site$request_uri

or

some.annotation.for.proxy_pass.path: /site$request_uri

That should achieve a rewrite without a rewrite and produce valid urls for the upstream service.

Anything else we need to know:

I tried dropping the regex altogether (matching on path /) and rewriting to /site$request_uri?, but then I end up with double encoding.

That’s fixable for the query part of the url with something like:

nginx.ingress.kubernetes.io/server-snippet: |
    rewrite ^ /site$request_uri;
    rewrite ^([^?]*)(\?.*) $1 break;

but that still leaves the path part double-encoded, so %7c becomes %257c, which is not the url being requested.

This is unexpected and quite difficult to sort out. Ideally, ingress-nginx would just handle it automagically. If that’s not an option, being able to override the proxy_pass argument with an annotation would probably be sufficient to make this work.

Relevant resources:

• https://trac.nginx.org/nginx/ticket/727
• https://stackoverflow.com/questions/28684300/

/kind bug