ingress-nginx: Cannot get multi-path working

I have tried every configuration in the universe.

“work”: Can access all URLs without an error in multiple browsers and with curl.

“does not work”: Sends back 404 or 502 errors. If you refresh on some URLs with some configs, it can toggle between 3 different error results, two are different 404s.

I have two backend services. They work consistently with port-forward, LoadBalance in the service, and basic ingress without paths.

I get the root to map, but not any service below that in the path.

What I want is “/” to route to a node app in nginx and “/service2” and subsequent services to route to various services used by that app. I couldn’t even get “/service1” and “/service2” working without the root path, which matched the examples. I tried following every example, then experimented into the unknown. No matter what I did, no path other than the root one would map successfully to a service.

I am running K8S 1.6.7 on GKE. Having ingress.kubernetes.io/rewrite-target: / doesn’t help. When these errors occur, it is simply not getting to the service.

When I am inside a pod, I can curl port 80 of a service no problem. curl -v http://myservice returns the expected 200. When I get a 200, on the node app, I get a log from nginx.

Note that I did not create any controllers. I am just using gce. Nevertheless, I tried adding kubernetes.io/ingress.class: "gce" with no effect.

I spent over 8 hours trying every possibility I could think of. I deleted and re-created the ingress probably 100-200 times. I even deleted and re-created the cluster.

Going back to the original configuration, where you cannot access either service via the Ingress LB, you consistently get 404 when trying to access the service. I have noticed it is slightly different when I include a slash at the end. Without it, it returns this to curl:

< Content-Length: 0
< Date: Sat, 12 Aug 2017 17:42:38 GMT
< Via: 1.1 google

When I add a slash, I get that with the content default backend - 404. E.g.,

curl -v http://host/collector/

Here is the original multi-path config:

kind: Ingress
metadata:
  name: ingress-combined
  annotations:
    ingress.kubernetes.io/rewrite-target: /
    kubernetes.io/ingress.global-static-ip-name: sc-static-ip
    kubernetes.io/ingress.class: "gce"
spec:
  rules:
  - http:
      paths:
      - path: /
        backend:
          serviceName: sc-admin
          servicePort: 80
      - path: /collector
        backend:
          serviceName: sc-collector
          servicePort: 80

This basic config works no matter which service I point it to:

kind: Ingress
metadata:
  name: basic-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: sc-static-ip
spec:
  backend:
    serviceName: sc-admin
    servicePort: 80

It is only multi-path with anything other than path: / that does not work.

Thank you in advance for your help.

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 19
  • Comments: 28 (4 by maintainers)

Most upvoted comments

I found a workaround. Using these paths works:

  - path: /*
  - path: /test/*

Note that http://host/test still resolves to the first service, whereas http://host/test/ resolves to the second service. The services don’t have to be human friendly URLs, so that is not an issue.

I clicked on every link in the Node app and couldn’t locate a single 404! YAY! curl seemed happy, too.

The question I have is how come I couldn’t find an example using a wildcard in the path in all the documentation on Ingress? The only reason I thought of it was because I figured out that GKE was using GCE URL Maps under the hood, and learned that it supports wild cards.

I did locate these discussions, which leave this as an open issue.

It has just been a long 10+ hour confusing ride due to documentation not being clear on this. And, to be sure, Google is #1 to earn credit here because I started out with and learned about Ingress from their GKE Setting up HTTP Load Balancing with Ingress documentation where their fanout example shows a root and a context under it pointing to two services WITHOUT asterisks! That was the example I started with, and the config I have been trying to get working. They really do need to update that page, and explain the impact of having or not having a wild card.

Hopefully, someone finds this issue in the first hour of their Ingress adventure. 😃

I am running K8S 1.6.7 on GKE. Having ingress.kubernetes.io/rewrite-target: / doesn’t help. When these errors occur, it is simply not getting to the service.

By default you get a GCE ingress controller running. Because of this:

  • kubernetes.io/ingress.class: "gce" annotation has no effect
  • ingress.kubernetes.io/rewrite-target annotation is not supported by the GCE ingress controller

Ingress version 0.22.0 or higher has changed the way how rewrite-target works. You’ll need to regex-match the path and add it to the rewrite-target. nginx.ingress.kubernetes.io/rewrite-target: /$2 and

  rules:
  - host: rewrite.bar.com
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 80
        path: /something(/|$)(.*)

Refer to changelog here How to article here

In my case, I based my solution in this tutorial and I have the following:

  1. FRONTEND: React Application
    • Path: /
    • Full URL doesn’t matter for application
  2. BACKEND
    • Path: /api
    • Full URL MATTERS for application
  3. I am using GCP

The only way it worked for me was creating 2 ingress, because If I use nginx.ingress.kubernetes.io/rewrite-target: / It breaks my backend that needs full url path

If I use nginx.ingress.kubernetes.io/rewrite-target: /$2 It breaks my frontend, showing a blank home page.

My solution:

  1. FRONTEND Ingress:
nginx.ingress.kubernetes.io/rewrite-target: /
path: /
  1. BACKEND Ingress:
nginx.ingress.kubernetes.io/rewrite-target: /$2
path: /api(/|$)(.*)

I attached full ingress file yaml.

Hope it helps and thank you so much, this thread helped me a lot!

I am facing the same issue using Nginx Ingress on AWS. Unfortunately it took me 10+ hours to find this thread. 😦

@nimendra Thank you for leaving that comment! We too had this, which was working until we upgraded from an old ingress-nginx version (0.9.0-beta.11 -> 0.22.0):

ingress.kubernetes.io/rewrite-target: /

After many hours, I found your comment and changed to this, which works:

nginx.ingress.kubernetes.io/rewrite-target: /

@lemonshow remove * from the paths

I sorted out by adding nginx.ingress.kubernetes.io/rewrite-target annotation. I’m using LetsEncrypt and not allowing http, but https.

apiVersion: extensions/v1beta1 kind: Ingress metadata: name: nginx-ingress namespace: default annotations: certmanager.k8s.io/cluster-issuer: letsencrypt-prod ingress.kubernetes.io/service-upstream: “true” kubernetes.io/ingress.class: “nginx” kubernetes.io/tls-acme: “true” kubernetes.io/ingress.allow-http: “false” nginx.ingress.kubernetes.io/from-to-www-redirect: “true” nginx.ingress.kubernetes.io/ssl-redirect: “true” nginx.ingress.kubernetes.io/force-ssl-redirect: “true” nginx.ingress.kubernetes.io/rewrite-target: /

Refer to Step 6 : Serving Multiple Applications on a Load Balancer on this Google Kubernetes Engine tutorial on Setting up HTTP Load Balancing with Ingress. It exactly answers the questions raised in this thread.

To route path / to backend service web on port 8080 and path /v2 to backend service web2 on port 8080 , the spec section of the ingress yaml file will be:

spec:
  rules:
  - http:
      paths:
      - path: /*
        backend:
          serviceName: web
          servicePort: 8080
      - path: /v2/*
        backend:
          serviceName: web2
          servicePort: 8080

Ensure that backend services respond with a 200OK for /request. Note that web2 needs to respond with a 200OK to a root path / ( not /v2/). This is important for Google Load Balancer to perform a successful health check on the two services before making them available. If they do not pass the health checks, you would get a 502 Error.

In our case, we have a angular frontend service with an express backend. We want to angular to serve / and express to serve /api for the same host.

So, in our express request handling file, we have the following added to respond with a 200OK for path /.

const app = express();
app.use('/', function (req, res, next) {
	res.sendStatus(200);
	next();
});

Hope this helps.

Hi, I have the Nginx Ingress running without issues in Production and HTTPS by using LetsEncrypt certificates:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
  namespace: default
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    ingress.kubernetes.io/service-upstream: "true"
    kubernetes.io/ingress.class: "nginx"
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.allow-http: "false"
    nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/proxy-body-size: "1M"
spec:
  tls:
    - hosts:
      - developer.test.com
      secretName: developer-test-com-tls
    - hosts:
      - broker.test.com
      secretName: broker-test-com-tls
    - hosts:
      - dashboard.test.com
      secretName: dashboard-test-com-tls
  rules:
    - host: developer.test.com
      http:
        paths:
          - path: /
            backend:
              serviceName: developer-docs-service
              servicePort: 8080
          - path: /api
            backend:
              serviceName: developer-service
              servicePort: 8080
    - host: broker.test.com
      http:
        paths:
          - path: /
            backend:
              serviceName: broker-rabbitmq-service
              servicePort: 15672
    - host: dashboard.test.com
      http:
        paths:
          - path: /
            backend:
              serviceName: dashboard-service
              servicePort: 80

I set up a quick repo with the resources I used for the production environment. You need to replace the “test” subdomain with your own and set up the SSL certificates.

Hope it helps.

if you create 2 ingress, do you have to pay for 2 load balancers in GCP?

@LorenzoR no with ingress-nginx

It works now! Actually I had another issue after this where I was testing with NLB but I comment out https. I saw 308 redirect to https. Uncomment https, things start to work. thanks a lot,

@erik777 I got the same issue. The base works but the multi-path not. I am using Nginx ingress controller + AWS NLB and try to bring up my two services with the Ingress. It always routes to default and I am getting 404. Anyone used with AWS EKS with luck? Use rewrite did not help either. Any other suggestion? thanks,

apiVersion: extensions/v1beta1 kind: Ingress metadata: name: test-ingress namespace: local annotations: ingress.kubernetes.io/rewrite-target: ‘/’ spec: #backend: #serviceName: app1-service #servicePort: 80 rules:

  • http: paths:
    • backend: serviceName: app1-service servicePort: 80 path: /*
    • backend: serviceName: app2-service servicePort: 80 path: /app2/*

In my case, I had two services, one running at / and another at /ui . I tried several combinations and the only one that worked with K8s 1.10 was

http://mydomain.com/* Service 1 Ingress kubernetes.io/ingress.class: “nginx” path: /

http://mydonain.com/ui/app/index.html Service 2 Ingress kubernetes.io/ingress.class: “nginx” nginx.ingress.kubernetes.io/rewrite-target: /ui path: /ui/*

Note that for service 2, my tomcat was also initialising at /ui , so the rewrite-target did the trick otherwise, I had to add two “ui” to my url like http://mydonain.com/ui/ui/app/index.html

I have a similar issue with Azure AKS (v1.9.6) and a nginx (nginx-ingress-controller-0.15.0) :

I want to have different paths to route requests to different backends.

The following configuration works, but when I request “https://xxx/iv-da/swagger”, the /swagger/v0/swagger.json AJAX call makes a request to “https://xxx/swagger/v0/swagger.json” without the “iv-da” context.

How could it keep the context for each backend’s inside requests ?

{
  "kind": "Ingress",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "name": "k8s-ingress",
    "namespace": "default",
    "annotations": {
      "certmanager.k8s.io/cluster-issuer": "letsencrypt-prod",
      "kubernetes.io/ingress.class": "nginx",
      "nginx.ingress.kubernetes.io/rewrite-target": "/"
    }
  },
  "spec": {
    "tls": [
      {
        "hosts": [
          "xxx"
        ],
        "secretName": "tls-secret"
      }
    ],
    "rules": [
      {
        "host": "xxx",
        "http": {
          "paths": [
            {
              "path": "/iv-da",
              "backend": {
                "serviceName": "iv-da",
                "servicePort": 80
              }
            },
            {
              "path": "/",
              "backend": {
                "serviceName": "config-server",
                "servicePort": 80
              }
            }
          ]
        }
      }
    ]
  }
}

@felipecrescencio if you create 2 ingress, do you have to pay for 2 load balancers in GCP?

@dolphub I ended up doing this :

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
  name: myingress
spec:
  rules:
  - host: myhost.com
    http:
      paths:
      - backend:
          serviceName: api
          servicePort: 80
        path: /domain-api
      - backend:
          serviceName: bff
          servicePort: 80
        path: /api
      - backend:
          serviceName: front
          servicePort: 80
        path: /
  tls:
  - hosts:
    - myhost.com
    secretName: tls-secret

And for the swagger problem, it is inside the code that we need to force the context, it is not an ingress problem.

Hope it helps.