ingress-nginx: Ingress canary service responded when sending a request with a canary header to an ingress controller that does not have a matching canary enabled ingress resource.

Use case: while trying to replace only one of the many API endpoints with a new implementation , trying to leverage ingress canary to switch over certain calls by header over to a new hosting, with keeping the majority of the calls as how they were

approach tried: for example if the call has a header Canary with value enabled, route the traffic to “new” service, otherwise route it to the old service, the rule used was to match with host and path

with the default ingress in place pointing to the “old” as such :

rules:
  - host: "api.{{ $.Values.ingress.zone }}.com"
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            serviceName:  old-service
            servicePort: 80

first attempt made: added a new ingress rule with canary enabled : true

rules:
  - host: "api.{{ $.Values.ingress.zone }}.com"
    http:
      paths:
        - path: /my-endpoint-to-be-replaced
          pathType: Exact
          backend:
            serviceName:  new-service
            servicePort: 80

the traffic appears to be routed either always to the new service or always to the old one

second attempt made added the canary ingress with the above"exact " rule
added another same “exact” rule to act as the “main” ingress, hoping this is the rule being overridden by the canary rule

the result shown for the target endpoint /my-endpoint-to-be-replaced, it worked as desired , however, when regression testing the other endpoints hosted by the “old” service, they are all following the canary patterns and being routed to the “new” service if match is successful, which ends up 404 as they are not ready yet

tested with version

repository: k8s.gcr.io/ingress-nginx/controller
tag: "v0.46.0"

/kind bug

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 37 (18 by maintainers)

Most upvoted comments

Talk at kubernetes.slack.com in the ingress-nginx-users channel please. There are several developers and engineers there. Thanks,

; Long

20 Jul 2021, 16:47 by @.***:

the documentation of the canary feature says it matches the rule with host name and path, are you suggesting that it should only match with host name because this is what you see it behaving ?

give me a reason why I would want to use canary if I can use a different hostname

do you think the fact that "service2 came into play at path > /> when the only place that points to service2 is the one canary ingress that matches exact path as > /product> " is a resonable behavior at all?

— You are receiving this because you were mentioned. Reply to this email directly, > view it on GitHub https://github.com/kubernetes/ingress-nginx/issues/7370#issuecomment-883311316> , or > unsubscribe https://github.com/notifications/unsubscribe-auth/ABGZVWTTGSHDDMZH7BUJZATTYVLMZANCNFSM5ASAPZFA> .

longwuyuan on Jul 20, 2021

No. I looked at the controller code. I can not understand much. But from the annotation docs and the code, I am convinced that either you can have canary routing “true” for a hostname Or you can use a different hostname for your google-service backend (my httpd:alpine backend).

Someone who understands the code has to comment now I suppose. Thanks,

; Long

20 Jul 2021, 16:23 by @.***:

thanks! does this title look happy to you ? have we agreed that this is not a logical behavior that should be corrected? hope I expressed everything clear

— You are receiving this because you were mentioned. Reply to this email directly, > view it on GitHub https://github.com/kubernetes/ingress-nginx/issues/7370#issuecomment-883297049> , or > unsubscribe https://github.com/notifications/unsubscribe-auth/ABGZVWWBZH2KNXTMPROONQLTYVIS5ANCNFSM5ASAPZFA> .

longwuyuan on Jul 20, 2021

Also, it is now clear that there is no data showing a bug here ? My suggestion is that you close this issue here now, and take a different approach to solvintg this problem as described below.

Please discuss this at kubernetes.slack.com in the channel ingress-nginx-users with the develoeprs and engineers there. If you find this to be a bug, then you can re-open this issue and add the data related to the bug.

longwuyuan on Jul 20, 2021