istio: Priority for VirtualService when conflict happen
Describe the feature request Hi, I’m wondering is it possible to set priority for each VirtualService when multi VirtualService Match get conflict.
In my case , I have 2 services , A and B. And my virtualservice configuration is following YAML.
When I visit www.example.com/api/xxx ,i noticed that the http request was arranged to the Service-A by istio-ingressgateway.
It seems the request also hit the virtualService Match of Service-A .
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: vsc-A
spec:
hosts:
- "www.example.com"
gateways:
- example-gateway
http:
- match:
- uri:
prefix: /
route:
- destination:
port:
number: 80
host: service-A
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: vsc-B
spec:
hosts:
- "www.example.com"
gateways:
- example-gateway
http:
- match:
- uri:
prefix: /api
route:
- destination:
port:
number: 80
host: service-B
Describe alternatives you’ve considered
I guess we can add define priority attribute in virtualService in following way. As 99 means the lowest level and 0 means the highest level, the request in my case will be arranged to the service-B
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: A
spec:
hosts:
- "www.example.com"
gateways:
- example-gateway
http:
- match:
- uri:
priority: 99
prefix: /
route:
- destination:
port:
number: 80
host: service-A
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: account-service-external-vsc
spec:
hosts:
- "www.example.com"
gateways:
- example-gateway
http:
- match:
- uri:
priority: 0
prefix: /api
route:
- destination:
port:
number: 80
host: service-B
Additional context
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Reactions: 47
- Comments: 66 (33 by maintainers)
Why is this issue closed? I met a similar situation, is there any solution?
This forces a significant deploy order for
VirtualServices.I vote no.
What if I need to update a
VirtualServicethat needs to have the highest priority? Then I need to redeploy all the otherVirtualServicestoo?Can we just have sensible defaults that can be overridden with some
prioritysetting at theVirtualServicelevel?EDIT For what it’s worth, the defaults are already quite sensible. I just need to let my deploy guy deploy in whatever order he wants. Who am I to tell him how to do his job?
EDIT 2 Actually we’re just going to combine all the
VirtualServicesinto a single resource.EDIT 3 Actually this is really annoying. We really need a better way to define which VirtualService takes priority. I have several VirtualServices that I cannot combine because they handle different hosts. However some of the backend prefixes clash. I’m stuck with one VirtualService that is eating all the requests that need to go to another VirtualService.
EDIT 4 For what it’s worth, why can’t we just look at the most specific path first? This would fix everything.
I am facing a similar scenario. I think the logic is sensible despite order or creation timestamp:
a. Priority:
exact>prefix>regexb. The longer route would always override the shorter one when using
prefixorregexuri, eg:/api/serviceA/action>/api/serviceAc. The header matched route would override the other one when they both matched in uri, eg:
/api/serviceA/action>/api/serviceA + header: version=1>/api/serviceAI would like to reopen this issue. We have a significant issue here with managing distributed VirtualServices. Having the most specific route match take priority, as @mehemken mentioned, would help. Although I can imagine this would also introduce issues (eg. how do you compare specificity between a
regexandprefixmatch rule).I propose a different solution. Though I need to specify that I’m unfamiliar with the codebase here, so I don’t know how much work it would be.
Why 1: Currently, there is no way to enforce prioritization. The priorities are loosely handed to the matching VS rule that was mostly recently created. This makes it an extremely difficult task for operations / infrastructure teams to manage distributed VSs. However, this distributed VS capability is a requirement, and is the actual recommendation from Istio itself! The best microservice developer experience would be to manage their own VS, and be notified when they attempt to create a VS rule that conflicts with another team’s VS rule. Then those two teams work together to come to an agreement. This takes ownership of prioritizing rules off of the DevOps teams.
Why 2: There is a somewhat acceptable way to manage prioritization within the same VS at the moment- the controller reads the VS rules from top to bottom. While this isn’t exactly ideal, it allows a microservice team to prioritize how their traffic. The microservice team is now already in charge of resolving conflicting routing paths, and has full autonomy to address their own problems. The DevOps teams are again relieved of the job of managing rule prioritization.
I vote reopen this issue too. Priority feature can make vs more flexible. And all virtual services with same hosts will be merged by create timestamp, the order is out of control.
For example with a problem we are having now: We have multiple virtual services with a same host, we need to set a default route to this host, but we can’t add the default route to any of virtual service cause we can’t ensure the orders of merged virtual services.
Can we publicly document this creation timestamp behavior today on https://istio.io/latest/docs/ops/best-practices/traffic-management/#split-virtual-services if we are not planning to modify this behavior?
Just to add another scenario where this would be useful (which I’ve hit now)…
We want to introduce a maintenance window (which redirects our app to another pod) by creating another virtualservice that overrides main one.
The reason we want to leave the first VS alone is because that is deployed by our pipelines with Helm. If we change the main VirtualService, then when we deploy the new version of our app (which we’ll probably be doing in maintenance mode) - then it’ll re-deploy the VS again (effectively turning off maintenance mode).
I want to create a new VS to enable maintenance mode, then delete it to turn off maintenance mode. Unfortunately, my second VS only works when I delete the first one. I’d like to be able to give my overridden VS a higher precedence.
implementing the priority field is definitely a must. we have a monolith which we are breaking down into microservices and all requests are sent with the same host. we cant distribute the routing rules for each microservice since its all being matched to the shorter uri which should actually be the last one matched if no other rules are met.
maybe you can dig into the code and contribute this to the original istio solution ?
Same as @amitde69 /stale, voting for reopen
/reopen
@howardjohn I don’t see in HTTPRoute possibility to set priority. This is from VirtualService.
Question here is how to handle situations properly when we have several VirtualService’s that serves traffic for the same url.
Example:
http://example.com/api->microservice1http://example.com/api/accounts->microservice2Need have possibility to set high priority for prefix
/api/accountsand route traffic tomicroservice2instead ofmicroservice1.Can you provide a solution here?
@hzxuzhonghu I understand they’re different, I only meant the behavior to give precedence to the longest path match makes intuitive sense for any “prefix” type path matching behavior.
I had I guess same issue. The part of Istio VirtualService configuration is being applied instantly. But, WHEN changing routes order to have the more specific route to be processed First and less specific After, then need to run this magic command on Kube to restart Istio System.
kubectl rollout restart deployment istiod -n istio-systemAnd only then the Route order for this specific case are being applied correctly.
@howardjohn @ramaraochavali This problem is worth a solution, it is usually a big blocker for people setting up south-north traffic policy. I am totally agree it should align with gateway API merging strategy
For all the above comments: the http match conditions are complex, code can not handle it well.
But looks like introducing priority for http route is necessary and is an imperious demand.
This would be an egregious breaking change that would cause front page news outages. “not well documented” does not imply we can break everyone unfortunately. There are literally millions of virtualservices in production at this point
My understanding is the issue here is not about breaking ties. it’s that if I create /foo before /foo/bar with VS,/foo takes precedence.
In gateway,/foo/bar does.
not stale
Yes, you are right, first route mateched can be guaranteed to be routed to
https://istio.io/latest/docs/ops/best-practices/traffic-management/#split-virtual-services
As the current behaviour is undefined, I think it’s not a breaking change (or requires a flag) to make it defined?
Different requirements, original issue indeed has two VS https://github.com/istio/istio/issues/10464#issue-390625341
User are most concerned about multi VS priority, we order them by creation timestamp, which is same as Gateway API.
But different on http match orders within a single VS with Gateway API. To be clear, gateway api does not order matches globally across multi HttpRoute.
I am ok if the requirement is to add priority to each VS
Hi Team, sorry to bother, I want to confirm if I only have
onevirtualserivce and it have 2 matches. First match: header + uri ===> aaa Second match: uri ===> aaa-oldIf my request has both header and uri, I can 100% route to aaa(First one)? If not, any suggestions?
I checked the document, the
spec.http.matchlist order should works?If ties still exist within an HTTPRoute, matching precedence MUST be granted to the FIRST matching rule (in list order) with a match meeting the above criteria.Thanks!
Yeah, that is ideal, in real use case users can still meet conflict. Even not sure how this ordered, not to mention regex semantics.
So in a word, k8s HTTPRoute almost have similar issue as istio VS
Hit this issue as well across unrelated VirtualService’s owned by different teams when migrating from Kubernetes Ingress to VirtualServices. We’ve decided to use regex paths by default to avoid this.
One of the frustrating things is this behavior (giving priority to whoever is first) is inconsistent with how K8s Ingress handles conflicts. It would be nice if there was an option on Gateway to opt into this behavior. Or vice-versa if an appropriate time was found to introduce the breaking change, allow people to opt into the old behavior.
@howardjohn can we reopen this? Do I understand correctly that if we do something as compatible with gateway-api as possible, it will be accepted in istio?
For example, I created a virtual service using
kubectl apply -f service-a.yaml.Then I have a new route:
I don’t want to use the below commands to recreate.
Instead, I want to use the below commands to reload.