istio: Priority for VirtualService when conflict happen

Why is this issue closed? I met a similar situation, is there any solution?

This forces a significant deploy order for VirtualServices.

I vote no.

What if I need to update a VirtualService that needs to have the highest priority? Then I need to redeploy all the other VirtualServices too?

Can we just have sensible defaults that can be overridden with some priority setting at the VirtualService level?

EDIT For what it’s worth, the defaults are already quite sensible. I just need to let my deploy guy deploy in whatever order he wants. Who am I to tell him how to do his job?

EDIT 2 Actually we’re just going to combine all the VirtualServices into a single resource.

EDIT 3 Actually this is really annoying. We really need a better way to define which VirtualService takes priority. I have several VirtualServices that I cannot combine because they handle different hosts. However some of the backend prefixes clash. I’m stuck with one VirtualService that is eating all the requests that need to go to another VirtualService.

EDIT 4 For what it’s worth, why can’t we just look at the most specific path first? This would fix everything.

I am facing a similar scenario. I think the logic is sensible despite order or creation timestamp:

a. Priority: exact > prefix > regex

b. The longer route would always override the shorter one when using prefix or regex uri, eg: /api/serviceA/action >/api/serviceA

c. The header matched route would override the other one when they both matched in uri, eg: /api/serviceA/action > /api/serviceA + header: version=1 >/api/serviceA

I would like to reopen this issue. We have a significant issue here with managing distributed VirtualServices. Having the most specific route match take priority, as @mehemken mentioned, would help. Although I can imagine this would also introduce issues (eg. how do you compare specificity between a regex and prefix match rule).

I propose a different solution. Though I need to specify that I’m unfamiliar with the codebase here, so I don’t know how much work it would be.

1. Prevent entirely conflicting / overriding matching rules to other VirtualServices
2. Do not prevent conflicting / overriding matching rules within a single VirtualService

Why 1: Currently, there is no way to enforce prioritization. The priorities are loosely handed to the matching VS rule that was mostly recently created. This makes it an extremely difficult task for operations / infrastructure teams to manage distributed VSs. However, this distributed VS capability is a requirement, and is the actual recommendation from Istio itself! The best microservice developer experience would be to manage their own VS, and be notified when they attempt to create a VS rule that conflicts with another team’s VS rule. Then those two teams work together to come to an agreement. This takes ownership of prioritizing rules off of the DevOps teams.

Why 2: There is a somewhat acceptable way to manage prioritization within the same VS at the moment- the controller reads the VS rules from top to bottom. While this isn’t exactly ideal, it allows a microservice team to prioritize how their traffic. The microservice team is now already in charge of resolving conflicting routing paths, and has full autonomy to address their own problems. The DevOps teams are again relieved of the job of managing rule prioritization.

implementing the priority field is definitely a must. we have a monolith which we are breaking down into microservices and all requests are sent with the same host. we cant distribute the routing rules for each microservice since its all being matched to the shorter uri which should actually be the last one matched if no other rules are met.

I’ve implement an operator and a crd “sub virtual service” to handle the priority. Please fell free to check whether it’s useful for you. https://github.com/ErikXu/virtual-service-go

maybe you can dig into the code and contribute this to the original istio solution ?

I vote reopen this issue too. Priority feature can make vs more flexible. And all virtual services with same hosts will be merged by create timestamp, the order is out of control.

For example with a problem we are having now: We have multiple virtual services with a same host, we need to set a default route to this host, but we can’t add the default route to any of virtual service cause we can’t ensure the orders of merged virtual services.

Same as @amitde69 /stale, voting for reopen

We can’t do the same thing as gateway-api since it wouldn’t be backwards compatible (it’s not a new field, just different implicit behavior)

https://istio.io/latest/docs/ops/best-practices/traffic-management/#split-virtual-services

As the current behaviour is undefined, I think it’s not a breaking change (or requires a flag) to make it defined?

This would be an egregious breaking change that would cause front page news outages. “not well documented” does not imply we can break everyone unfortunately. There are literally millions of virtualservices in production at this point

@howardjohn @ramaraochavali This problem is worth a solution, it is usually a big blocker for people setting up south-north traffic policy. I am totally agree it should align with gateway API merging strategy

For all the above comments: the http match conditions are complex, code can not handle it well.

type HTTPMatchRequest struct {
	// The name assigned to a match. The match's name will be
	// concatenated with the parent route's name and will be logged in
	// the access logs for requests matching this route.
	Name string `protobuf:"bytes,11,opt,name=name,proto3" json:"name,omitempty"`
	// URI to match
	// values are case-sensitive and formatted as follows:
	//
	// - `exact: "value"` for exact string match
	//
	// - `prefix: "value"` for prefix-based match
	//
	// - `regex: "value"` for RE2 style regex-based match (https://github.com/google/re2/wiki/Syntax).
	//
	// **Note:** Case-insensitive matching could be enabled via the
	// `ignore_uri_case` flag.
	Uri *StringMatch `protobuf:"bytes,1,opt,name=uri,proto3" json:"uri,omitempty"`
	// URI Scheme
	// values are case-sensitive and formatted as follows:
	//
	// - `exact: "value"` for exact string match
	//
	// - `prefix: "value"` for prefix-based match
	//
	// - `regex: "value"` for RE2 style regex-based match (https://github.com/google/re2/wiki/Syntax).
	//
	Scheme *StringMatch `protobuf:"bytes,2,opt,name=scheme,proto3" json:"scheme,omitempty"`
	// HTTP Method
	// values are case-sensitive and formatted as follows:
	//
	// - `exact: "value"` for exact string match
	//
	// - `prefix: "value"` for prefix-based match
	//
	// - `regex: "value"` for RE2 style regex-based match (https://github.com/google/re2/wiki/Syntax).
	//
	Method *StringMatch `protobuf:"bytes,3,opt,name=method,proto3" json:"method,omitempty"`
	// HTTP Authority
	// values are case-sensitive and formatted as follows:
	//
	// - `exact: "value"` for exact string match
	//
	// - `prefix: "value"` for prefix-based match
	//
	// - `regex: "value"` for RE2 style regex-based match (https://github.com/google/re2/wiki/Syntax).
	//
	Authority *StringMatch `protobuf:"bytes,4,opt,name=authority,proto3" json:"authority,omitempty"`
	// The header keys must be lowercase and use hyphen as the separator,
	// e.g. _x-request-id_.
	//
	// Header values are case-sensitive and formatted as follows:
	//
	// - `exact: "value"` for exact string match
	//
	// - `prefix: "value"` for prefix-based match
	//
	// - `regex: "value"` for RE2 style regex-based match (https://github.com/google/re2/wiki/Syntax).
	//
	// If the value is empty and only the name of header is specfied, presence of the header is checked.
	// **Note:** The keys `uri`, `scheme`, `method`, and `authority` will be ignored.
	Headers map[string]*StringMatch `protobuf:"bytes,5,rep,name=headers,proto3" json:"headers,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
	// Specifies the ports on the host that is being addressed. Many services
	// only expose a single port or label ports with the protocols they support,
	// in these cases it is not required to explicitly select the port.
	Port uint32 `protobuf:"varint,6,opt,name=port,proto3" json:"port,omitempty"`
	// One or more labels that constrain the applicability of a rule to source (client) workloads
	// with the given labels. If the VirtualService has a list of gateways specified
	// in the top-level `gateways` field, it must include the reserved gateway
	// `mesh` for this field to be applicable.
	SourceLabels map[string]string `protobuf:"bytes,7,rep,name=source_labels,json=sourceLabels,proto3" json:"source_labels,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
	// Names of gateways where the rule should be applied. Gateway names
	// in the top-level `gateways` field of the VirtualService (if any) are overridden. The gateway
	// match is independent of sourceLabels.
	Gateways []string `protobuf:"bytes,8,rep,name=gateways,proto3" json:"gateways,omitempty"`
	// Query parameters for matching.
	//
	// Ex:
	//
	// - For a query parameter like "?key=true", the map key would be "key" and
	//   the string match could be defined as `exact: "true"`.
	//
	// - For a query parameter like "?key", the map key would be "key" and the
	//   string match could be defined as `exact: ""`.
	//
	// - For a query parameter like "?key=123", the map key would be "key" and the
	//   string match could be defined as `regex: "\d+$"`. Note that this
	//   configuration will only match values like "123" but not "a123" or "123a".
	//
	// **Note:** `prefix` matching is currently not supported.
	QueryParams map[string]*StringMatch `protobuf:"bytes,9,rep,name=query_params,json=queryParams,proto3" json:"query_params,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
	// Flag to specify whether the URI matching should be case-insensitive.
	//
	// **Note:** The case will be ignored only in the case of `exact` and `prefix`
	// URI matches.
	IgnoreUriCase bool `protobuf:"varint,10,opt,name=ignore_uri_case,json=ignoreUriCase,proto3" json:"ignore_uri_case,omitempty"`
	// withoutHeader has the same syntax with the header, but has opposite meaning.
	// If a header is matched with a matching rule among withoutHeader, the traffic becomes not matched one.
	WithoutHeaders map[string]*StringMatch `protobuf:"bytes,12,rep,name=without_headers,json=withoutHeaders,proto3" json:"without_headers,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
	// Source namespace constraining the applicability of a rule to workloads in that namespace.
	// If the VirtualService has a list of gateways specified in the top-level `gateways` field,
	// it must include the reserved gateway `mesh` for this field to be applicable.
	SourceNamespace      string   `protobuf:"bytes,13,opt,name=source_namespace,json=sourceNamespace,proto3" json:"source_namespace,omitempty"`
	XXX_NoUnkeyedLiteral struct{} `json:"-"`
	XXX_unrecognized     []byte   `json:"-"`
	XXX_sizecache        int32    `json:"-"`
}

But looks like introducing priority for http route is necessary and is an imperious demand.

@hzxuzhonghu I understand they’re different, I only meant the behavior to give precedence to the longest path match makes intuitive sense for any “prefix” type path matching behavior.

We can’t do the same thing as gateway-api since it wouldn’t be backwards compatible (it’s not a new field, just different implicit behavior)

https://istio.io/latest/docs/ops/best-practices/traffic-management/#split-virtual-services

As the current behaviour is undefined, I think it’s not a breaking change (or requires a flag) to make it defined?

Hit this issue as well across unrelated VirtualService’s owned by different teams when migrating from Kubernetes Ingress to VirtualServices. We’ve decided to use regex paths by default to avoid this.

One of the frustrating things is this behavior (giving priority to whoever is first) is inconsistent with how K8s Ingress handles conflicts. It would be nice if there was an option on Gateway to opt into this behavior. Or vice-versa if an appropriate time was found to introduce the breaking change, allow people to opt into the old behavior.

@howardjohn can we reopen this? Do I understand correctly that if we do something as compatible with gateway-api as possible, it will be accepted in istio?

For example, I created a virtual service using kubectl apply -f service-a.yaml.

# service-a.yaml 
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: service-a-vs
spec:
  hosts:
    - "www.example.com"
  gateways:
  - example-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        port:
          number: 80
        host: service-a

Then I have a new route:

# service-b.yaml 
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: service-b-vs
spec:
  hosts:
    - "www.example.com"
  gateways:
  - example-gateway
  http:
  - match:
    - uri:
        prefix: /api
    route:
    - destination:
        port:
          number: 80
        host: service-b

I don’t want to use the below commands to recreate.

kubectl delete -f service-a.yaml
kubectl apply -f service-b.yaml 
kubectl apply -f service-a.yaml 

Instead, I want to use the below commands to reload.

kubectl apply -f service-b.yaml 
kubectl apply -f service-a.yaml