envoy: Changing locality priority level causes the locality to be duplicated

Title: Changing the priority on a locality in a cluster with active health checks configured results in the locality being duplicated

Description: mpuncel/envoy@b4e3af34dd06 has a failing test case demonstrating this.

Example: if there are two localities L1 and L2 which are both included in an EDS update with priority 0 set for each, followed by an update with L1 in p0 and L2 in p0, Envoy ends up thinking that L2 is in both priority 0 and priority 1.

This seems to be due to an assumption within EdsClusterImpl that a locality will never move to a different priority. For that reason I’m guessing this would be a complex behavior change to implement.

Changing the behavior here to allow moving a locality to a different priority is useful for situations like disaster recovery exercises, for instance to test that an upstream service can take over for the geographically closest locality in the event of a datacenter or network outage.

Using the recently added drain_connections_on_host_removal cluster option might be a viable workaround, but with this behavior change this would be possible without giving up the eventual consistency design choice of Envoy (and i’m also not sure if the draining is destructive in this scenario).