istio: Secret update caused massive SecretTrigger
Describe the feature request
Currently NewCredentialsController watches secrets in the mesh and doesn’t honor the discoverySelector:
fieldSelector := fields.AndSelectors(
fields.OneTermNotEqualSelector("type", "helm.sh/release.v1"),
fields.OneTermNotEqualSelector("type", string(v1.SecretTypeServiceAccountToken))).String()
secrets := kclient.NewFiltered[*v1.Secret](kc, kclient.Filter{
FieldSelector: fieldSelector,
In our mesh, there is churn for the opaque secret, not very high but around 1.7k per two minutes:
However that rate of churn caused millions of secret push triggers:
There are around 4.5k sidecar in the mesh and 12 istiod pods, so each istiod has around 400 XDS connections.
The root cause of the millions of secret push trigger is that, for each secret update, it’s a kind of configure update and will trigger each istiod full push to all xds clients:
// Send a signal to all connections, with a push event.
func (s *DiscoveryServer) StartPush(req *model.PushRequest) {
// Push config changes, iterating over connected envoys.
if log.DebugEnabled() {
currentlyPending := s.pushQueue.Pending()
if currentlyPending != 0 {
log.Debugf("Starting new push while %v were still pending", currentlyPending)
}
}
req.Start = time.Now()
for _, p := range s.AllClients() {
s.pushQueue.Enqueue(p, req)
}
}
So the 1.7k churn will increase 400 times due to each secret churn will cause push trigger to all proxies connected to the istiod and result to millions of secret trigger.
Proposed solutions
- Add an option to exclude the Opaque secret since in our case we don’t need istiod to watch Opaque secret.
fields.OneTermNotEqualSelector("type", string(v1.SecretTypeOpaque)),
- When istiod sends push to all clients, it can check if the trigger reason is secret and will enque only if the secret namespace matches proxy namespace(ideally secret should not be cross namespace referred), some draft code like below:
for _, p := range s.AllClients() {
if len(req.Reason) == 1 && req.Reason[0] == model.SecretTrigger {
for c := range req.ConfigsUpdated {
if len(req.ConfigsUpdated) == 1 && c.Namespace != p.proxy.Metadata.Namespace {
contiue;
}
}
}
s.pushQueue.Enqueue(p, req)
}
Affected product area (please put an X in all that apply)
[ ] Ambient [ ] Docs [ ] Dual Stack [ ] Installation [ x] Networking [ ] Performance and Scalability [ ] Extensions and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure
Affected features (please put an X in all that apply)
[ ] Multi Cluster [ ] Virtual Machine [ ] Multi Control Plane
Additional context
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 15 (15 by maintainers)
I do not think so, if you want a metric indicating the number of pushes there is already one. I think you may just be looking at the wrong metric for your usage