istio: Secret update caused massive SecretTrigger

Describe the feature request

Currently NewCredentialsController watches secrets in the mesh and doesn’t honor the discoverySelector:

        fieldSelector := fields.AndSelectors(
		fields.OneTermNotEqualSelector("type", "helm.sh/release.v1"),
		fields.OneTermNotEqualSelector("type", string(v1.SecretTypeServiceAccountToken))).String()
	secrets := kclient.NewFiltered[*v1.Secret](kc, kclient.Filter{
		FieldSelector: fieldSelector,

In our mesh, there is churn for the opaque secret, not very high but around 1.7k per two minutes: Screenshot 2024-07-04 at 18 59 07

However that rate of churn caused millions of secret push triggers: Screenshot 2024-07-04 at 19 00 40

There are around 4.5k sidecar in the mesh and 12 istiod pods, so each istiod has around 400 XDS connections.

Screenshot 2024-07-04 at 19 02 33

The root cause of the millions of secret push trigger is that, for each secret update, it’s a kind of configure update and will trigger each istiod full push to all xds clients:

// Send a signal to all connections, with a push event.
func (s *DiscoveryServer) StartPush(req *model.PushRequest) {
	// Push config changes, iterating over connected envoys.
	if log.DebugEnabled() {
		currentlyPending := s.pushQueue.Pending()
		if currentlyPending != 0 {
			log.Debugf("Starting new push while %v were still pending", currentlyPending)
		}
	}
	req.Start = time.Now()
	for _, p := range s.AllClients() {
		s.pushQueue.Enqueue(p, req)
	}
}

So the 1.7k churn will increase 400 times due to each secret churn will cause push trigger to all proxies connected to the istiod and result to millions of secret trigger.

Proposed solutions

  1. Add an option to exclude the Opaque secret since in our case we don’t need istiod to watch Opaque secret.
fields.OneTermNotEqualSelector("type", string(v1.SecretTypeOpaque)),
  1. When istiod sends push to all clients, it can check if the trigger reason is secret and will enque only if the secret namespace matches proxy namespace(ideally secret should not be cross namespace referred), some draft code like below:
        for _, p := range s.AllClients() {
		if len(req.Reason) == 1 && req.Reason[0] == model.SecretTrigger {
			for c := range req.ConfigsUpdated {
				if len(req.ConfigsUpdated) == 1 && c.Namespace != p.proxy.Metadata.Namespace {
					contiue;
				}
			}
		}
		s.pushQueue.Enqueue(p, req)
	}

Affected product area (please put an X in all that apply)

[ ] Ambient [ ] Docs [ ] Dual Stack [ ] Installation [ x] Networking [ ] Performance and Scalability [ ] Extensions and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure

Affected features (please put an X in all that apply)

[ ] Multi Cluster [ ] Virtual Machine [ ] Multi Control Plane

Additional context

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (15 by maintainers)

Most upvoted comments

I do not think so, if you want a metric indicating the number of pushes there is already one. I think you may just be looking at the wrong metric for your usage