envoy: Slow startup speed triggered by "Secret is updated"

Title: Slow startup speed triggered by “Secret is updated”

Description: We have had a few reports of slow startup speed of Envoy. They seem primarily revolving around slow SDS processing.

This has been reported by multiple users. In one case, we see 1600 clusters with SDS references to two shared certificates.

If we look at a plot of “Secret is updated” logs, we see an interesting pattern:

Graph is a bit odd but this is showing time (y axis, seconds since arbitrary time) vs “Secret is updated” logs (x axis, 1 tick per log occurrence).

So we see three distinct phases:

1. process all of them extremely fast
2. process all of them slowly, 1st time
3. process all of them slowly, 2nd time

2/3 correlate with

2022-01-12T05:11:59.628553Z     debug   envoy config    gRPC config update took 4372 ms! Resources names: default
2022-01-12T05:12:04.193882Z     debug   envoy config    gRPC config update took 4537 ms! Resources names: ROOTCA

It would be ideal if it did not take 5s (x2) to process these clusters

cc @lambdai