pulumi: very slow `pulumi up`, possibly due to slow snapshot-related operations
our organization stacks are taking unbearable amount of time to run even on a simple no change pulumi up. Stack with 165 resources takes ~11 minutes to complete with no changes made and no refresh option used.
--logtostderr -v1 option generates ~15M log file with a steady stream of events with no obvious gaps, except for a single entry:
I0903 11:06:28.070124 1901 snapshot.go:473] Marked old state snapshot as done: urn:pulumi:gke01-london::gcp-gke::nakhoda:CertManager$kubernetes:core/v1:Namespace$kubernetes:yaml:ConfigFile::cert-manager-role-binding
each Marked old state snapshot as done seems to be taking 2.3-2.5 seconds and there is one such record for ever resource, so just that step alone is 6.3 minutes. I am yet to find where rest of time is spent, but that is a start.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 4
- Comments: 22 (10 by maintainers)
Commits related to this issue
- Cache ciphertext for secret properties. This caching is enabled by wrapping the `secrets.Manager` returned by `DefaultSecretsProvider.OfType` in an outer `secrets.Manager` that cooperates with `stack... — committed to pulumi/pulumi by pgavlin 5 years ago
- Cache ciphertext for secret properties. (#3183) This caching is enabled by wrapping the `secrets.Manager` returned by `DefaultSecretsProvider.OfType` in an outer `secrets.Manager` that cooperates w... — committed to pulumi/pulumi by pgavlin 5 years ago
@lukehoban How about
Okay, build
1.2.0-alpha.1568668603has support for emitting tracing data to a file. You can either build from source (just build atHEADinmaster), or install this version like so:curl -sSL https://get.pulumi.com | sh -s -- --version 1.2.0-alpha.1568668603. Once installation has finished, you can collect a local execution trace by passing afile:URL to--tracing:You can then email us the trace and we may be able to glean some more information.