provider-terraform: Reconciliations blocking unexpectedly
What happened?
Fairly regularly, we’re seeing reconciliations back up behind long running terraform apply/destroy operations. We have ~180 workspaces of two types: those which provision GKE clusters (which take 10-15 minutes to apply/destroy), and those which provision DNS records (which take 10-20 seconds to apply/destroy). On average there are twice as many DNS workspaces as GKE workspaces.
I wrote a tool to parse the provider debug logs to illustrate this; the results are in this spreadsheet. The results show each individual reconciliation, ordered by the time at which they started/finished. For confidentiality reasons the workspace names have been replaced with hashes (the first 8 bytes of a sha256 hash).
There are multiple examples of the errant behaviour, but to pick one let’s look at row 3469 of the ‘Ordered by Start Time’ sheet:
- We can see that this is a creation of a GKE cluster which took ~10 minutes to complete, finishing at 2024-02-01T11:13:24Z. Note that the
did initfield on this row isunknownbecause there is no logging when a Workspace’s checksum is blank. - The next 3 reconciliations (of up-to-date workspaces) took less than 4 seconds each as expected
- Row 3473 is the first example which appeared to get stuck; it finished ~30 seconds after row 3469
- Row 3474 is another create operation. This appeared to start ~2 minutes after the first, but then took ~20 minutes instead of ~10. Reading GCP logs, it looks like that cluster wasn’t actually created until 2024-02-01T11:14:20Z, which suggests that reconciliation didn’t actually start until the first creation completed
- The next 25+ rows are other reconciliations which started after the first 4, and all completed at around 2024-02-01T11:14:06Z (42 seconds after the first reconciliation completed). This appears to be these reconciliations getting stuck behind the first.
What this looks like in the logs is:
# The first creation
2024-02-01T11:04:33.467Z DEBUG provider-terraform Reconciling {"controller": "managed/workspace.tf.upbound.io", "request": {"name":"redacted-workspace-1"}}
# Other reconciliations start
2024-02-01T11:04:50.169Z DEBUG provider-terraform Reconciling {"controller": "managed/workspace.tf.upbound.io", "request": {"name":"redacted-workspace-2"}}
2024-02-01T11:04:50.174Z DEBUG provider-terraform Checksums match - skip running terraform init {"request": "redacted-workspace-2"}
2024-02-01T11:05:09.230Z DEBUG provider-terraform Reconciling {"controller": "managed/workspace.tf.upbound.io", "request": {"name":"redacted-workspace-3"}}
2024-02-01T11:05:09.234Z DEBUG provider-terraform Checksums match - skip running terraform init {"request": "redacted-workspace-3"}
# Repeated for many other workspaces
# The first creation finishes and the workspace is immediately requeued
2024-02-01T11:13:24.291Z DEBUG provider-terraform Successfully requested creation of external resource {"controller": "managed/workspace.tf.upbound.io", "request": {"name":"redacted-workspace-1"}, "uid": "0254a9c2-8a4f-4209-aba0-60937789a667", "version": "947058634", "external-name": "", "external-name": "redacted-workspace-1"}
2024-02-01T11:13:24.305Z DEBUG provider-terraform Reconciling {"controller": "managed/workspace.tf.upbound.io", "request": {"name":"redacted-workspace-1"}}
# Other reconciliations finish (not in the same order that they started)
2024-02-01T11:13:55.363Z DEBUG provider-terraform External resource is up to date {"controller": "managed/workspace.tf.upbound.io", "request": {"name":"redacted-workspace-4"}, "uid": "2b811eb9-b837-4e42-8515-5b7a8656eecd", "version": "946986250", "external-name": "redacted-workspace-4", "requeue-after": "2024-02-01T11:44:55.013Z"}
2024-02-01T11:14:04.429Z DEBUG provider-terraform External resource is up to date {"controller": "managed/workspace.tf.upbound.io", "request": {"name":"redacted-workspace-5"}, "uid": "1a9cfddf-9228-4a31-a0a4-6e69902b9b1d", "version": "946873818", "external-name": "redacted-workspace-5", "requeue-after": "2024-02-01T11:55:20.401Z"}
# Repeated for many other workspaces
This behaviour can be seen in the metrics - each time we have a GKE cluster which needs creating/deleting, the number of active workers starts to grow and the reconcile rate drops to 0. Once it finishes, we see a big spike in the reconcile rate as all the backup up reconciles finish at once.
Hypotheses and debugging
- My first instinct was to look at the lock which guards the TF plugin cache, but reading the code the only thing which does a write lock is a call to Init(), which happens rarely (except for after the provider restarts, which invalidates all of the checksums). Every other terraform operation uses
RLock, which should allow for multiple concurrent operations - I’m currently working on reproducing this with a smaller number of resources, with a doctored provider image which wraps the terraform binary to log all invocations
What environment did it happen in?
- Crossplane Version: 0.14.2
- Provider Version: 0.12.0
- Kubernetes Version: v1.24.17-gke.2266000
- Kubernetes Distribution: GKE
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 32 (12 by maintainers)
Well it’s pretty reliably doing that, but yes - the current implementation of remote repos seems to limit the concurrency to 1.
I wonder if we could use TF_DATA_DIR (https://developer.hashicorp.com/terraform/cli/config/environment-variables#tf_data_dir) to store the
.terraformdirectory somewhere else so that it’s not removed when the download directory is removed, and then presumably if the remote content hasn’t changed the checksum would be the same and it would not be necessary to runterraform init.I don’t know if this is directly related to #230 but it seems like it might be similar. If we move the
.terraformdirectory for all Workspaces to a separate path it might solve both problems.I think it makes sense to do it in #240
The provider does remove the current workspace and recreate it every time for remote repos: https://github.com/upbound/provider-terraform/blob/main/internal/controller/workspace/workspace.go#L233
It is using go-getter which has issues when retrieving a git repo into an existing directory (https://github.com/hashicorp/go-getter/issues/114), so the workaround is to remove the directory and re-pull the repo. Definitely not very efficient.
It might be better to use go-git to clone the remote repo and then use
fetchto determine if the repo has been updated, and only runterraform initwhenfetchreturns new commits.You’re right - my mistake. Apologies @ytsarev
I think @ytsarev was referring to the
-lock=falseflag?Should I remove the locking for plans in #240, or is that best done separately?
A better overall solution may be to replace the in-memory storage of terraform workspaces with a persistent volume that can be sized as needed. That would allow sufficient storage space to run with caching disabled without using excessive memory, and it also allows for pod updates to not have to re-run terraform init on all Workspaces since the workspaces are persistent. I’m pretty sure we could do this today using a
DeploymentRuntimeConfigmounted on/tfJust thinking out loud - do we need to be RLock’ing for
terraform plan? If it fails due to some race condition we’ll just retry it on the next reconciliation anyway.initandapplydefinitinitely need to Lock/RLock, and I’m not sure aboutdelete. But it seems like we could mitigate much of the problem if we didn’t lock onplanand just allow the reconciliation to try again if it fails due to some race condition. @ytsarev @negz ?