provider-terraform: Reconciliations blocking unexpectedly

What happened?

Fairly regularly, we’re seeing reconciliations back up behind long running terraform apply/destroy operations. We have ~180 workspaces of two types: those which provision GKE clusters (which take 10-15 minutes to apply/destroy), and those which provision DNS records (which take 10-20 seconds to apply/destroy). On average there are twice as many DNS workspaces as GKE workspaces.

I wrote a tool to parse the provider debug logs to illustrate this; the results are in this spreadsheet. The results show each individual reconciliation, ordered by the time at which they started/finished. For confidentiality reasons the workspace names have been replaced with hashes (the first 8 bytes of a sha256 hash).

There are multiple examples of the errant behaviour, but to pick one let’s look at row 3469 of the ‘Ordered by Start Time’ sheet:

  • We can see that this is a creation of a GKE cluster which took ~10 minutes to complete, finishing at 2024-02-01T11:13:24Z. Note that the did init field on this row is unknown because there is no logging when a Workspace’s checksum is blank.
  • The next 3 reconciliations (of up-to-date workspaces) took less than 4 seconds each as expected
  • Row 3473 is the first example which appeared to get stuck; it finished ~30 seconds after row 3469
  • Row 3474 is another create operation. This appeared to start ~2 minutes after the first, but then took ~20 minutes instead of ~10. Reading GCP logs, it looks like that cluster wasn’t actually created until 2024-02-01T11:14:20Z, which suggests that reconciliation didn’t actually start until the first creation completed
  • The next 25+ rows are other reconciliations which started after the first 4, and all completed at around 2024-02-01T11:14:06Z (42 seconds after the first reconciliation completed). This appears to be these reconciliations getting stuck behind the first.

What this looks like in the logs is:

# The first creation
2024-02-01T11:04:33.467Z	DEBUG	provider-terraform	Reconciling	{"controller": "managed/workspace.tf.upbound.io", "request": {"name":"redacted-workspace-1"}}

# Other reconciliations start
2024-02-01T11:04:50.169Z	DEBUG	provider-terraform	Reconciling	{"controller": "managed/workspace.tf.upbound.io", "request": {"name":"redacted-workspace-2"}}
2024-02-01T11:04:50.174Z	DEBUG	provider-terraform	Checksums match - skip running terraform init	{"request": "redacted-workspace-2"}
2024-02-01T11:05:09.230Z	DEBUG	provider-terraform	Reconciling	{"controller": "managed/workspace.tf.upbound.io", "request": {"name":"redacted-workspace-3"}}
2024-02-01T11:05:09.234Z	DEBUG	provider-terraform	Checksums match - skip running terraform init	{"request": "redacted-workspace-3"}
# Repeated for many other workspaces

# The first creation finishes and the workspace is immediately requeued
2024-02-01T11:13:24.291Z	DEBUG	provider-terraform	Successfully requested creation of external resource	{"controller": "managed/workspace.tf.upbound.io", "request": {"name":"redacted-workspace-1"}, "uid": "0254a9c2-8a4f-4209-aba0-60937789a667", "version": "947058634", "external-name": "", "external-name": "redacted-workspace-1"}
2024-02-01T11:13:24.305Z	DEBUG	provider-terraform	Reconciling	{"controller": "managed/workspace.tf.upbound.io", "request": {"name":"redacted-workspace-1"}}

# Other reconciliations finish (not in the same order that they started)
2024-02-01T11:13:55.363Z	DEBUG	provider-terraform	External resource is up to date	{"controller": "managed/workspace.tf.upbound.io", "request": {"name":"redacted-workspace-4"}, "uid": "2b811eb9-b837-4e42-8515-5b7a8656eecd", "version": "946986250", "external-name": "redacted-workspace-4", "requeue-after": "2024-02-01T11:44:55.013Z"}
2024-02-01T11:14:04.429Z	DEBUG	provider-terraform	External resource is up to date	{"controller": "managed/workspace.tf.upbound.io", "request": {"name":"redacted-workspace-5"}, "uid": "1a9cfddf-9228-4a31-a0a4-6e69902b9b1d", "version": "946873818", "external-name": "redacted-workspace-5", "requeue-after": "2024-02-01T11:55:20.401Z"}
# Repeated for many other workspaces

This behaviour can be seen in the metrics - each time we have a GKE cluster which needs creating/deleting, the number of active workers starts to grow and the reconcile rate drops to 0. Once it finishes, we see a big spike in the reconcile rate as all the backup up reconciles finish at once.

provider-terraform-graphs

Hypotheses and debugging

  • My first instinct was to look at the lock which guards the TF plugin cache, but reading the code the only thing which does a write lock is a call to Init(), which happens rarely (except for after the provider restarts, which invalidates all of the checksums). Every other terraform operation uses RLock, which should allow for multiple concurrent operations
  • I’m currently working on reproducing this with a smaller number of resources, with a doctored provider image which wraps the terraform binary to log all invocations

What environment did it happen in?

  • Crossplane Version: 0.14.2
  • Provider Version: 0.12.0
  • Kubernetes Version: v1.24.17-gke.2266000
  • Kubernetes Distribution: GKE

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 32 (12 by maintainers)

Most upvoted comments

If the plugin cache is disabled, then you get lots of traffic re-downloading TF providers on each reconciliation.

Well it’s pretty reliably doing that, but yes - the current implementation of remote repos seems to limit the concurrency to 1.

I wonder if we could use TF_DATA_DIR (https://developer.hashicorp.com/terraform/cli/config/environment-variables#tf_data_dir) to store the .terraform directory somewhere else so that it’s not removed when the download directory is removed, and then presumably if the remote content hasn’t changed the checksum would be the same and it would not be necessary to run terraform init.

I don’t know if this is directly related to #230 but it seems like it might be similar. If we move the .terraform directory for all Workspaces to a separate path it might solve both problems.

Should I remove the locking for plans in https://github.com/upbound/provider-terraform/pull/240, or is that best done separately?

I think it makes sense to do it in #240

The provider does remove the current workspace and recreate it every time for remote repos: https://github.com/upbound/provider-terraform/blob/main/internal/controller/workspace/workspace.go#L233

It is using go-getter which has issues when retrieving a git repo into an existing directory (https://github.com/hashicorp/go-getter/issues/114), so the workaround is to remove the directory and re-pull the repo. Definitely not very efficient.

It might be better to use go-git to clone the remote repo and then use fetch to determine if the repo has been updated, and only run terraform init when fetch returns new commits.

I think @ytsarev was referring to the -lock=false flag?

You’re right - my mistake. Apologies @ytsarev

I think @ytsarev was referring to the -lock=false flag?

Should I remove the locking for plans in #240, or is that best done separately?

A better overall solution may be to replace the in-memory storage of terraform workspaces with a persistent volume that can be sized as needed. That would allow sufficient storage space to run with caching disabled without using excessive memory, and it also allows for pod updates to not have to re-run terraform init on all Workspaces since the workspaces are persistent. I’m pretty sure we could do this today using a DeploymentRuntimeConfig mounted on /tf

Just thinking out loud - do we need to be RLock’ing for terraform plan? If it fails due to some race condition we’ll just retry it on the next reconciliation anyway. init and apply definitinitely need to Lock/RLock, and I’m not sure about delete. But it seems like we could mitigate much of the problem if we didn’t lock on plan and just allow the reconciliation to try again if it fails due to some race condition. @ytsarev @negz ?