terraform-provider-snowflake: Crash when MFA used on 2 providers

Terraform CLI and Provider Versions

Terraform v1.7.5
on linux_amd64
+ provider registry.terraform.io/hashicorp/null v3.2.2
+ provider registry.terraform.io/snowflake-labs/snowflake v0.87.3-pre

Terraform Configuration

terraform {
  required_providers {
    snowflake = {
      source  = "Snowflake-Labs/snowflake"
      version = "0.87.3-pre"
    }
  }
}

provider "snowflake" {
  alias         = "security"
  account       = "account"
  user          = var.snowflake_username
  password      = var.snowflake_password
  role          = "SECURITYADMIN"
  authenticator = "UsernamePasswordMFA"

  client_request_mfa_token = true
}


provider "snowflake" {
  alias         = "system"
  account       = "account"
  user          = var.snowflake_username
  password      = var.snowflake_password
  role          = "SYSADMIN"
  authenticator = "UsernamePasswordMFA"

  client_request_mfa_token = true
}

Also requires a whole application’s definition. You need a non-trivial number of resources to trigger this race condition. The key is having both client_request_mfa_token = true

Expected Behavior

Provider should cache MFA response to prevent hundreds of Duo messages. Each provider should independently store the provider.

Actual Behavior

Plan operation crashes, stack trace attached. I assume this is providers in child modules not namespacing credentials properly, or some kind of operation that needs a Mutex guard.

Steps to Reproduce

  1. terraform plan

How much impact is this issue causing?

Medium

Logs

https://gist.github.com/Oracen/2b818c2d25dabe7bffbd366176b39fea

Additional Information

Workaround for now is to use accounts without 2FA for deployment of our core Snowflake architecture. This will work short term but is obviously undesirable.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (4 by maintainers)

Commits related to this issue

Most upvoted comments

Hey @Oracen @sfc-gh-dszmolka. We have bumped the driver to v1.10.0, we should do the release tomorrow.

Great to hear that! I am closing the ticket then. Thanks for the help @sfc-gh-dszmolka.

Yes, it was. Worked like a charm. Appreciate it!

Will do so monday

Hey @Oracen, we released v0.91.0 yesterday. Please upgrade to the newest version and let us know if this fixes your issues.

fix in gosnowflake released with 1.10.0

I’ll confirm that splitting into separate TF configs with single providers does work, but may require modifications to your RBAC strategy. If you have a nice divide between resources + policies it works well, with only the additional CICD time + mental overhead as a cost.

If you’re blocked, probably worth doing, but the easier way is to be nice to your DBAs and get them to set MINS_TO_BYPASS_MFA= X for a suitable value of X

Thanks, @sfc-gh-dszmolka, for the update and for the fast fix. I will schedule a bump of the version on the provider side.

short update regarding the timeline and progress: gosnowflake fix merged, and the new version of the driver is expected to be released towards end of April 2024

afterwards, the TF provider also needs to bump the gosnowflake dependency + a new version released which carries the fixed gosnowflake

A possible workaround would be also to split the current configuration into two separate workspaces so that only a single provider is used in the given run. It may require some work, though.

hi folks - so the issue is handled over there in gosnowflake so please head to the linked issue for progress tracking; I’ll update that one with the progress.

For any viable workaround, I’m not sure if we have any, if your operations on the (tons of) resources cannot complete due to the race condition on the temporal credential cache file. Until the underlying issue is fixed, could implementing a (little) sleep every now and then between resources help maybe, to reduce the contention on the same credential cache?