terraform-provider-azurerm: terraform plan hangs forever after trying to upgrade provider to 1.37.0

The delay is caused by this call https://github.com/terraform-providers/terraform-provider-azurerm/blob/6ed5554bad427aae8872c6ead28b53844b209aa5/azurerm/resource_arm_storage_account.go#L1164.

This eventually calls the code from github.com/Azure/go-autorest/autorest v0.9.2, which was upgraded from 0.9.0 in 1.37.0 release:

$ git diff v1.36.1 v1.37.0 go.mod
diff --git a/go.mod b/go.mod
index 05c0f84a5..97df6065b 100644
--- a/go.mod
+++ b/go.mod
@@ -1,13 +1,13 @@
 module github.com/terraform-providers/terraform-provider-azurerm

 require (
-       github.com/Azure/azure-sdk-for-go v34.1.0+incompatible
-       github.com/Azure/go-autorest/autorest v0.9.0
+       github.com/Azure/azure-sdk-for-go v36.2.0+incompatible
+       github.com/Azure/go-autorest/autorest v0.9.2

the delay is introduced in https://github.com/Azure/go-autorest/blob/master/autorest/sender.go

$ git diff autorest/v0.9.0 autorest/v0.9.2 sender.go
diff --git a/autorest/sender.go b/autorest/sender.go
index e582489..5e595d7 100644
--- a/autorest/sender.go
+++ b/autorest/sender.go
@@ -289,10 +289,6 @@ func doRetryForStatusCodesImpl(s Sender, r *http.Request, count429 bool, attempt
                        return
                }
                resp, err = s.Do(rr.Request())
-               // if the error isn't temporary don't bother retrying
-               if err != nil && !IsTemporaryNetworkError(err) {
-                       return
-               }
                // we want to retry if err is not nil (e.g. transient network failure).  note that for failed authentication
                // resp and err will both have a value, so in this case we don't want to retry as it will never succeed.
                if err == nil && !ResponseHasStatusCode(resp, codes...) || IsTokenRefreshError(err) {

The debug output here is:

2019-11-28T14:50:12.538+0100 [DEBUG] plugin.terraform-provider-azurerm_v1.36.2: DDDDD ERR Get https://<storage_account_name>.queue.core.windows.net/?comp=properties&restype=service: dial tcp: lookup <storage_account_name>.queue.core.windows.net: no such host

This will never be a successful call for storage accounts that do not have a queue.

This has been released in version 1.41.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 1.41.0"
}
# ... other configuration ...

If I understand this issue correctly, it’s supposed to be resolved by https://github.com/terraform-providers/terraform-provider-azurerm/pull/5316. Just as a note for everyone looking into this.

@tombuildsstuff could you please assess this one, as it kills all plans with storage accounts that do not have queues.

👋

From what I can tell this should be fixed in #5316 - since that’s been merged I’m going to close this issue for the moment; but if you’re still seeing this after upgrading to v1.41 (once that’s released) please let us know and we’ll take another look 😃

Thanks!

I seem to be having this issue (terraform just hangs when refreshing it’s state) with the 1.41 version of the azurerm provider using terraform version 0.12.19

Any ideas, as I’m pretty much blocked applying updates to our clusters?

I’m using remote state (azure storage account) and I can see terraform acquiring a state lock, refreshing all the resources and then it just hangs there…

I see the same behavior on storage_account resource, it actually manages to come out of this after some 20 minutes.

` $ TF_LOG=TRACE terraform plan --target module.tfe.module.data.azurerm_storage_account.this … 2019-11-27T09:02:29.498+0100 [DEBUG] plugin.terraform-provider-azurerm_v1.37.0_x4: POST /subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Storage/storageAccounts/XXX/listKeys?%24expand=kerb&api-version=2019-04-01 HTTP/1.1 …

2019-11-27T09:02:29.648+0100 [DEBUG] plugin.terraform-provider-azurerm_v1.37.0_x4: [DEBUG] AzureRM Response for https://management.azure.com/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Storage/storageAccounts/XXX/listKeys?%24expand=kerb&api-version=2019-04-01: 2019-11-27T09:02:29.648+0100 [DEBUG] plugin.terraform-provider-azurerm_v1.37.0_x4: HTTP/2.0 200 OK 2019-11-27T09:02:29.648+0100 [DEBUG] plugin.terraform-provider-azurerm_v1.37.0_x4: {“keys”:[{“keyName”:“key1”,“value”:“XXX”,“permissions”:“FULL”},{“keyName”:“key2”,“value”:“XXX”,“permissions”:“FULL”}]}

… 2019/11/27 09:23:57 [TRACE] dag/walk: vertex “provider.azurerm.fxi_production1_services (close)” is waiting for “module.tfe.module.data.azurerm_storage_account.this” 2019/11/27 09:23:59 [TRACE] dag/walk: vertex “root” is waiting for “provider.azurerm.fxi_production1_services (close)” 2019/11/27 09:24:02 [TRACE] dag/walk: vertex “provider.azurerm.fxi_production1_services (close)” is waiting for “module.tfe.module.data.azurerm_storage_account.this” 2019/11/27 09:24:04 [TRACE] dag/walk: vertex “root” is waiting for “provider.azurerm.fxi_production1_services (close)” 2019/11/27 09:24:04 [TRACE] module.tfe.module.data: eval: *terraform.EvalWriteState 2019/11/27 09:24:04 [TRACE] EvalWriteState: recording 1 dependencies for module.tfe.module.data.azurerm_storage_account.this 2019/11/27 09:24:04 [TRACE] EvalWriteState: writing current state object for module.tfe.module.data.azurerm_storage_account.this 2019/11/27 09:24:04 [TRACE] [walkRefresh] Exiting eval tree: module.tfe.module.data.azurerm_storage_account.this 2019/11/27 09:24:04 [TRACE] vertex “module.tfe.module.data.azurerm_storage_account.this”: visit complete 2019/11/27 09:24:04 [TRACE] vertex “module.tfe.module.data.azurerm_storage_account.this”: dynamic subgraph completed successfully 2019/11/27 09:24:04 [TRACE] vertex “module.tfe.module.data.azurerm_storage_account.this”: visit complete `