opentofu: The provider is slow when creating/destroying resources

Hi opentofu developers,

I am switching my workload from terraform to opentofu. When I use 1.6.0-alpha5 and aliyun cloud provider 1.213.0, the provider is really slow to orchestrate the resources.

I opened the DEBUG mode, and it turns out the following message is streaming out continously. It usually takes ~100 seconds before the start of creating/destroying resources.

Is there anything wrong with the provider or openTofu ? Thanks a lot!

2023-11-27T19:09:42.665+0800 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.opentofu.org/aliyun/alicloud/1.213.0/linux_amd64/terraform-provider-alicloud_v1.213.0 pid=30472
2023-11-27T19:09:42.665+0800 [DEBUG] provider: plugin exited
2023-11-27T19:09:42.665+0800 [DEBUG] created provider logger: level=debug
2023-11-27T19:09:42.665+0800 [INFO]  provider: configuring client automatic mTLS
2023-11-27T19:09:42.675+0800 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.opentofu.org/aliyun/alicloud/1.213.0/linux_amd64/terraform-provider-alicloud_v1.213.0 args=[".terraform/providers/registry.opentofu.org/aliyun/alicloud/1.213.0/linux_amd64/terraform-provider-alicloud_v1.213.0"]
2023-11-27T19:09:42.676+0800 [DEBUG] provider: plugin started: path=.terraform/providers/registry.opentofu.org/aliyun/alicloud/1.213.0/linux_amd64/terraform-provider-alicloud_v1.213.0 pid=30479
2023-11-27T19:09:42.676+0800 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.opentofu.org/aliyun/alicloud/1.213.0/linux_amd64/terraform-provider-alicloud_v1.213.0
2023-11-27T19:09:42.762+0800 [INFO]  provider.terraform-provider-alicloud_v1.213.0: configuring server automatic mTLS: timestamp="2023-11-27T19:09:42.761+0800"
2023-11-27T19:09:42.793+0800 [DEBUG] provider.terraform-provider-alicloud_v1.213.0: plugin address: address=/tmp/plugin3192086639 network=unix timestamp="2023-11-27T19:09:42.793+0800"
2023-11-27T19:09:42.793+0800 [DEBUG] provider: using plugin: version=5
2023-11-27T19:09:42.925+0800 [DEBUG] No provider meta schema returned
2023-11-27T19:09:43.056+0800 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2023-11-27T19:09:43.059+0800 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.opentofu.org/aliyun/alicloud/1.213.0/linux_amd64/terraform-provider-alicloud_v1.213.0 pid=30479
2023-11-27T19:09:43.059+0800 [DEBUG] provider: plugin exited
2023-11-27T19:09:43.059+0800 [DEBUG] created provider logger: level=debug
2023-11-27T19:09:43.060+0800 [INFO]  provider: configuring client automatic mTLS
2023-11-27T19:09:43.068+0800 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.opentofu.org/aliyun/alicloud/1.213.0/linux_amd64/terraform-provider-alicloud_v1.213.0 args=[".terraform/providers/registry.opentofu.org/aliyun/alicloud/1.213.0/linux_amd64/terraform-provider-alicloud_v1.213.0"]
2023-11-27T19:09:43.069+0800 [DEBUG] provider: plugin started: path=.terraform/providers/registry.opentofu.org/aliyun/alicloud/1.213.0/linux_amd64/terraform-provider-alicloud_v1.213.0 pid=30486
2023-11-27T19:09:43.069+0800 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.opentofu.org/aliyun/alicloud/1.213.0/linux_amd64/terraform-provider-alicloud_v1.213.0
2023-11-27T19:09:43.158+0800 [INFO]  provider.terraform-provider-alicloud_v1.213.0: configuring server automatic mTLS: timestamp="2023-11-27T19:09:43.158+0800"
2023-11-27T19:09:43.188+0800 [DEBUG] provider: using plugin: version=5
2023-11-27T19:09:43.189+0800 [DEBUG] provider.terraform-provider-alicloud_v1.213.0: plugin address: address=/tmp/plugin1095469596 network=unix timestamp="2023-11-27T19:09:43.188+0800"

About this issue

Original URL
State: closed
Created 7 months ago
Comments: 17 (6 by maintainers)

Most upvoted comments

This should be fixed @zhenrong-wang. I’ll be closing this issue, but feel free to reopen if you hit any issues please.

Thanks opentofu team! With beta2 version, the problem reported in this issue got resolved.

Creating an HPC stack in Alicloud is as smooth as terraform now.

Fantastic job! We are step closer to switch to openTofu.

@cube2222

zhenrong-wang on Dec 5, 2023

@zhenrong-wang Thank you very much for supporting us with additional details - we appreciate your contribution a lot! We will continue digging into the issue on our side tomorrow, and will keep you posted. Thanks!

kislerdm on Nov 27, 2023

Hi @zhenrong-wang! Thanks a lot for your collaboration! May I kindly ask to confirm if your setup was identical expect for binary your executed, i.e. tofu vs. terraform? In order words, did you ran tofu/terraform apply/destroy commands on the same machine in the same VPC/availability zone/region? Also, would it be possible to share the logs after running the TF_LOG=trace terraform apply command, so we could compare tofu vs. terraform side-by-side? Thanks! For context, we have a couple of guesses and would like to verify them, but it’d take us extra time in order to reproduce the issue because we don’t have experience with the hpcopr “wrapper” you used, neither do we have a lot of experience with ali cloud. Thank you very much for your support! 🙏🏻

Sure, I will run the terraform version and post the log here.

One thing is for sure: only the binary (executable) changed. All other elements kept unchanged.

In terms of the “wrapper”, it is just another way the hpcopr CLI use to run terraform/tofu commands. Nothing different from running terraform/tofu directly.

Please wait for my log with terraform.

Thanks!

terraform-trace-creating-1.203.0-provider.log terraform-trace-destroying-1.203.0-provider.log

@kislerdm Please check this out. For this time with terraform, destroying the stack took 240 seconds, a bit longer; while creating it took 137 seconds.

Here is how the hpcopr run terraform/tofu. I used -parallelism=1000 to guarantee the concurrency.

Screenshot_2023-11-28-01-45-17-761_com.github.android-edit.jpg

zhenrong-wang on Nov 27, 2023

Hi @zhenrong-wang! Thanks a lot for your collaboration!

May I kindly ask to confirm if your setup was identical expect for binary your executed, i.e. tofu vs. terraform? In order words, did you ran tofu/terraform apply/destroy commands on the same machine in the same VPC/availability zone/region?

Also, would it be possible to share the logs after running the TF_LOG=trace terraform apply command, so we could compare tofu vs. terraform side-by-side? Thanks!

For context, we have a couple of guesses and would like to verify them, but it’d take us extra time in order to reproduce the issue because we don’t have experience with the hpcopr “wrapper” you used, neither do we have a lot of experience with ali cloud.

Thank you very much for your support! 🙏🏻

Sure, I will run the terraform version and post the log here.

One thing is for sure: only the binary (executable) changed. All other elements kept unchanged.

In terms of the “wrapper”, it is just another way the hpcopr CLI use to run terraform/tofu commands. Nothing different from running terraform/tofu directly.

Please wait for my log with terraform.

Thanks!

zhenrong-wang on Nov 27, 2023

@zhenrong-wang Hey! Thanks a lot for sharing the details!

Could you please share the logs for the verbosity level TRACE (as it was requested in the first comment) to help us with identification of root cause: TF_LOG=trace tofu apply? Thanks!

Sure, Let me modify the wrapper to generate a new group of logs.

In order to compare, I will keep on using the provider version 1.203.0.

Please wait minutes.

zhenrong-wang on Nov 27, 2023

Hey Zhenrong! Thanks for your prompt reply!

Did I get it right, the provider version used with OpenTofu differs from the one used with terraform? If it’s the case, could you please try to rerun your flow using identical provider versions, e.g. “terraform-provider-alicloud_1.203.0_linux_amd64.zip” in both cases? Thanks!

Hi @kislerdm ,

The reason why I upgraded the provider version from 1.203.0 to the latest one (1.213.0) for openTofu, is because the same scenario (low speed and wait 100 secs to start) occurs in version 1.203.0.

Sure I can rerun the test in minutes. But the situation will probably be the same. Please wait minutes.

zhenrong-wang on Nov 27, 2023