terrakube: Workspace Importer is not importing state

Bug description šŸž

I’ve just noticed that whenever I use the Workspace Importer to import a workspace from Terraform Cloud to Terrakube whilst the actual operation gets reported as being successful, when I actually go to look at the workspace in Terrakube there is no state! Looking at the API log I can see:

INFO 1 --- [nio-8080-exec-8] o.t.a.p.i.t.services.WorkspaceService    : VCS ID:
ERROR 1 --- [nio-8080-exec-8] o.t.a.p.i.t.services.WorkspaceService    : URI is not absolute

But unfortunately no more information from the log.

The (sanitised) payload being sent to the workspaces/ endpoint in this case was:

{
    "organizationId": "xxxxxxxx-8372-4fec-8500-xxxxxxxxxxxx",
    "vcsId": "",
    "id": "ws-49WuV6tyxxxxxxxx",
    "organization": "xxxxxx",
    "name": "xxxxxx-xxxxxxx-xxxxxxxxxx",
    "terraformVersion": "1.5.7",
    "description": null
}

I’ve tried this for a few of our workspaces and unfortunately I seem to get the same issue each time.

Steps to reproduce

  1. Import a workspace from Terraform Cloud using the Workspace Importer
  2. Check the imported workspace and look at the empty state

Expected behavior

State should be imported.

Example repository

No response

Anything else?

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 21 (9 by maintainers)

Most upvoted comments

When you do a terraform pull, internally the terraform CLI is using the following API method to download the RAW state file.

When you do a terraform state push, internally the terraform CLI is using the following API method, depending of the terraform CLI version it could also use this API method (if I remember correctly this one is use in terraform above 1.6.X) to upload the terraform state and the json state in two different request.

So I guess that when you run terraform state push you are using a terraform version above 1.3 that is including the ā€œjson-stateā€ that is generated internally by terraform CLI.

If you tried to do a terraform state push with an earlier terraform version below 1.3 you won’t see the json state representation in the UI.

Finally if you will be using the "local execution mode" when the terraform CLI run the apply and do the request to create state using the method it wont include the property ā€œjson-stateā€ when using terraform version bellow 1.3.

So as an alternative for now you could try to migrate at least to terraform 1.3 so when you use the terraform cli in local execution mode and run an apply it will create the ā€œjson stateā€

By the way we are having another discussion of using the ā€œjson state in the UIā€ here, maybe in the near future we will have to adjust that part of the UI and use the RAW state there insted of the json representation

Excellent. Thanks for the swift response fixing this @jcanizalez šŸ‘

I have added this to 2.20.0-beta.4 if you want to test it. However, it has a known issue, #733, due to which you won’t be able to see the Modules in the Registry section. You can either try this version or wait for the next beta release, in which we will attempt to fix this issue.

Thanks @dcopestake This information has been invaluable; finally, I succeeded in recreating the issue by using a workspace with Terraform versions earlier than 1.3. I have updated the importer to import the state, allowing its use even if the JSON state is not available. However, it will display a warning like this:

image

Hi @dcopestake I’ve tested using MinIO and confirmed that it’s working fine. As a result, I have released version 2.20.0-beta.3. This version introduces support for importing tags and also enhances logging to better debug your issue. However, it also includes a known issue with the Registry for Redis, preventing the visibility of Modules in the UI. If you’re interested in trying out the importer, you can either upgrade to this version or wait for the next beta release, where we aim to resolve the Redis issue. Below is an example of a successful log entry that you can use for comparison in your logs.

UI image

API

2024-02-21 18:11:36.769  INFO 41688 --- [nio-8080-exec-3] o.t.a.p.i.t.services.WorkspaceService    : Importing Workspace: terraform-sample-repository
2024-02-21 18:11:36.786  INFO 41688 --- [nio-8080-exec-3] o.t.a.p.i.t.services.WorkspaceService    : Workspace created: 2cc91fa4-6553-4fae-bc4b-6298410d1502
2024-02-21 18:11:37.220  INFO 41688 --- [nio-8080-exec-3] o.t.a.p.i.t.services.WorkspaceService    : Variables imported: 1
2024-02-21 18:11:37.415  INFO 41688 --- [nio-8080-exec-3] o.t.a.p.i.t.services.WorkspaceService    : Tags imported: 0
2024-02-21 18:11:37.616  INFO 41688 --- [nio-8080-exec-3] o.t.a.p.i.t.services.WorkspaceService    : State download URL: https://app.terraform.io/api/state-versions/sv-xxxxxx/hosted_state
2024-02-21 18:11:37.616  INFO 41688 --- [nio-8080-exec-3] o.t.a.p.i.t.services.WorkspaceService    : State download JSON URL: https://app.terraform.io/api/state-versions/sv-xxxxxxx/hosted_json_state
2024-02-21 18:11:37.638  INFO 41688 --- [nio-8080-exec-3] o.t.a.p.i.t.services.WorkspaceService    : History created: af3abc96-82fb-4b34-bf6b-1a0c577ce36d
2024-02-21 18:11:38.566  INFO 41688 --- [nio-8080-exec-3] o.t.a.p.i.t.services.WorkspaceService    : State downloaded: 1258
2024-02-21 18:11:38.566  INFO 41688 --- [nio-8080-exec-3] o.t.a.p.i.t.services.WorkspaceService    : State JSON downloaded: 768
2024-02-21 18:11:38.567  INFO 41688 --- [nio-8080-exec-3] o.t.a.p.s.aws.AwsStorageTypeServiceImpl  : terraformJsonStateFile: tfstate/f5365c9e-bc11-4781-b649-45a281ccdd4a/2cc91fa4-6553-4fae-bc4b-6298410d1502/state/af3abc96-82fb-4b34-bf6b-1a0c577ce36d.json
2024-02-21 18:11:38.865  INFO 41688 --- [nio-8080-exec-3] o.t.a.p.s.aws.AwsStorageTypeServiceImpl  : terraformStateFile: tfstate/f5365c9e-bc11-4781-b649-45a281ccdd4a/2cc91fa4-6553-4fae-bc4b-6298410d1502/terraform.tfstate

After Importing I can even execute a Plan and the plan is running using the imported state image

@alfespa17 if you get some time please try out and let us know any findings

Hello @dcopestake, I’m currently developing and testing the importer from Gitpod using local storage, and it’s working well. Based on the feedback provided by you and @iam-take, I suspect the issue might be related to use the importer with a cloud storage. I need to create an Azure storage account or use S3 to test it in that environment. I will try to configure environment this week to debug

When you open gitpod you can use the following command in the terminal.

./scripts/setupDevelopmentEnvironmentMinio.sh

It will do all the setup so you can use MINIO as backend storage inside Gitpod, that is basically the same code that when you are using a S3 in AWS. I think that can help you to test

Hello @dcopestake, I’m currently developing and testing the importer from Gitpod using local storage, and it’s working well. Based on the feedback provided by you and @iam-take, I suspect the issue might be related to use the importer with a cloud storage. I need to create an Azure storage account or use S3 to test it in that environment. I will try to configure environment this week to debug

We experienced the same issue when trying to migrate state into Terrakube. We had used the Azure Storage Account as backend storage for Terrakube. When looking into the API log we see the same kind of logging as @dcopestake. One difference is that I did see the state file in the Azure Storage Account. It just shows as empty in Terrakube.

It seems not to matter where your orginal state is located the import goes wrong.