infracost: HCL parsing does not work with modules that have a `source` in a private Terraform registry

HCL parsing does not work if the Terraform code contains a module with a source in a private Terraform registry. Private git modules work ok.

Example this code:

provider "aws" {
  region                      = "us-east-1"
  skip_credentials_validation = true
  skip_requesting_account_id  = true
  access_key                  = "mock_access_key"
  secret_key                  = "mock_secret_key"
}

module "ec2_cluster" {
  source  = "app.terraform.io/infracost/ec2-instance/aws"
  version = "~> 2.0"

  name           = "my-cluster"
  instance_count = 5

  ami                    = "ami-ebd02392"
  instance_type          = "t2.micro"
  key_name               = "user1"
  monitoring             = true
  vpc_security_group_ids = ["sg-12345678"]
  subnet_id              = "subnet-eddcdzz4"
}

Shows this error:

Error: Error loading Terraform modules: error downloading 'file:///examples/terraform/.infracost/terraform_modules/ec2_cluster/app.terraform.io/infracost/ec2-instance/aws': source path error: stat /examples/terraform/.infracost/terraform_modules/ec2_cluster/app.terraform.io/infracost/ec2-instance/aws: no such file or directory

There’s two issues which are causing this:

  1. Our code assumes that the module API is always at <host>/v1/modules but this can be different for different registries. We should use the terraform-svchost package to discover the correct service URLs
  2. Our code does not pass through any credentials when pulling remote registry modules.

I’ve created a work in progress branch here.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 3
  • Comments: 17 (8 by maintainers)

Most upvoted comments

@aliscott should we re-open this issue until gitlab support is confirmed as working by @EppO or should we create a new issue to track it? (@EppO I really appreciate your help with that)

Thanks @EppO - I’ve added a fix for the relative download paths in https://github.com/infracost/infracost/pull/1795.

go-getter supports redirects via the X-Terraform-Get header, can’t you just use go-getter with the download module URL from the get-go?

Yeah you’re right, but this requires the credentials to be passed through to go-getter which we aren’t currently doing, so I went with a different approach.

in https://www.terraform.io/registry/api-docs#download-source-code-for-a-specific-module-version, docs say:

The value of X-Terraform-Get may instead be a relative URL, indicated by beginning with /, ./ or …/, in which case it is resolved relative to the full URL of the download endpoint.

so I assume infracost needs to support both, relative and absolute.

@EppO thanks for all this manual debugging, yup it looks like that might be the issue right there. Normally x-terraform-get returns a fully qualified URL which we use to download the tar. I’ll put a fix in for this tomorrow (it’s late here in the UK) and ping you to test.

no rush, let me know when I can try again.

@EppO thanks! Just to double check, you’re using a private TF module hosted on GitLab Terraform Registry?

That’s indeed what I’m using

I’ve not used that before, does GitLab support public modules too? Or would those just be public git repos?

I guess that depends if your repo is public or private. I would assume it supports both. I will give it a try if you need.