terratest: Terraform Apply hang unresponsive

Terraform Project Structure:

-- terraform-root
---- main.tf
---- variables.tf
---- outputs.tf
---- test/
-------- basic_test.go
---- examples/
-------- elasticsearch-auth-iam-user/main.tf

my basic_test.go looks like the following

package test

import (
    "fmt"
    "testing"
    "github.com/gruntwork-io/terratest/modules/terraform"
)

func TestTerraformBasicExample(t *testing.T) {

	terraformOptions := &terraform.Options{
		TerraformDir: "../ examples/elasticsearch-auth-iam-user",

		Vars: map[string]interface{}{
                        "domain_name": "test",
            	        "stack": "elasticsearch",
                        "workspace": "stg",
		},
	}

	defer terraform.Destroy(t, terraformOptions)
	terraform.InitAndApply(t, terraformOptions)
}

cd into test run “go test -timeout 5m”.

I get “terraform init” succeeded but “terraform apply” hanged without any outputs until timing out.

Log looks like

[ec2-user@ip-10-104-88-81 test]$ go test -timeout 5m
init...................TestTerraformBasicExample 2018-08-20T10:37:20Z retry.go:69: Running terraform [init -upgrade=false]
TestTerraformBasicExample 2018-08-20T10:37:20Z command.go:52: Running command terraform with args [init -upgrade=false]
TestTerraformBasicExample 2018-08-20T10:37:20Z command.go:96: Initializing modules...
TestTerraformBasicExample 2018-08-20T10:37:20Z command.go:96: - module.elasticsearch_security_group
TestTerraformBasicExample 2018-08-20T10:37:20Z command.go:96:   Getting source "git::ssh://git@ssh.git.domain/iot-infrastructure/tf-module-aws-security-group.git?ref=master"
TestTerraformBasicExample 2018-08-20T10:37:21Z command.go:96:
TestTerraformBasicExample 2018-08-20T10:37:21Z command.go:96: Initializing provider plugins...
TestTerraformBasicExample 2018-08-20T10:37:21Z command.go:96: - Checking for available provider plugins on https://releases.hashicorp.com...
TestTerraformBasicExample 2018-08-20T10:37:21Z command.go:96: - Downloading plugin for provider "aws" (1.32.0)...
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:96:
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:96: The following providers do not have any version constraints in configuration,
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:96: so the latest version was installed.
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:96:
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:96: To prevent automatic upgrades to new major versions that may contain breaking
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:96: changes, it is recommended to add version = "..." constraints to the
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:96: corresponding provider blocks in configuration, with the constraint strings
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:96: suggested below.
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:96:
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:96: * provider.aws: version = "~> 1.32"
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:96:
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:96: Terraform has been successfully initialized!
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:96:
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:96: You may now begin working with Terraform. Try running "terraform plan" to see
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:96: any changes that are required for your infrastructure. All Terraform commands
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:96: should now work.
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:96:
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:96: If you ever set or change modules or backend configuration for Terraform,
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:96: rerun this command to reinitialize your working directory. If you forget, other
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:96: commands will detect it and remind you to do so if necessary.
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:20 [INFO] Terraform version: 0.11.7  41e50bd32a8825a84535e353c3674af8ce799161
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:20 [INFO] Go runtime version: go1.10.1
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:20 [INFO] CLI args: []string{"/usr/local/bin/terraform", "init", "-upgrade=false"}
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:20 [DEBUG] Attempting to open CLI config file: /home/ec2-user/.terraformrc
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:20 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:20 [INFO] CLI command args: []string{"init", "-upgrade=false"}
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:20 [DEBUG] command: loading backend config file: /home/ec2-user/go/src/tf-module-aws-elasticsearch
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:20 [ERR] Checkpoint error: Get https://checkpoint-api.hashicorp.com/v1/check/terraform?arch=386&os=linux&signature=1876a657-05db-8573-ee03-ee17a78287f6&version=0.11.7: Forbidden
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:21 [DEBUG] found "git::ssh://git@ssh.git.tech.rz.db.de/iot-infrastructure/tf-module-aws-security-group.git?ref=master" in ".terraform/modules/30207133dde280c8832b06615f77d133": true
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:21 [TRACE] "git::ssh://git@ssh.git.tech.rz.db.de/iot-infrastructure/tf-module-aws-security-group.git?ref=master" stored in ".terraform/modules/30207133dde280c8832b06615f77d133"
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:21 [INFO] command: empty terraform config, returning nil
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:21 [DEBUG] command: no data state file found for backend config
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:21 [DEBUG] New state was assigned lineage "33329760-d197-8ad8-7d93-841dc70d9b45"
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:21 [INFO] command: backend initialized: <nil>
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:21 [DEBUG] checking for provider in "."
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:21 [DEBUG] checking for provider in "/usr/local/bin"
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:21 [DEBUG] checking for provisioner in "."
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:21 [DEBUG] checking for provisioner in "/usr/local/bin"
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:21 [INFO] Failed to read plugin lock file .terraform/plugins/linux_386/lock.json: open .terraform/plugins/linux_386/lock.json: no such file or directory
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:21 [INFO] command: backend <nil> is not enhanced, wrapping in local
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:21 [DEBUG] checking for provider in "."
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:21 [DEBUG] checking for provider in "/usr/local/bin"
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:21 [DEBUG] plugin requirements: "aws"=""
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:21 [DEBUG] fetching provider info for aws version 1.32.0
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:21 [DEBUG] getting provider "aws" version "1.32.0"
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:21 [DEBUG] plugin cache is disabled, so downloading aws 1.32.0 from https://releases.hashicorp.com/terraform-provider-aws/1.32.0/terraform-provider-aws_1.32.0_linux_386.zip?checksum=sha256:b9b35d1040c7d764e7dc0a1083826f8d9b63f49f46a53ee521c6c6c138130d64
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:22 [DEBUG] looking for the aws 1.32.0 plugin we just installed
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:22 [DEBUG] checking for provider in ".terraform/plugins/linux_386"
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:22 [DEBUG] found provider "terraform-provider-aws_v1.32.0_x4"
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:22 [DEBUG] all plugins found discovery.PluginMetaSet{discovery.PluginMeta{Name:"aws", Version:"1.32.0", Path:"/home/ec2-user/go/src/tf-module-aws-elasticsearch/.terraform/plugins/linux_386/terraform-provider-aws_v1.32.0_x4"}:struct {}{}}
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:22 [DEBUG] filtered plugins discovery.PluginMetaSet{discovery.PluginMeta{Name:"aws", Version:"1.32.0", Path:"/home/ec2-user/go/src/tf-module-aws-elasticsearch/.terraform/plugins/linux_386/terraform-provider-aws_v1.32.0_x4"}:struct {}{}}
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:22 [DEBUG] checking for provider in "."
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:22 [DEBUG] checking for provider in "/usr/local/bin"
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:22 [DEBUG] checking for provider in ".terraform/plugins/linux_386"
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:22 [DEBUG] found provider "terraform-provider-aws_v1.32.0_x4"
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:22 [DEBUG] found valid plugin: "aws", "1.32.0", "/home/ec2-user/go/src/tf-module-aws-elasticsearch/.terraform/plugins/linux_386/terraform-provider-aws_v1.32.0_x4"
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:23 [DEBUG] checking for provider in ".terraform/plugins/linux_386"
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:23 [DEBUG] found provider "terraform-provider-aws_v1.32.0_x4"
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:100: 2018/08/20 10:37:23 [DEBUG] plugin: waiting for all plugin processes to complete...
apply...................TestTerraformBasicExample 2018-08-20T10:37:23Z retry.go:69: Running terraform [apply -input=false -lock=false -auto-approve -var access_policies="a" -var domain_name="test" -var stack="elasticsearch" -var workspace="stg"]
TestTerraformBasicExample 2018-08-20T10:37:23Z command.go:52: Running command terraform with args [apply -input=false -lock=false -auto-approve -var access_policies="a" -var domain_name="test" -var stack="elasticsearch" -var workspace="stg"]

main.tf looks like

# If comment out this module usage, terratest works
/*
module "elasticsearch_security_group" {
  source = "git::ssh://git@ssh.git.domain/iot-infrastructure/tf-module-aws-security-group.git?ref=master"
  name = "elasticsearch_security_group"
  vpc_id = "${var.vpc_id}"
  ingress_cidr_blocks = ["0.0.0.0/0"]
}
*/
resource "aws_elasticsearch_domain" "es_test" {
  domain_name           = "test"

  cluster_config {
    instance_type = "r3.large.elasticsearch"
    instance_count = "1"
    dedicated_master_enabled = "false"
    zone_awareness_enabled = "false"
  }
}

# meta parameters for module

variable "vpc_id" {
  type = "string"
  default = "vpc-42f2792a"
}

# Provider
provider "aws" {
  region = "eu-central-1"
}

If I comment module “elasticsearch_security_group” out. It works. Can it possible terratest has problem with module ?

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 32 (12 by maintainers)

Most upvoted comments

Tests fail for me with TF_LOG set because it modifies stdout and some of the code is reading it, but the tests don’t hang…