terraform-provider-external: "external" provider returns an invalid json

This issue was originally opened by @Phydeauxman as hashicorp/terraform#17632. It was migrated here as a result of the provider split. The original body of the issue is below.

Terraform Version

Terraform v0.11.3
+ provider.azurerm v1.2.0
+ provider.external v1.0.0

Terraform Configuration Files

variable ilbase_resourceId {
  default = "/subscriptions/<my_subscription>/resourceGroups/my-rg/providers/Microsoft.Web/hostingEnvironments/my-ilbase/capacities/virtualip"
}


data "external" "aseVip" {
  program = ["az", "resource", "show", "--ids", "${var.ilbase_resourceId}"]
}

Debug Output

Crash Output

Expected Behavior

The JSON object would be returned to terraform

Actual Behavior

Terraform config generates error:

data.external.aseVip: data.external.aseVip: command "az" produced invalid JSON: json: cannot unmarshal object into Go value of type string

Steps to Reproduce

Additional Context

If after I provision the App Service Environment I run the command below:

az resource show --ids "/subscriptions/<my_subscription>/resourceGroups/my-rg/providers/Microsoft.Web/hostingEnvironments/my-ilbase/capacities/virtualip"

It returns the JSON object below:

{
      "additionalProperties": {
        "internalIpAddress": "10.10.1.11",
        "outboundIpAddresses": [
          "52.224.70.119"
        ],
        "serviceIpAddress": "52.224.70.119",
        "vipMappings": []
      },
      "id": null,
      "identity": null,
      "kind": null,
      "location": null,
      "managedBy": null,
      "name": null,
      "plan": null,
      "properties": null,
      "sku": null,
      "tags": null,
      "type": null
}

References

About this issue

Original URL
State: open
Created 6 years ago
Reactions: 11
Comments: 18 (2 by maintainers)

Most upvoted comments

Hi @Phydeauxman! Sorry this didn’t work as expected.

At the moment this data source is limited to dealing only with simple string values, because the provider model requires each attribute to have a type and this one is typed as “map of strings”. The additionalProperties property in your example is hitting this error because it’s not convertible to a string.

It’s currently necessary to flatten the result to use only a single level of keys and only string values. For the lists in your example, it’d be required to use some sort of delimiter and then use the split function to divide the single string into a list.

We do intend to make this more flexible in future but need to do some other work in Terraform Core first so that it’s possible within provider schema to describe the result type of this attribute. (Or more accurately, to describe that its type is not known until it is read.) Since it seems we don’t already have an issue open to represent this, I’m going to label this one to represent that feature. We need to get through some other work first so there won’t be any immediate action here, but we will return to this once that underlying work is done and share more details.

apparentlymart on Mar 19, 2018

This provider was originally written before Terraform had a jsondecode function. I agree that with that function now present it would be better to have a new data source that can just run a program and capture its output as a string.

I would suggest doing that as a new data source in the local provider though, not as an extension of the external data source, because the external data source was designed for use with programs tailored to work with Terraform (as a middle-ground to allow writing small glue programs rather than entire new Terraform providers), not for executing arbitrary existing software with no modifications.

As an example, the new data source might look like this:

data "local_exec" "example" {
  program = ["example", "program", "generating", "csv"]
}

output "example" {
  value = csvdecode(data.local_exec.example.stdout)
}

As with external, it would be important to use this only for programs that don’t have externally-visible side-effects, because the program would be run during the planning phase rather than the apply phase. But unlike external it would impose no constraints on the output except that it be UTF-8 encoded (because Terraform language strings are Unicode, not raw bytes) and leave the user to decide how and whether to parse it.

I don’t work directly on either the external or the local providers, so I’m suggesting the above just as a design starting point, and I can’t promise it would be accepted exactly like that. If you’re interested in working on such a thing I’d suggest starting by opening an issue in the local provider repository to describe what you’re intending to do and get feedback from the team that maintains that provider before doing work on it, in case there are design constraints for that provider that I’m not considering.

apparentlymart on Jun 10, 2020

Hello @apparentlymart , What if we add a raw_result attribute which represents the raw output string from command executed and then use jsondecode built-in function to process the JSON.

Raw output and raw input sound much more useful than json to me. Relatively few legacy command line programs handle JSON. For those that do, the terraform json encoding and decoding functions are present. For csv, yaml, base64, newline delimited text, and all of the other formats in common use, adding a json wrapper is unwelcome overhead.

jeremykatz on Jun 9, 2020