terraform-provider-confluent: Not able to create a dedicated cluster with encryption_key on GCP

Update: we

Added new confluent_byok_key resource and a corresponding data source in a Preview lifecycle stage. Added 2 new examples for confluent_byok_key resource: dedicated-public-aws-byok-kafka-acls dedicated-public-azure-byok-kafka-acls Added support for new computed byok_key block of confluent_kafka_cluster resource and a corresponding data source in a Preview lifecycle stage.

in our latest 1.36.0 version of TF Provider.

Note: all the examples can be run in a single step 👏

cc @larrywax @patrickherrera @pkmec @schlbra @pixie79 @derekwinters

Thanks @linouk23 I think I can imagine how that could happen given you need to setup the policies to allow Cluster access before you can create the Cluster that needs those policies 😃 I checked out the API link you shared and I’m not sure I understand what it is trying to do. It says it is for “Create a Key”, but expects the ARN of an existing key?

Anyway, in answer to your second question, I’ve never encountered an arrangement like that. I would have thought complete ownership within the team was more common, with a DevOps approach to provisioning and supporting an application. In my situation I’ve developed all the infrastructure-as-code to deploy a number of Clusters to different regions and support different environments (dev/staging/production for the moment). Creating and configuring Keys is part of that, and they sit right alongside the code to call the Confluent Provider to provision the Cluster itself. Various people have reviewed the code from a security perspective etc, but all ownership is within that repository and there are no external dependencies. It wouldn’t be the worst thing in the world to provision the Key in a separate step in the same repository if that remove the dependency but I don’t really see how it would be different. Terraform will create things in the right order so whether the Key is created “externally” or not would still have the same dependency wouldn’t it?

Is there any chance you could share a branch with your example as it stands at the moment? Cheers

@cijujoseph @felippe-mendonca @schlbra @harideveloper

Update: we

Added GCP support for confluent_byok_key resource and a corresponding data source in a General Availability lifecycle stage (https://github.com/confluentinc/terraform-provider-confluent/issues/65). Example: dedicated-public-gcp-byok-kafka-acls

in our latest 1.56.0 version of TF Provider.

Note: all BYOK examples can be run in a single step 👏

Update: we

Added new confluent_byok_key resource and a corresponding data source in a Preview lifecycle stage. Added 2 new examples for confluent_byok_key resource: dedicated-public-aws-byok-kafka-acls dedicated-public-azure-byok-kafka-acls Added support for new computed byok_key block of confluent_kafka_cluster resource and a corresponding data source in a Preview lifecycle stage.

in our latest 1.36.0 version of TF Provider.

Note: all the examples can be run in a single step 👏

cc @larrywax @patrickherrera @pkmec @schlbra @pixie79 @derekwinters

This is great news, thank you for the update!

@schlbra thanks for asking! I’m not sure whether I can disclose a lot other than the fact we’re working on it.

@linouk23, are you able to disclose when this feature is planned on the roadmap and is it safe to assume the feature will be supported by both GCP and AWS?

We faced the same issue with AWS (and Terraform). The workaround I did was to run the CLI first (following these instructions) and extract all the account IDs from the generated statements. Luckily you don’t even need to provide a valid ARN for the key so you can run this before you have a key. Then quit the CLI and add those account IDs to your Terraform or other IaC tool and use that during your deployment to generate a key that allows access to those accounts. It does appear you need to provide them all, even though each cluster would have a specific account you would think.

This is a manual step unfortunately, but it only needs to be done once and then everything else can be automated as many times as you like. Creating a Cluster manually and importing each time defeats the point. We did find that the list of Account IDs will change (rarely), so deploying a new cluster might fail until you update the list, but existing ones will continue to work.

So we have some Terraform config with:

confluent_account_ids = [
  "XXX", "YYY", "XXX"
]

And then create a policy which is applied to the Key with the correct permissions (which you will also get from the output of the CLI):

(snip)

  dynamic "statement" {
    for_each = var.confluent_account_ids 
    content {
      sid = "Enable ${statement.value} IAM User Permissions"
      principals {
        type = "AWS"
        identifiers = [
          "arn:aws:iam::${statement.value}:root"
        ]
      }
      effect = "Allow"
      actions = [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncryptFrom",
        "kms:ReEncryptTo",
        "kms:GenerateDataKey",
        "kms:GenerateDataKeyWithoutPlaintext",
        "kms:DescribeKey",
        "kms:CreateGrant",
        "kms:list*",
        "kms:RevokeGrant"
      ]
      resources = [
        "*"
      ]
    }
  }

(snip)

To add more context,

• BYOK AWS – you might ask our support for AWS account IDs / look it up in the CLI so it could work but managing permissions would be a manual step.

• BYOK GCP – creation via TF is not possible (the “getting Google Group ID” step is missing – exactly as you mentioned).

Overall I feel like the best workaround is to create a cluster using CLI / Confluent Cloud Console and then import it via TF.

OK I double checked and it seems like it’s indeed a limitation that we’ll try to address in future versions but there’s no timeline for it yet.

It might be a good idea to keep this issue open until it’s fixed.

For now, the only workaround is to create a cluster using CLI / Confluent Cloud Console and then import it via TF.