terraform-provider-confluent: error creating Tag Binding / Business Metadata Binding 404
Hello,
we are using new confluent_business_metadata_binding and confluent_tag_binding and we are adding the tags and business metadata on topic and schema.
The common case for us is create:
- topic
- schema
- tags
- business metadata
in one run. The topics/schema contains usually 3-4 tags, 1-2 business metadata.
But when are running the terraform in some case we are getting error:
│ Error: Provider produced inconsistent result after apply │ │ When applying changes to │ module.....confluent_business_metadata_binding.business_metadata_topic_entity_type["Team"], │ provider "provider[\"registry.terraform.io/confluentinc/confluent\"]" │ produced an unexpected new value: Root resource was present, but now │ absent. │ │ This is a bug in the provider, which should be reported in the provider's │ own issue tracker.
Error: error creating Tag Binding 404 Not Found
Error creating Business Metadata Binding 404 Not Found
When we rerun terraform again, in some cases it will finish successfully, but in other case there is error regarding the tags or metadata that already exists.
error creating Business Metadata Binding ".../Team/....:topic/kafka_topic": Business-metadata attribute already exists in entity: ....
We have to manually removed tags/metadata to able run the terraform successful.
Configuration looks like something like this:
# create tags for open on topic
resource "confluent_tag_binding" "open-topic-tagging" {
schema_registry_cluster {
id = confluent_schema_registry_cluster.main.id
}
rest_endpoint = confluent_schema_registry_cluster.main.rest_endpoint
credentials {
key = confluent_api_key.env-manager-schema-registry-api-key.id
secret = confluent_api_key.env-manager-schema-registry-api-key.secret
}
tag_name = confluent_tag.open.name
entity_name = "${confluent_schema_registry_cluster.main.id}:${confluent_kafka_cluster.basic.id}:${confluent_kafka_topic.purchase.topic_name}"
entity_type = local.topic_entity_type
}
# create tags for private on topic
resource "confluent_tag_binding" "private-topic-tagging" {
schema_registry_cluster {
id = confluent_schema_registry_cluster.main.id
}
rest_endpoint = confluent_schema_registry_cluster.main.rest_endpoint
credentials {
key = confluent_api_key.env-manager-schema-registry-api-key.id
secret = confluent_api_key.env-manager-schema-registry-api-key.secret
}
tag_name = confluent_tag.private.name
entity_name = "${confluent_schema_registry_cluster.main.id}:${confluent_kafka_cluster.basic.id}:${confluent_kafka_topic.purchase.topic_name}"
entity_type = local.topic_entity_type
}
# create tags for open on schema
resource "confluent_tag_binding" "open-schema-tagging" {
schema_registry_cluster {
id = confluent_schema_registry_cluster.main.id
}
rest_endpoint = confluent_schema_registry_cluster.main.rest_endpoint
credentials {
key = confluent_api_key.env-manager-schema-registry-api-key.id
secret = confluent_api_key.env-manager-schema-registry-api-key.secret
}
tag_name = confluent_tag.open.name
entity_name = "${confluent_schema_registry_cluster.main.id}:${confluent_kafka_cluster.basic.id}:${confluent_kafka_topic.purchase.topic_name}"
entity_type = local.topic_entity_type
}
# create tags for private on schema
resource "confluent_tag_binding" "private-schema-tagging" {
schema_registry_cluster {
id = confluent_schema_registry_cluster.main.id
}
rest_endpoint = confluent_schema_registry_cluster.main.rest_endpoint
credentials {
key = confluent_api_key.env-manager-schema-registry-api-key.id
secret = confluent_api_key.env-manager-schema-registry-api-key.secret
}
tag_name = confluent_tag.private.name
entity_name = "${confluent_schema_registry_cluster.main.id}:${confluent_kafka_cluster.basic.id}:${confluent_kafka_topic.purchase.topic_name}"
entity_type = local.schema_entity_type
}
# create business metadata for team on topic
resource "confluent_business_metadata_binding" "topic-bm-binding-team" {
schema_registry_cluster {
id = confluent_schema_registry_cluster.main.id
}
rest_endpoint = confluent_schema_registry_cluster.main.rest_endpoint
credentials {
key = confluent_api_key.env-manager-schema-registry-api-key.id
secret = confluent_api_key.env-manager-schema-registry-api-key.secret
}
business_metadata_name = confluent_business_metadata.main.name
entity_name = "${confluent_schema_registry_cluster.main.id}:${confluent_kafka_cluster.basic.id}:${confluent_kafka_topic.purchase.topic_name}"
entity_type = local.topic_entity_type
attributes = {
"team" = "Sam"
}
}
# create business metadata for team on schema
resource "confluent_business_metadata_binding" "schema-bm-binding-team" {
schema_registry_cluster {
id = confluent_schema_registry_cluster.main.id
}
rest_endpoint = confluent_schema_registry_cluster.main.rest_endpoint
credentials {
key = confluent_api_key.env-manager-schema-registry-api-key.id
secret = confluent_api_key.env-manager-schema-registry-api-key.secret
}
business_metadata_name = confluent_business_metadata.main.name
entity_name = "${confluent_schema_registry_cluster.main.id}:.:${confluent_schema.purchase.schema_identifier}"
entity_type = local.schema_entity_type
attributes = {
"team" = "Sam"
}
}
locals {
topic_entity_type = "kafka_topic"
schema_entity_type = "sr_schema"
record_entity_type = "sr_record"
}
The version
confluent = {
source = "confluentinc/confluent"
version = "~> 1.48.0"
}
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 23 (3 by maintainers)
Commits related to this issue
- Update ksql-acls example (#282) — committed to javabrett/terraform-provider-confluent by linouk23 2 years ago
@dandrivetime Thanks for sharing. This is our assumption as well, we have added a 5s sleep before creating the binding, the other thing we can do is to increase the sleep time, we will contact our SR as well to investigate.
@linouk23 - yes the SR’s are in the same environment. We have a test and prod env. They’re both type:
dedicated(running in azure).I’m not sure why this is so different for us, vs. your local testing! Even if I immediately ( as fast as I can up arrow and press the enter key) run the code again, it will work the 2nd time.
This suggests there’s not really anything wrong with the inputs/resource setup in our code, but rather something going on with coordination we’re not able to see.
EDIT: forgot to mention, the topic always goes 1st in our setup. The tag binding doesn’t start until after TF gets the “creation complete” message, so the dependency is recognized.
@zhenli00 - It’s a reference. (@chawleejay and I work together). It seems like what’s happening is the schema registry is not yet “aware” of the newly created topic (even though terraform receives the “Creation complete” message). If the same code is executed a 2nd time, the tag binding completes without error.
This doesn’t occur on adding a tag to an existing topic, which supports the theory - though it’s just a guess.
` confluent-test Initializing the backend… confluent-test Initializing provider plugins… confluent-test - Reusing previous version of confluentinc/confluent from the dependency lock file confluent-test - Using previously-installed confluentinc/confluent v1.51.0 confluent-test Terraform has been successfully initialized!
confluent-test Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: confluent-test + create
confluent-test
Do you want to perform these actions? Terraform will perform the actions described above. Only ‘yes’ will be accepted to approve. confluent-test Enter a value: yes confluent-test confluent_kafka_topic.tp1 (tp1): Creating… confluent-test confluent_kafka_topic.tp1 (tp1): Still creating… [10s elapsed] confluent-test confluent_kafka_topic.tp1 (tp1): Creation complete after 11s [id=<removed cluster id>/public.iac.tag.test] confluent-test confluent_tag_binding.tp1-Public (tp1-Public): Creating… confluent-test ╷ │ Error: error creating Tag Binding 404 Not Found │ │ with confluent_tag_binding.tp1-Public (tp1-Public), │ on cdk.tf.json line 46, in resource.confluent_tag_binding.tp1-Public (tp1-Public): │ 46: } │ ╵
0 Stacks deploying 1 Stack done 0 Stacks waiting Invoking Terraform CLI failed with exit code 1 `
@marconak-itera thanks for testing it out!
Could you try new version 1.50.0: https://github.com/confluentinc/terraform-provider-confluent/blob/master/CHANGELOG.md#1500-july-21-2023 too when you’ve got a moment? We’ve increased timeouts so it should be even more reliable. Thanks!
This was fixed in
1.49.0: https://github.com/confluentinc/terraform-provider-confluent/blob/master/CHANGELOG.md#1490-july-17-2023@marconak-itera could you try again and see whether you can reproduce the issue? Thanks!