terraform-provider-confluent: Unable to import connectors

Hi, I am trying to import my existing connectors into the confluent terraform provider but am running into issues.

I am using version 1.0.0 of the terraform provider running the following commands:

confluent_connector.postgres_cdc_connector[0]: Importing from ID "env-81ymm0/lkc-380q7o/PostgresCDCConnector_0"...
confluent_connector.postgres_cdc_connector[0]: Import prepared!
  Prepared confluent_connector for import
confluent_connector.postgres_cdc_connector[0]: Refreshing state... [id=lcc-7yzx5w]
╷
│ Error: Cannot import non-existent remote object
│ 
│ While attempting to import an existing object to "confluent_connector.postgres_cdc_connector[0]", the provider detected that no object exists with the given id. Only pre-existing objects can be
│ imported; check that the id is correct and that it is associated with the provider's configured region or endpoint, or use "terraform apply" to create a new remote object for this resource.

I can see that these connectors do exist, but the provider is not finding them

confluent connect --environment $environ --cluster $cluster list
      ID     |               Name               | Status  |  Type  | Trace  
-------------+----------------------------------+---------+--------+--------
  lcc-7yzx5w | PostgresCDCConnector_0           | RUNNING | source |        
  lcc-o3ydz9 | SalesforceSObjectSinkConnector_0 | RUNNING | sink   |

I have also tried importing with the following ways:

<envid>/<clusterid>/<connectorname>
<envid>/<clusterid>/<connectorid>
<connectorid>

Please let me know if you need additional information.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 23

Most upvoted comments

@afoley-st there’s one suspicious line in your resource definition:
resource "confluentcloud_connector" "postgres_cdc_connector" {
...
  provider       = mongeyconfluentcloud
Sounds like you’re using a different TF Provider and not TF Provider for Confluent. Could you take a look at it?

Yeah, apologies here. I was looking at an old branch. Please see new comment, thanks!

afoley-st on Jul 19, 2022

Oh, sorry - thats our old connector config. Let me fetch the new one. Apologies.

afoley-st on Jul 19, 2022

Sure thing. Here is my resource definition:


resource "confluentcloud_connector" "postgres_cdc_connector" {
  count          = length(local.cdc_connectors)
  depends_on     = [confluent_kafka_topic.topics]
  provider       = mongeyconfluentcloud
  cluster_id     = module.confluent_cluster.cluster_id
  environment_id = module.confluent_cluster.environment_id
  name           = "PostgresCDCConnector_${count.index}"

  config = merge(
    (element(local.cdc_connectors, count.index))["config"],
    {
      "connector.class"   = "PostgresCdcSource",
      "database.port"     = "5432",
      "database.dbname"   = data.vault_generic_secret.edge_secrets.data.cloudsql_db_name,
      "database.hostname" = data.vault_generic_secret.edge_secrets.data.cloudsql_proxy_public_ip_address,
      "database.user"     = data.vault_generic_secret.edge_secrets.data.edge_cloudsql_repl_username,
      "table.include.list" = join(
        ",",
        (element(local.cdc_connectors, count.index))["config"]["table.include.list"]
      )
    }
  )
  config_sensitive = {}
}

And the curl response:

alex@Alexanders-MacBook-Pro terraform % curl --request GET --url "https://api.confluent.cloud/connect/v1/environments/env-81ymm0/clusters/lkc-380q7o/connectors?expand=id,status,info" --header "Authorization: Basic $TOKEN" | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2727  100  2727    0     0   8979      0 --:--:-- --:--:-- --:--:--  9307
{
  "PostgresCDCConnector_0": {
    "status": {
      "name": "PostgresCDCConnector_0",
      "connector": {
        "state": "RUNNING",
        "worker_id": "PostgresCDCConnector_0",
        "trace": ""
      },
      "tasks": [
        {
          "id": 0,
          "state": "RUNNING",
          "worker_id": "PostgresCDCConnector_0",
          "msg": ""
        }
      ],
      "type": "source"
    },
    "info": {
      "name": "PostgresCDCConnector_0",
      "type": "source",
      "config": {
        "cloud.environment": "prod",
        "cloud.provider": "gcp",
        "connector.class": "PostgresCdcSource",
        "database.dbname": "<redacted>",
        "database.hostname": "<redacted>",
        "database.password": "****************",
        "database.port": "5432",
        "database.server.name": "<redacted>",
        "database.sslmode": "disable",
        "database.user": "<redacted>",
        "kafka.api.key": "****************",
        "kafka.api.secret": "****************",
        "kafka.endpoint": "SASL_SSL://pkc-419q3.us-east4.gcp.confluent.cloud:9092",
        "kafka.region": "us-east4",
        "name": "PostgresCdcSourceConnector",
        "output.data.format": "JSON",
        "table.include.list": "edge.user,edge.user_status,edge.account_balance,edge.user_spend,edge.user_deposit,edge.limit_increases,edge.limit_settings,edge.cage_transactions,edge.payment_transaction,edge.order,edge.position,edge.execution,edge.session_login,edge.activity_ledger,edge.settings,edge.user_settings",
        "tasks.max": "1"
      },
      "tasks": [
        {
          "connector": "PostgresCDCConnector_0",
          "task": 0
        }
      ]
    },
    "id": {
      "id": "lcc-7yzx5w",
      "id_type": "ID"
    },
    "extensions": {}
  },
  "SalesforceSObjectSinkConnector_0": {
    "status": {
      "name": "SalesforceSObjectSinkConnector_0",
      "connector": {
        "state": "RUNNING",
        "worker_id": "SalesforceSObjectSinkConnector_0",
        "trace": ""
      },
      "tasks": [
        {
          "id": 0,
          "state": "RUNNING",
          "worker_id": "SalesforceSObjectSinkConnector_0",
          "msg": ""
        }
      ],
      "type": "sink"
    },
    "info": {
      "name": "SalesforceSObjectSinkConnector_0",
      "type": "sink",
      "config": {
        "cloud.environment": "prod",
        "cloud.provider": "gcp",
        "connector.class": "SalesforceSObjectSink",
        "input.data.format": "JSON_SR",
        "kafka.api.key": "****************",
        "kafka.api.secret": "****************",
        "kafka.endpoint": "SASL_SSL://pkc-419q3.us-east4.gcp.confluent.cloud:9092",
        "kafka.region": "us-east4",
        "name": "SalesforceSObjectSinkConnector_Import_Object",
        "request.max.retries.time.ms": "1000",
        "salesforce.consumer.key": "****************",
        "salesforce.consumer.secret": "****************",
        "salesforce.custom.id.field.name": "External_Id__c",
        "salesforce.instance": "<redacted>",
        "salesforce.object": "Import_Object__c",
        "salesforce.object.override.event.type": "true",
        "salesforce.password": "****************",
        "salesforce.password.token": "****************",
        "salesforce.sink.object.operation": "upsert",
        "salesforce.use.custom.id.field": "true",
        "salesforce.username": "<redacted>",
        "tasks.max": "1",
        "topics": "salesforce.sink.import-object"
      },
      "tasks": [
        {
          "connector": "SalesforceSObjectSinkConnector_0",
          "task": 0
        }
      ]
    },
    "id": {
      "id": "lcc-o3ydz9",
      "id_type": "ID"
    },
    "extensions": {}
  }
}

afoley-st on Jul 19, 2022