terraform-provider-ec: Race condition on creating Deployment with custom config that relies on keystore secret

In scenario when you specify custom elasticsearch.yaml settings using elasticsearch.config.user_settings_yaml and settings provided there requires you to use keystore for storing secure parameters you end in race condition when creating new deployment / editing existing.

Readiness Checklist

• I am running the latest version (yes 3.0)
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
• I am reporting the issue to the correct repository (for multi-repository projects)

Expected Behavior

Apply should work without failing.

Current Behavior

Example of configuration documentation: https://www.elastic.co/guide/en/cloud/current/ec-securing-clusters-oidc-op.html#ec-securing-oidc-okta (steps 2 & 3)

What is happening:

1. Deployment is being created/updated with new elasticsearch.yaml settings
2. Keystore secret depends on deployment so it is waiting for 1. to finish
3. Elasticsearch cannot be restarted in step 1. which makes apply either hang (when creating deployment from scratch) or immediately fail (when applying to existing deployment).

## Terraform definition

resource "ec_deployment" "race-condition-cluster" {
  name = "race-condition-cluster"

  region                 = "eu-west-1"
  version                = "7.16.3"
  deployment_template_id = "aws-observability"

  elasticsearch {
    topology {
      id         = "hot_content"
      size       = "1g"
      zone_count = 1
    }

    config {
      user_settings_yaml = <<EOF
xpack.security.authc.realms.oidc.oidc1:
  order: 2
  rp.client_id: client_id
  rp.response_type: "code"
  rp.requested_scopes: ["openid", "email"]
  rp.redirect_uri: "https://kibana.example.com/api/security/oidc/callback"
  op.issuer: "https://sample.okta.com"
  op.authorization_endpoint: "https://sample.okta.com/oauth2/v1/authorize"
  op.token_endpoint: "https://sample.okta.com/oauth2/v1/token"
  op.userinfo_endpoint: "https://sample.okta.com/oauth2/v1/userinfo"
  op.endsession_endpoint: "https://sample.okta.com/oauth2/v1/logout"
  op.jwkset_path: "https://sample.okta.com/oauth2/v1/keys"
  claims.principal: email
  claim_patterns.principal: "^([^@]+)@sample\\.com$"
EOF
    }
  }

  kibana {
    topology {
      size       = "1g"
      zone_count = 1
    }

    config {
      user_settings_yaml = <<EOF
xpack.security.authc.providers:
  oidc.oidc1:
    order: 0
    realm: oidc1
    description: "Log in with Okta"
EOF
    }
  }
}

resource "ec_deployment_elasticsearch_keystore" "okta_secret" {
  deployment_id = ec_deployment.race-condition-cluster.id
  setting_name  = "xpack.security.authc.realms.oidc.oidc1.rp.client_secret"
  value         = "my_super_secret"
}

Steps to Reproduce

Apply sample terraform config provided above

Context

We are able to deal with it by simply temporarily commenting out custom settings and doing two applies to fully apply this feature. We would love to have it without manual operations to be done

Possible Solution

Allow to configure custom elasticsearch.yml entries via separate TF resource

Your Environment

• Version used: 0.3.0
• Running against Elastic Cloud SaaS or Elastic Cloud Enterprise and version: EC SaaS
• Environment name and version (e.g. Go 1.9): Terraform 1.1.4