cdk-organizations: Organizations API not able to handle concurrent requests.

We are seeing an error when trying to use this library to attach Service Control Policies, and create Organizational Units. It seems like the AWS Organizations API is not able to handle concurrent requests.

Example code for SCPs:

export class appStack extends cdk.Stack {
    constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
        super(scope, id, props);
        const org = new Organization(this, "master", {
            featureSet: FeatureSet.ALL
        })

        org.attachPolicy(scps.getPolicy(this, scps.denyLeaveOrg))
        org.attachPolicy(scps.getPolicy(this, scps.denyNewRegionsPolicy))
        org.attachPolicy(scps.getPolicy(this, scps.denyOutsideEuCentral1AndUsEast1))
        org.attachPolicy(scps.getPolicy(this, scps.denyCdkBootstrap))
    }
}

CDK can synthesise templates and deploy a Cloudformation Stack. This fails with the following error:

Received response status [FAILED] from custom resource. Message returned: AWS Organizations can’t complete your request because it conflicts with another attempt to modify the same entity. Try again later.

We also see the same error reported when trying to create multiple OUs that are at the same level.

Example code for OUs:

export class appStack extends cdk.Stack {
    constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
        super(scope, id, props);
        const org = new Organization(this, "master", {
            featureSet: FeatureSet.ALL
        })

        const security = new OrganizationalUnit(this, "security", {
            organizationalUnitName: "Security",
            parent: org.root,
        })

        const deployments = new OrganizationalUnit(this, "deployments", {
            organizationalUnitName: "Deployments",
            parent: org.root,
        })
    }
}

And again the same error of

Received response status [FAILED] from custom resource. Message returned: AWS Organizations can’t complete your request because it conflicts with another attempt to modify the same entity. Try again later.

Would it be possible to add an exponential backoff to these requests?

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 19 (6 by maintainers)

Commits related to this issue

feat: sequentially chain accounts Create a dependency chain of accounts in the scope (app or stack) to avoid. AWS Organization only can create one account at a given time. Fixes #447 — committed to pepperize/cdk-organizations by pflorek 2 years ago
fix: scope of policy attachment and delegated administrator avoid circular dependencies, when chaining parent-child and sibling dependencies. If policy attachments are of same dependency group it wil... — committed to pepperize/cdk-organizations by pflorek 2 years ago
fix: specified policy type is already enabled https://github.com/pepperize/cdk-organizations/issues/447 — committed to pepperize/cdk-organizations by pflorek 2 years ago
fix: sequentially chain organization resources enforce a sequential dependency chain of organization resources which cannot be deployed in parallel fixes: https://github.com/pepperize/cdk-organizati... — committed to pepperize/cdk-organizations by pflorek 2 years ago

Most upvoted comments

Thanks @pflorek !

I’ll be testing all of those resources with DependencyChain next week (I’m away for a few days but will run the tests on Wed or Thursday and will let you know how it went) 😃

lkolchin on Sep 10, 2022

@pflorek Sure. Created a separate #632 issue for the second part.

lkolchin on Sep 25, 2022

Hi,

Having the same issues with concurrency 😦

Fairly new to CDK and would love to have an elegant solution similar to what I’m doing with terraform (Would have been great to pass an array of config objects to custom resource so it could take care of concurrency…).

Ideally I’d like to pass a config (buildConfig) for ous, accounts, scps etc with something like this:

{
  "ProjectName": "man-org",
  "AWSRegion": "ap-southeast-2",
  "AppName": "man-org",
  "Environment": "master",
  "assume_role_name": "OrganizationAccountAccessRole",
  "organization_accounts": {
    "man-dev": {
      "accountName": "man-dev",
      "email": "my.email+man-dev@mantalus.com",
      "roleName": "OrganizationAccountAccessRole",
      "iamUserAccessToBilling": "IamUserAccessToBilling.ALLOW",
      "parent": "dev",
      "tags": {
        "ou": "dev"
      }
    },
    "man-network": {
      "accountName": "man-network",
      "email": "my.email+man-network@mantalus.com",
      "roleName": "OrganizationAccountAccessRole",
      "iamUserAccessToBilling": "IamUserAccessToBilling.ALLOW",
      "parent": "network",
      "tags": {
        "ou": "network"
      }
    },
    "man-test": {
      "accountName": "man-test",
      "email": "my.email+man-test@mantalus.com",
      "roleName": "OrganizationAccountAccessRole",
      "iamUserAccessToBilling": "IamUserAccessToBilling",
      "parent": "test",
      "tags": {
        "ou": "test"
      }
    }
  },
  "organization_aws_service_access_principals": [
    "aws-artifact-account-sync.amazonaws.com",
    "ram.amazonaws.com",
    "sso.amazonaws.com",
    "access-analyzer.amazonaws.com",
    "config.amazonaws.com",
    "config-multiaccountsetup.amazonaws.com",
    "securityhub.amazonaws.com"
  ],
  "account_admin_delegation": {
    "man-network": [
      "sso.amazonaws.com"
    ],
    "man-dev": [
      "config.amazonaws.com",
      "config-multiaccountsetup.amazonaws.com",
      "access-analyzer.amazonaws.com"
    ]
  },
  "organization_enabled_policy_types": [
    "SERVICE_CONTROL_POLICY"
  ],
  "organization_feature_set": "ALL",
  "ous_map_policies": {
    "core": [
      "deny-regions",
      "leave-org",
      "guardrails"
    ],
    "dev": [
      "deny-regions",
      "leave-org",
      "guardrails"
    ],
    "lab": [
      "deny-regions",
      "leave-org",
      "guardrails"
    ],
    "network": [
      "deny-regions",
      "leave-org",
      "guardrails"
    ],
    "prod": [
      "deny-regions",
      "leave-org",
      "guardrails"
    ],
    "test": [
      "deny-regions",
      "leave-org",
      "guardrails"
    ]
  },
  "tags": {
    "repo": "https://github.com/man-Infrastructure/man-aws-org-main"
  }
}

I figured a work-around for OUs and Accounts to cater for dependencies config:

    // Create all OUs in the current organizations root
    let ousArray: Array<OrganizationalUnit> = [];
    let ousMap = new Map() // For future use?
    for (let [key, value] of Object.entries(buildConfig.ous_map_policies)) {
      let ou = new OrganizationalUnit(this, key, {
        organizationalUnitName: key,
        parent: organization.root,
        importOnDuplicate: true,
      });
      ousArray.push(ou);
      ousMap.set(key, ou)
      // console.log("OUkey, OU 👉", key, ou);

    }

    // Organizations API is not able to handle concurrent requests
    for(let i=0; i<ousArray.length; i++){
      // console.log(ousArray[i]);
      //  Apparently: Received response status [FAILED] from custom resource. Message returned: AWS Organizations can't complete your request because it conflicts with another attempt to modify the same entity
      if (i > 0 ) 
      {
        ousArray[i].node.addDependency(ousArray[i-1]);
      }
    }

But can’t do the same workaround for enablePolicyType/enableAwsServiceAccess/delegateAdministrator/attachPolicy and Policy? as I’m getting - Property 'node' does not exist on type 'void' 😦

Any idea how to do that? (Saw your PR - https://github.com/pepperize/cdk-organizations/pull/547 but can’t figure how to make it work with the above enablePolicyType/enableAwsServiceAccess/delegateAdministrator/attachPolicy and Policy?

lkolchin on Aug 22, 2022

Hey @moltar ,

thank you for your ideas. I’ve just created a poc with the Aspects: https://github.com/pepperize/cdk-organizations/pull/547

WDYT???

pflorek on Aug 12, 2022

OK, thanks for the feedback. We will be able to work around this… but IMO I think that the consumers of the library should not have to worry about this API limitation.

If you are not going to change from the lambdas, it would be great to update the examples to show how to attach multiple SCPs at the same level, and explain how these dependencies are then defined.

kiwi-33 on Jun 27, 2022