pulumi: How to solve Chicken & Egg problem / cyclic dependencies / update resources?

Write-up of the explorations into circular dependencies during our recent hack day: https://www.pulumi.com/blog/exploring-circular-dependencies/

I just stumbled across this problem with what I believe should be a rather common use case:

Let’s say there are two web apps (in my case Azure App Services): a backend and a frontend. The frontend obviously needs the backend URL in order to connect to it. The backend however needs the URL of the frontend host to successfully configure CORS. Since both settings are set during/before resource creation, but the URLs are only known after their creation, there is a cyclic dependency that seems impossible to break.

This is a very simple example and it was not that hard to spot, but as @BerndWessels already said there will surely be many more and also much more complex scenarios.

Since this has been reported almost a year ago (and the linked issue even more than 3 years ago), I dare to ask: is this still on the roadmap or even actively worked on? I can imagine that it would be beneficial for so many use cases, but also a pretty hard nut to crack.

I am wondering if there is any progress on the matter?

A quick update on this issue: I’ve got a draft proposal for an approach which I think could be used to solve these cyclic dependencies in a method which wouldn’t break the core requirements of Pulumi as a declaritive system.

This is being tracked as a separate issue:

• https://github.com/pulumi/pulumi/issues/13999

Where we’re at right now is stil to use the workaround discussed at the begging of the blog article - referencing your own stack and deploying twice. Any future developments to avoid the second deployment are likely to be a way off for the time being.

const currentStack = new pulumi.StackReference(
  `acme-org/${pulumi.getProject()}/${pulumi.getStack()}`
);

const ping = new CallbackFunction("ping", {
  callback: pingPongHandler,
  environment: {
    variables: {
      // Get the "pong" function from the stack. On the first deploy this will be set 
      // to a default value (an empty string). On the second deploy, once ping and 
      // pong both exist, this will be updated to be the real value of pong 
      // and complete the creation of the cyclic reference. 
      OPPONENT_FN_NAME: currentStack
        .getOutput("pongName")
        .apply((pongName) => pongName ?? ""),
    },
  },
});

const pong = new CallbackFunction("pong", {
  callback: pingPongHandler,
  environment: {
    variables: {
      OPPONENT_FN_NAME: ping.name,
    },
  },
});

export const pingName = ping.name;
export const pongName = pong.name;

@niklasravnsborg there is a specific issue for this case: https://github.com/pulumi/pulumi-azure-native/issues/578

I made an example of how to solve the “chicken and egg” w/ automation API in https://www.youtube.com/watch?v=N6Wn8dKgJ34 (code at https://github.com/pulumi/pulumitv/blob/master/modern-infrastructure-wednesday/2020-10-21/)

It’s not quite a solution but IMHO I found a good enough work around. It is similar to the approach from @leezen. In order to break the cycle, the code basically checks if a resource that is part of the cycle already exists. If it does it uses the required value from that resource but if it doesn’t it uses some temporary default value. One still has to run pulumi up twice (or possibly more, depending on the complexity of the case) but it should be fairly simple to make your ci run it until there are no more changes.

Anyway here is my example from DigitalOcean (A-record requires IP, IP comes from LB which requires cert, cert requires A-record):

import * as pulumi from "@pulumi/pulumi";
import * as ocean from "@pulumi/digitalocean";

const baseDomainStr = "example.com"
const subDomainStr = "mytest";
const region = ocean.Region.FRA1;
const dropletTag = "mytag:myvalue"


const baseDomain = ocean.Domain.get(baseDomainStr, baseDomainStr);
const loadBalancerName = `${subDomainStr}-loadbalancer`;

ocean.getLoadBalancer({name: loadBalancerName}).then(
    // LoadBalancer has already been created, use its IP
    (result) => setupLoadbalancer(result.ip),

    // LoadBalancer has not been created yet, use "0.0.0.0" as temporary value
    () => setupLoadbalancer("0.0.0.0")
)

function setupLoadbalancer(ip: string): ocean.LoadBalancer {
    let dnsRecord = new ocean.DnsRecord(
        `${subDomainStr}-a-record`,
        {
            name: `*.${subDomainStr}`,
            domain: baseDomain.name,
            type: "A",
            value: ip,
        },
    );

    let cert = new ocean.Certificate(
        `${subDomainStr}-cert`,
        {
            name: `${subDomainStr}-cert`,
            domains: [
                baseDomain.name,
                dnsRecord.fqdn
            ],
            type: ocean.CertificateType.LetsEncrypt
        }
    );

    let loadBalancer = new ocean.LoadBalancer(
        loadBalancerName,
        {
            name: loadBalancerName,
            dropletTag: dropletTag,
            region: region,
            disableLetsEncryptDnsRecords: true,
            forwardingRules: [{
                entryPort: 443,
                entryProtocol: "https",
                certificateName: cert.name,
                targetPort: 80,
                targetProtocol: "http",
                tlsPassthrough: false
            }],
            healthcheck: {
                port: 22,
                protocol: "tcp",
            },
        },
    );

    return loadBalancer;
}

Adding two new usecases to this “feature request”, centered around a “a child is being replaced I need to replace its parent as well” case (new keyword : interdependent resources).

Case one. I have a list of EC2 Elastic IP pre-whitelisted at one of my provider. I iterate over the list and find one IP that is not already allocated each time an EC2 instance is created (aborting if no free IP is found). In this scenario the dynamic resource, let’s call it PreWhitelistedEipPool, is a parent of ec2.Instance via its output PreWhitelistedEipPool.selected_ip given to the instance as input. If the instance is replaced I want PreWhitelistedEipPool to select another free IP and not return the previously returned one which is still associated to the instance being replaced, otherwise the whole “create before delete” strategy fails.

Case two. For provisioning purposes I’m trying to generate a new short-lived SSH key pair each time an EC2 instance is created. In this setup the EC2 instance depends on the key pair but if the EC2 Instance is replaced I have no way to mark the key pair as “needed to be replaced as well”.

From what I can tell, the automation API is not available in C# yet (https://github.com/pulumi/pulumi/issues/5596). Therefore, I worked around the problem in a sort-of-gross way adding logic to an Output that has side-effects. You can isolate the side-effects to non-dry runs by checking Deployment.Instance.IsDryRun. This is probably ill-advised for reasons I don’t understand but I figured I’d mention it.

For my case, I wanted to create an AAD app registration (via the AzureAD package) and consider it into an Azure App Service. The circular dependency is that you need the app service URL for the allowed redirect URLs on the app registration but you need the app registration client ID for the app service configuration.

Example is here: https://github.com/joelverhagen/ExplorePackages/blob/b255c7564059b27eb22d9cd0ec2facf11a6606fd/src/ExplorePackages.Infrastructure/MyStack.cs#L101-L161

attach a Policy

Can this step not be used to break the cycle? That is - can the policy be attached after the Lambda registered to the PostConfirmation Trigger? Or does that somehow block the registration of the handler?

Hi @BerndWessels,

I don’t think that we have a satisfying answer here, unfortunately. I do think that this idea (being able to express a new desired state for a resource as part of an update) is something that’s very interesting a bullet we’ll have to bite off at some point, but we don’t have plans to do so at this time.

We’ll keep this issue open to track the the request.