pulumi: Exception thrown in stack results in resources marked for deletion

If an exception happens while executing my stack code, it causes the program to exit, and any resource that would have been defined after the exception gets marked for deletion (or worse actually deleted). I have only tried this in the dotnet SDK.

Expected behavior

I would have expected the program exiting on error would cause pulumi to not move forward with the preview/up operation.

Current behavior

Currently, pulumi does see the exception, and logs it as a diagnostic. But it then continues as if your program was complete and any resources that weren’t yet created act as if you were removing them from your source code / stack.

Steps to reproduce

public class Program
{
    public static async Task Main(string[] args)
        => await Deployment.RunAsync<MyStack>();
}

public class MyStack : Stack
{
    public MyStack()
    {
        var a = new Pulumi.Random.RandomInteger("a", new Pulumi.Random.RandomIntegerArgs { Min = 1, Max = 100 });

        if (Environment.GetEnvironmentVariable("oops") == "yes") throw new Exception("oops");

        var b = new Pulumi.Random.RandomInteger("b", new Pulumi.Random.RandomIntegerArgs { Min = 1, Max = 100 });

        var c = new Pulumi.Random.RandomInteger("c", new Pulumi.Random.RandomIntegerArgs { Min = 1, Max = 100 });
    }
}

successful pulumi up :

root@5ecb18015b1a:/app# oops=no pulumi up -y --non-interactive
Previewing update (test):
    pulumi:pulumi:Stack test-test  running 'dotnet build -nologo .'
    pulumi:pulumi:Stack test-test    Restore completed in 36.76 ms for /app/PulumiIssue.csproj.
    pulumi:pulumi:Stack test-test    PulumiIssue -> /app/bin/Debug/netcoreapp3.1/PulumiIssue.dll
    pulumi:pulumi:Stack test-test
    pulumi:pulumi:Stack test-test      0 Warning(s)
    pulumi:pulumi:Stack test-test  'dotnet build -nologo .' completed successfully

 +  pulumi:pulumi:Stack test-test create 'dotnet build -nologo .' completed successfully
 +  random:index:RandomInteger a create
 +  random:index:RandomInteger c create
 +  random:index:RandomInteger b create
 +  pulumi:pulumi:Stack test-test create

Resources:
    + 4 to create

Updating (test):
    pulumi:pulumi:Stack test-test  running 'dotnet build -nologo .'
    pulumi:pulumi:Stack test-test    Restore completed in 34.75 ms for /app/PulumiIssue.csproj.
    pulumi:pulumi:Stack test-test    PulumiIssue -> /app/bin/Debug/netcoreapp3.1/PulumiIssue.dll
    pulumi:pulumi:Stack test-test
    pulumi:pulumi:Stack test-test      0 Warning(s)
    pulumi:pulumi:Stack test-test  'dotnet build -nologo .' completed successfully

 +  pulumi:pulumi:Stack test-test creating 'dotnet build -nologo .' completed successfully
 +  random:index:RandomInteger c creating
 +  random:index:RandomInteger a creating
 +  random:index:RandomInteger b creating
 +  random:index:RandomInteger c created
 +  random:index:RandomInteger a created
 +  random:index:RandomInteger b created
 +  pulumi:pulumi:Stack test-test created

Resources:
    + 4 created

Duration: 3s

followed by a failed preview attempt:

root@5ecb18015b1a:/app# oops=yes pulumi preview --non-interactive
Previewing update (test):
    pulumi:pulumi:Stack test-test  running 'dotnet build -nologo .'
    pulumi:pulumi:Stack test-test    Restore completed in 38.84 ms for /app/PulumiIssue.csproj.
    pulumi:pulumi:Stack test-test    PulumiIssue -> /app/bin/Debug/netcoreapp3.1/PulumiIssue.dll
    pulumi:pulumi:Stack test-test
    pulumi:pulumi:Stack test-test      0 Warning(s)
    pulumi:pulumi:Stack test-test  'dotnet build -nologo .' completed successfully

    pulumi:pulumi:Stack test-test running 'dotnet build -nologo .' completed successfully
    random:index:RandomInteger a
    pulumi:pulumi:Stack test-test running error: Running program '/app/bin/Debug/netcoreapp3.1/PulumiIssue.dll' failed with an unhandled exception:
 -  random:index:RandomInteger c delete
 -  random:index:RandomInteger b delete
    pulumi:pulumi:Stack test-test  1 error

Diagnostics:
  pulumi:pulumi:Stack (test-test):
    error: Running program '/app/bin/Debug/netcoreapp3.1/PulumiIssue.dll' failed with an unhandled exception:
    System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
     ---> System.Exception: oops
       at PulumiIssue.MyStack..ctor() in /app/Program.cs:line 21
       --- End of inner exception stack trace ---
       at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean wrapExceptions, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& hasNoDefaultCtor)
       at System.RuntimeType.CreateInstanceDefaultCtorSlow(Boolean publicOnly, Boolean wrapExceptions, Boolean fillCache)
       at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, Boolean wrapExceptions)
       at System.Activator.CreateInstance[T]()
       at Pulumi.Deployment.Runner.<>c__4`1.<RunAsync>b__4_0()
       at Pulumi.Deployment.Runner.RunAsync[TStack](Func`1 stackFactory)

or if you aren’t careful, a problematic up:

root@5ecb18015b1a:/app# oops=yes pulumi up -y --non-interactive
Previewing update (test):
    pulumi:pulumi:Stack test-test  running 'dotnet build -nologo .'
    pulumi:pulumi:Stack test-test    Restore completed in 35.49 ms for /app/PulumiIssue.csproj.
    pulumi:pulumi:Stack test-test    PulumiIssue -> /app/bin/Debug/netcoreapp3.1/PulumiIssue.dll
    pulumi:pulumi:Stack test-test
    pulumi:pulumi:Stack test-test      0 Warning(s)
    pulumi:pulumi:Stack test-test  'dotnet build -nologo .' completed successfully

    pulumi:pulumi:Stack test-test running 'dotnet build -nologo .' completed successfully
    random:index:RandomInteger a
    pulumi:pulumi:Stack test-test running error: Running program '/app/bin/Debug/netcoreapp3.1/PulumiIssue.dll' failed with an unhandled exception:
 -  random:index:RandomInteger c delete
 -  random:index:RandomInteger b delete
    pulumi:pulumi:Stack test-test  1 error

Diagnostics:
  pulumi:pulumi:Stack (test-test):
    error: Running program '/app/bin/Debug/netcoreapp3.1/PulumiIssue.dll' failed with an unhandled exception:
    System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
     ---> System.Exception: oops
       at PulumiIssue.MyStack..ctor() in /app/Program.cs:line 21
       --- End of inner exception stack trace ---
       at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean wrapExceptions, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& hasNoDefaultCtor)
       at System.RuntimeType.CreateInstanceDefaultCtorSlow(Boolean publicOnly, Boolean wrapExceptions, Boolean fillCache)
       at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, Boolean wrapExceptions)
       at System.Activator.CreateInstance[T]()
       at Pulumi.Deployment.Runner.<>c__4`1.<RunAsync>b__4_0()
       at Pulumi.Deployment.Runner.RunAsync[TStack](Func`1 stackFactory)


Updating (test):
    pulumi:pulumi:Stack test-test  running 'dotnet build -nologo .'
    pulumi:pulumi:Stack test-test    Restore completed in 37.44 ms for /app/PulumiIssue.csproj.
    pulumi:pulumi:Stack test-test    PulumiIssue -> /app/bin/Debug/netcoreapp3.1/PulumiIssue.dll
    pulumi:pulumi:Stack test-test
    pulumi:pulumi:Stack test-test      0 Warning(s)
    pulumi:pulumi:Stack test-test  'dotnet build -nologo .' completed successfully

    pulumi:pulumi:Stack test-test running 'dotnet build -nologo .' completed successfully
    random:index:RandomInteger a
    pulumi:pulumi:Stack test-test running error: Running program '/app/bin/Debug/netcoreapp3.1/PulumiIssue.dll' failed with an unhandled exception:
 -  random:index:RandomInteger c deleting
 -  random:index:RandomInteger b deleting
 -  random:index:RandomInteger c deleted
 -  random:index:RandomInteger b deleted
    pulumi:pulumi:Stack test-test **failed** 1 error

Diagnostics:
  pulumi:pulumi:Stack (test-test):
    error: Running program '/app/bin/Debug/netcoreapp3.1/PulumiIssue.dll' failed with an unhandled exception:
    System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
     ---> System.Exception: oops
       at PulumiIssue.MyStack..ctor() in /app/Program.cs:line 21
       --- End of inner exception stack trace ---
       at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean wrapExceptions, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& hasNoDefaultCtor)
       at System.RuntimeType.CreateInstanceDefaultCtorSlow(Boolean publicOnly, Boolean wrapExceptions, Boolean fillCache)
       at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, Boolean wrapExceptions)
       at System.Activator.CreateInstance[T]()
       at Pulumi.Deployment.Runner.<>c__4`1.<RunAsync>b__4_0()
       at Pulumi.Deployment.Runner.RunAsync[TStack](Func`1 stackFactory)

Resources:
    - 2 deleted
    2 unchanged

Duration: 3s

Context (Environment)

This tripped me up as I was iterating on a stack I have been working on (luckily no real resources were harmed). Towards the beginning of my stack I usually config.Require all of my inputs. When I misspelled one of my config names, it failed super early and therefore pretty much every resource was marked for deletion.

This test was run in the pulumi/pulumi:v3.2.1 docker container.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 16 (16 by maintainers)

Most upvoted comments

@mikhailshilkov OK so #7299 should resolve this. I brought in one the tests that @vipentti wrote to confirm.

+3
orionstudt on Jun 15, 2021

I believe this behavior has existed since the very first release of the .NET plugin.

The problem is that you return Task from the Main method and not Task<int>. It should be:

public static async Task<int> Main(string[] args)
{
  return await Deployment.RunAsync<MyStack>();
}

With this code, RunAsync would return 32 for your example and the deployment would fail appropriately. I believe we use the Task<int> variant in all templates and examples.

I agree this is a confusing and dangerous behavior, so I welcome any improvement suggestions. cc @t0yv0 for awareness.

+1
mikhailshilkov on May 31, 2021
Read more comments on GitHub
← pulumi: Error: package.json export path for ".pnpm/tslib@2.4.0/node_modules/tslib/tslib.js" not found
pulumi: failed to compute archive hash for "code": archive/tar: missed writing 2026611 bytes →