pulumi: [sdk/dotnet] Possible race condition results in `NullReferenceException`

Recently I’ve been experiencing random failures with NullReferenceException which looks due to a race condition:

Diagnostics:
  pulumi:pulumi:Stack (aws-eks-alpha):
    error: Running program 'D:\Devel\Mps\devops-gemini-pulumi\Gemini\bin\Debug\gemini.dll' failed with an unhandled exception:
    System.NullReferenceException: Object reference not set to an instance of an object.
       at Task<HashSet<string>> Pulumi.Deployment.GetAllTransitivelyReferencedCustomResourceUrnsAsync(HashSet<Resource> resources)+(CustomResource r) => { } in /_/sdk/dotnet/Pulumi/Deployment/Deployment_Prepare.cs:line 152
       at bool System.Linq.Enumerable+SelectEnumerableIterator<TSource, TResult>.MoveNext()
       at Task<TResult[]> System.Threading.Tasks.Task.WhenAll<TResult>(IEnumerable<Task<TResult>> tasks)
       at async Task<HashSet<string>> Pulumi.Deployment.GetAllTransitivelyReferencedCustomResourceUrnsAsync(HashSet<Resource> resources) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_Prepare.cs:line 153
       at async Task<PrepareResult> Pulumi.Deployment.PrepareResourceAsync(string label, Resource res, bool custom, bool remote, ResourceArgs args, ResourceOptions options) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_Prepare.cs:line 80
       at async Task<(string urn, string id, Struct data, ImmutableDictionary<string, ImmutableHashSet<Resource>> dependencies)> Pulumi.Deployment.RegisterResourceAsync(Resource resource, bool remote, Func<string, Resource> newDependency, ResourceArgs args, ResourceOptions options) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_RegisterResource.cs:line 27
       at async Task<(string urn, string id, Struct data, ImmutableDictionary<string, ImmutableHashSet<Resource>> dependencies)> Pulumi.Deployment.ReadOrRegisterResourceAsync(Resource resource, bool remote, Func<string, Resource> newDependency, ResourceArgs args, ResourceOptions options) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_ReadOrRegisterResource.cs:line 73
       at async Task Pulumi.Deployment.CompleteResourceAsync(Resource resource, bool remote, Func<string, Resource> newDependency, ResourceArgs args, ResourceOptions options, ImmutableDictionary<string, IOutputCompletionSource> completionSources) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_ReadOrRegisterResource.cs:line 89
       at async void Pulumi.Deployment+Runner+<>c__DisplayClass10_0.<WhileRunningAsync>g__HandleCompletion|0(?)+HandleCompletion(?) x 2 in /_/sdk/dotnet/Pulumi/Deployment/Deployment.Runner.cs:line 137
       at async Task<int> Pulumi.Deployment+Runner.WhileRunningAsync() in /_/sdk/dotnet/Pulumi/Deployment/Deployment.Runner.cs:line 177
       at async void Pulumi.Deployment+Runner+<>c__DisplayClass10_0.<WhileRunningAsync>g__HandleCompletion|0(?)+HandleCompletion(?) x 2 in /_/sdk/dotnet/Pulumi/Deployment/Deployment.Runner.cs:line 137
       at void System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

Diagnostics:
  pulumi:pulumi:Stack (k8s-alpha):
    error: Running program 'D:\Devel\Mps\devops-gemini-pulumi\Gemini\bin\Debug\gemini.dll' failed with an unhandled exception:
    System.NullReferenceException: Object reference not set to an instance of an object.
       at Output<T> Pulumi.Input<T>.op_Implicit(Input<T> input) in /_/sdk/dotnet/Pulumi/Core/Input.cs:line 43
       at async Task<OutputData<ValueTuple<T1, T2, T3, T4, T5, T6, T7, ValueTuple<T8>>>> Pulumi.Output<T>.TupleHelperAsync<T1, T2, T3, T4, T5, T6, T7, T8>(Input<T1> item1, Input<T2> item2, Input<T3> item3, Input<T4> item4, Input<T5> item5, Input<T6> item6, Input<T7> item7, Input<T8> item8)+GetData(?) in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 328
       at async Task<OutputData<ValueTuple<T1, T2, T3, T4, T5, T6, T7, ValueTuple<T8>>>> Pulumi.Output<T>.TupleHelperAsync<T1, T2, T3, T4, T5, T6, T7, T8>(Input<T1> item1, Input<T2> item2, Input<T3> item3, Input<T4> item4, Input<T5> item5, Input<T6> item6, Input<T7> item7, Input<T8> item8) in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 318
       at async void Pulumi.Deployment+Runner+<>c__DisplayClass13_0.<WhileRunningAsync>g__HandleCompletion|0(?)+HandleCompletion(?) in /_/sdk/dotnet/Pulumi/Deployment/Deployment.Runner.cs:line 141
       at async Task<OutputData<U>> Pulumi.Output<T>.ApplyHelperAsync<U>(Task<OutputData<T>> dataTask, Func<T, Output<U>> func) in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 254
       at async Task<bool> Pulumi.Utilities.OutputUtilities.GetIsKnownAsync<T>(Output<T> output) in /_/sdk/dotnet/Pulumi/Core/OutputUtilities.cs:line 53
       at async Task<ImmutableHashSet<Resource>> Pulumi.Output<T>.Pulumi.IOutput.GetResourcesAsync() in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 159
       at async Task<OutputData<ValueTuple<T1, T2, T3, T4, T5, T6, T7, ValueTuple<T8>>>> Pulumi.Output<T>.TupleHelperAsync<T1, T2, T3, T4, T5, T6, T7, T8>(Input<T1> item1, Input<T2> item2, Input<T3> item3, Input<T4> item4, Input<T5> item5, Input<T6> item6, Input<T7> item7, Input<T8> item8)+GetData(?) in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 329
       at async void Pulumi.Deployment+Runner+<>c__DisplayClass13_0.<WhileRunningAsync>g__HandleCompletion|0(?)+HandleCompletion(?) x 2 in /_/sdk/dotnet/Pulumi/Deployment/Deployment.Runner.cs:line 141
       at async Task<OutputData<ValueTuple<T1, T2, T3, T4, T5, T6, T7, ValueTuple<T8>>>> Pulumi.Output<T>.TupleHelperAsync<T1, T2, T3, T4, T5, T6, T7, T8>(Input<T1> item1, Input<T2> item2, Input<T3> item3, Input<T4> item4, Input<T5> item5, Input<T6> item6, Input<T7> item7, Input<T8> item8) in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 314
       at async void Pulumi.Deployment+Runner+<>c__DisplayClass13_0.<WhileRunningAsync>g__HandleCompletion|0(?)+HandleCompletion(?) in /_/sdk/dotnet/Pulumi/Deployment/Deployment.Runner.cs:line 141
       at async Task<OutputData<U>> Pulumi.Output<T>.ApplyHelperAsync<U>(Task<OutputData<T>> dataTask, Func<T, Output<U>> func) x 2 in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 254
       at async void Pulumi.Deployment+Runner+<>c__DisplayClass13_0.<WhileRunningAsync>g__HandleCompletion|0(?)+HandleCompletion(?) in /_/sdk/dotnet/Pulumi/Deployment/Deployment.Runner.cs:line 141
       at async Task<OutputData<object>> Pulumi.Output<T>.Pulumi.IOutput.GetDataAsync() in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 164
       at async void Pulumi.Deployment+Runner+<>c__DisplayClass13_0.<WhileRunningAsync>g__HandleCompletion|0(?)+HandleCompletion(?) in /_/sdk/dotnet/Pulumi/Deployment/Deployment.Runner.cs:line 141
       at async Task<object> Pulumi.Serialization.Serializer.SerializeAsync(string ctx, object prop, bool keepResources) in /_/sdk/dotnet/Pulumi/Serialization/Serializer.cs:line 138
       at async Task<SerializationResult> Pulumi.Deployment.SerializeFilteredPropertiesAsync(string label, IDictionary<string, object> args, Predicate<string> acceptKey, bool keepResources) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_Serialization.cs:line 53

Diagnostics:
  pulumi:pulumi:Stack (k8s-alpha):
    error: Running program 'D:\Devel\Mps\devops-gemini-pulumi\Gemini\bin\Debug\gemini.dll' failed with an unhandled exception:
    System.NullReferenceException: Object reference not set to an instance of an object.
       at Output<T> Pulumi.Input<T>.op_Implicit(Input<T> input) in /_/sdk/dotnet/Pulumi/Core/Input.cs:line 43
       at async Task<OutputData<ValueTuple<T1, T2, T3, T4, T5, T6, T7, ValueTuple<T8>>>> Pulumi.Output<T>.TupleHelperAsync<T1, T2, T3, T4, T5, T6, T7, T8>(Input<T1> item1, Input<T2> item2, Input<T3> item3, Input<T4> item4, Input<T5> item5, Input<T6> item6, Input<T7> item7, Input<T8> item8)+GetData(?) in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 328
       at async Task<OutputData<ValueTuple<T1, T2, T3, T4, T5, T6, T7, ValueTuple<T8>>>> Pulumi.Output<T>.TupleHelperAsync<T1, T2, T3, T4, T5, T6, T7, T8>(Input<T1> item1, Input<T2> item2, Input<T3> item3, Input<T4> item4, Input<T5> item5, Input<T6> item6, Input<T7> item7, Input<T8> item8) in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 318
       at async Task<OutputData<U>> Pulumi.Output<T>.ApplyHelperAsync<U>(Task<OutputData<T>> dataTask, Func<T, Output<U>> func) in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 254
       at async void Pulumi.Deployment+Runner+<>c__DisplayClass13_0.<WhileRunningAsync>g__HandleCompletion|0(?)+HandleCompletion(?) x 2 in /_/sdk/dotnet/Pulumi/Deployment/Deployment.Runner.cs:line 141
       at async Task<bool> Pulumi.Utilities.OutputUtilities.GetIsKnownAsync<T>(Output<T> output) in /_/sdk/dotnet/Pulumi/Core/OutputUtilities.cs:line 53
       at async Task<OutputData<ValueTuple<T1, T2, T3, T4, T5, T6, T7, ValueTuple<T8>>>> Pulumi.Output<T>.TupleHelperAsync<T1, T2, T3, T4, T5, T6, T7, T8>(Input<T1> item1, Input<T2> item2, Input<T3> item3, Input<T4> item4, Input<T5> item5, Input<T6> item6, Input<T7> item7, Input<T8> item8) in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 314
       at async void Pulumi.Deployment+Runner+<>c__DisplayClass13_0.<WhileRunningAsync>g__HandleCompletion|0(?)+HandleCompletion(?) x 2 in /_/sdk/dotnet/Pulumi/Deployment/Deployment.Runner.cs:line 141
       at async Task<OutputData<U>> Pulumi.Output<T>.ApplyHelperAsync<U>(Task<OutputData<T>> dataTask, Func<T, Output<U>> func) in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 254
       at async void Pulumi.Deployment+Runner+<>c__DisplayClass13_0.<WhileRunningAsync>g__HandleCompletion|0(?)+HandleCompletion(?) in /_/sdk/dotnet/Pulumi/Deployment/Deployment.Runner.cs:line 141
       at async Task<OutputData<U>> Pulumi.Output<T>.ApplyHelperAsync<U>(Task<OutputData<T>> dataTask, Func<T, Output<U>> func) in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 254
       at async void Pulumi.Deployment+Runner+<>c__DisplayClass13_0.<WhileRunningAsync>g__HandleCompletion|0(?)+HandleCompletion(?) in /_/sdk/dotnet/Pulumi/Deployment/Deployment.Runner.cs:line 141
       at async Task<OutputData<object>> Pulumi.Output<T>.Pulumi.IOutput.GetDataAsync() in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 164
       at void System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

I haven’t had time to dig into them yet. As usual I’m using the very latest version of Pulumi & Pulumi.Automation (3.6.0), although the above were captured just before upgrading.

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 29 (27 by maintainers)

Most upvoted comments

Many thanks @t0yv0 for fixing this tricky issue which has been blocking us for months from deploying with pulumi into prod. 🎉

gitfool on Dec 15, 2022

Good to know I’m not alone. I was beginning to think not many others were using dotnet or the dotnet automation api, or that I was somehow abusing the dotnet automation api.

gitfool on Nov 5, 2021

We are hitting this issue with varying degrees of frequency (about 1 in 20) during our deploys. Its becoming a real pain, since we can no longer reliably do a deployment without anyone attending to it.

d-colwell on Nov 5, 2021

@t0yv0 I hit the issue again using Pulumi & Pulumi.Automation 3.10.3 (which include your recent fix):

Diagnostics:
  pulumi:pulumi:Stack (k8s-ci):
    error: Running program 'D:\Devel\Mps\devops-gemini-pulumi\Gemini\bin\Debug\gemini.dll' failed with an unhandled exception:
    System.NullReferenceException: Object reference not set to an instance of an object.
       at Task<HashSet<string>> Pulumi.Deployment.GetAllTransitivelyReferencedResourceUrnsAsync(HashSet<Resource> resources)+(Resource r) => { } in /_/sdk/dotnet/Pulumi/Deployment/Deployment_Prepare.cs:line 162
       at bool System.Linq.Enumerable+WhereSelectEnumerableIterator<TSource, TResult>.MoveNext()
       at Task<TResult[]> System.Threading.Tasks.Task.WhenAll<TResult>(IEnumerable<Task<TResult>> tasks)
       at async Task<HashSet<string>> Pulumi.Deployment.GetAllTransitivelyReferencedResourceUrnsAsync(HashSet<Resource> resources) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_Prepare.cs:line 163
       at async Task<PrepareResult> Pulumi.Deployment.PrepareResourceAsync(string label, Resource res, bool custom, bool remote, ResourceArgs args, ResourceOptions options) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_Prepare.cs:line 80
       at async Task<(string urn, string id, Struct data, ImmutableDictionary<string, ImmutableHashSet<Resource>> dependencies)> Pulumi.Deployment.RegisterResourceAsync(Resource resource, bool remote, Func<string, Resource> newDependency, ResourceArgs args, ResourceOptions options) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_RegisterResource.cs:line 27
       at async Task<(string urn, string id, Struct data, ImmutableDictionary<string, ImmutableHashSet<Resource>> dependencies)> Pulumi.Deployment.ReadOrRegisterResourceAsync(Resource resource, bool remote, Func<string, Resource> newDependency, ResourceArgs args, ResourceOptions options) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_ReadOrRegisterResource.cs:line 73
       at async Task Pulumi.Deployment.CompleteResourceAsync(Resource resource, bool remote, Func<string, Resource> newDependency, ResourceArgs args, ResourceOptions options, ImmutableDictionary<string, IOutputCompletionSource> completionSources) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_ReadOrRegisterResource.cs:line 89
       at async Task<T> Pulumi.Output<T>.GetValueAsync(T whenUnknown) in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 153
       at async Task Pulumi.Deployment.RegisterResourceOutputsAsync(Resource resource, Output<IDictionary<string, object>> outputs) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_RegisterResourceOutputs.cs:line 28

Note: this happened when I ran a second pass after an update to verify there were no further updates; i.e. it happened when there were no expected stack changes.

gitfool on Aug 21, 2021

@t0yv0 I keep hitting this issue, sometimes several times a day.

I created gitfool/Pulumi.Dungeon as a playground and example of using the dotnet automation api, using a generic host with hierarchical configuration sources (bypassing pulumi configuration), dependency injection and spectreconsole/spectre.console for the cli; but I’ve only released the skeleton for now. I doubt this will be enough to help you repro this issue though so I’ll follow up with even more code in private.

gitfool on Aug 5, 2021

@t0yv0 I ran 1000 iterations yesterday from the cli, to no avail.

This morning I tried again, manually running each iteration from Visual Studio 2019, and it failed on the 10th iteration:

  pulumi:pulumi:Stack (k8s-dev):
    error: Running program 'D:\Devel\Mps\devops-gemini-pulumi\Gemini\bin\Debug\gemini.dll' failed with an unhandled exception:
    System.Exception: Uninitialized `Urn` property on Pulumi.Kubernetes.ApiExtensions.V1Beta1.CustomResourceDefinition resource. Please report to https://github.com/pulumi/pulumi/issues
       at Output<string> Pulumi.Resource.get_Urn() in /workspaces/pulumi/sdk/dotnet/Pulumi/Resources/Resource.cs:line 74
       at Task<HashSet<string>> Pulumi.Deployment.GetAllTransitivelyReferencedCustomResourceUrnsAsync(Resource rootResource, HashSet<Resource> resources)+(CustomResource r) => { } in /workspaces/pulumi/sdk/dotnet/Pulumi/Deployment/Deployment_Prepare.cs:line 162
       at bool System.Linq.Enumerable+SelectEnumerableIterator<TSource, TResult>.MoveNext()
       at Task<TResult[]> System.Threading.Tasks.Task.WhenAll<TResult>(IEnumerable<Task<TResult>> tasks)
       at async Task<HashSet<string>> Pulumi.Deployment.GetAllTransitivelyReferencedCustomResourceUrnsAsync(Resource rootResource, HashSet<Resource> resources) in /workspaces/pulumi/sdk/dotnet/Pulumi/Deployment/Deployment_Prepare.cs:line 164
       at async Task<PrepareResult> Pulumi.Deployment.PrepareResourceAsync(string label, Resource res, bool custom, bool remote, ResourceArgs args, ResourceOptions options) in /workspaces/pulumi/sdk/dotnet/Pulumi/Deployment/Deployment_Prepare.cs:line 80
       at async Task<(string urn, string id, Struct data, ImmutableDictionary<string, ImmutableHashSet<Resource>> dependencies)> Pulumi.Deployment.RegisterResourceAsync(Resource resource, bool remote, Func<string, Resource> newDependency, ResourceArgs args, ResourceOptions options) in /workspaces/pulumi/sdk/dotnet/Pulumi/Deployment/Deployment_RegisterResource.cs:line 27
       at async Task<(string urn, string id, Struct data, ImmutableDictionary<string, ImmutableHashSet<Resource>> dependencies)> Pulumi.Deployment.ReadOrRegisterResourceAsync(Resource resource, bool remote, Func<string, Resource> newDependency, ResourceArgs args, ResourceOptions options) in /workspaces/pulumi/sdk/dotnet/Pulumi/Deployment/Deployment_ReadOrRegisterResource.cs:line 73
       at async Task Pulumi.Deployment.CompleteResourceAsync(Resource resource, bool remote, Func<string, Resource> newDependency, ResourceArgs args, ResourceOptions options, ImmutableDictionary<string, IOutputCompletionSource> completionSources) in /workspaces/pulumi/sdk/dotnet/Pulumi/Deployment/Deployment_ReadOrRegisterResource.cs:line 89
       at async void Pulumi.Deployment+Runner+<>c__DisplayClass13_0.<WhileRunningAsync>g__HandleCompletion|0(?)+HandleCompletion(?) in /workspaces/pulumi/sdk/dotnet/Pulumi/Deployment/Deployment.Runner.cs:line 141
       at async Task<T> Pulumi.Output<T>.GetValueAsync() in /workspaces/pulumi/sdk/dotnet/Pulumi/Core/Output.cs:line 153
       at async Task<int> Pulumi.Deployment+Runner.WhileRunningAsync() in /workspaces/pulumi/sdk/dotnet/Pulumi/Deployment/Deployment.Runner.cs:line 181
       at async Task Pulumi.Deployment.RegisterResourceOutputsAsync(Resource resource, Output<IDictionary<string, object>> outputs) in /workspaces/pulumi/sdk/dotnet/Pulumi/Deployment/Deployment_RegisterResourceOutputs.cs:line 28
       at async void Pulumi.Deployment+Runner+<>c__DisplayClass13_0.<WhileRunningAsync>g__HandleCompletion|0(?)+HandleCompletion(?) x 2 in /workspaces/pulumi/sdk/dotnet/Pulumi/Deployment/Deployment.Runner.cs:line 141

gitfool on Jul 17, 2021

Understood… certainly anything you can do is appreciated but not required 🙏

I’ll ping here if I get to fixing the pulumi/pulumi-dotnet#32 that I suspect might be related.

t0yv0 on Dec 15, 2022