pulumi: Grpc.Core.RpcException: "Failed to deserialize response message." when using "cert-manager.crds.yaml" for ConfigFile

When using the following code in my C# Pulumi project:

var customResourceDefinitions = new ConfigFile("cert-manager-crds", new ConfigFileArgs
{
    File = "https://github.com/jetstack/cert-manager/releases/download/v0.14.0/cert-manager.crds.yaml"
});

and do pulumi up, the process fails with:

Diagnostics:
  pulumi:pulumi:Stack (ya-infra-shared-dev):
    error: Running program 'C:\Projects\Ya\Shared\infra\ya-infra-shared\bin\Debug\netcoreapp3.1\Ya.Infra.Shared.dll' failed with an unhandled exception:
    Grpc.Core.RpcException: Status(StatusCode=Internal, Detail="Failed to deserialize response message.")
       at Pulumi.GrpcMonitor.InvokeAsync(InvokeRequest request)
       at Pulumi.Deployment.InvokeAsync[T](String token, InvokeArgs args, InvokeOptions options, Boolean convertResult)
       at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
       at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
       at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
       at Pulumi.Output`1.Pulumi.IOutput.GetDataAsync()
       at Pulumi.Serialization.Serializer.SerializeAsync(String ctx, Object prop)
       at Pulumi.Deployment.SerializeFilteredPropertiesAsync(String label, IDictionary`2 args, Predicate`1 acceptKey)
       at Pulumi.Deployment.SerializeAllPropertiesAsync(String label, IDictionary`2 args)
       at Pulumi.Deployment.RegisterResourceOutputsAsync(Resource resource, Output`1 outputs)
       at Pulumi.Deployment.Runner.WhileRunningAsync()

I’m trying to automate the installation of cert-manager as described here. One of its first steps mentions I should install some custom resource definitions separately.

Perhaps I should be using a different class to get the cert-manager.crds.yaml applied?

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 3
Comments: 24 (18 by maintainers)

Commits related to this issue

Update pulumi deps to v2.14.0 This release fixes https://github.com/pulumi/pulumi/issues/4224 where users were unable to install large CRDs using .NET. — committed to pulumi/pulumi-kubernetes by lblackstone 4 years ago

Most upvoted comments

Here is the draft of the forking plan:

Fork protocolbuffers/protobuf to Pulumi org (done): https://github.com/pulumi/protobuf
Checkout the latest stable tag (v3.13.0.1 as of today).
Create a pulumi-fork branch.
Adjust DefaultRecursionLimit in two spots from 100 to 1000.

Adjust the csproj file to something like

<GeneratePackageOnBuild>true</GeneratePackageOnBuild>
<Description>Pulumi fork to override DefaultRecursionLimit - C# runtime library for Protocol Buffers - Google's data interchange format.</Description>
<GeneratePackageOnBuild>true</GeneratePackageOnBuild>

Build the package.
Publish it under Pulumi organization in NuGet as Pulumi.Protobuf.
Update the reference in pulumi/pulumi to the fork.
Publish a new version of Pulumi NuGet.
Update Pulumi.Kubernetes to use it.

I ran this sequence locally, without any publishing, and it does fix the original problem. I am able to deploy the CRDs from https://github.com/jetstack/cert-manager/releases/download/v0.14.0/cert-manager.crds.yaml.

Any concerns about this proposal, technically or from the IP perspective?

mikhailshilkov on Nov 9, 2020

https://github.com/pulumi/pulumi-kubernetes/releases/tag/v2.7.2 is out now with the fix

lblackstone on Nov 19, 2020

Thanks to everyone for waiting for so long: it took a while to investigate the issue, find an issue in the Protobuf .NET library, discuss it with maintainers, and finally come up with a workaround.

The issue is now fixed in the latest alpha release (dotnet add package Pulumi.Kubernetes --version 2.8.0-alpha.1605734708) and will be shipped in the next release. cc @lblackstone - we may want to cut ~2.8.1~ 2.7.2 whenever that works for you.

mikhailshilkov on Nov 19, 2020

The issue in gRPC: https://github.com/grpc/grpc/issues/22682

mikhailshilkov on Apr 16, 2020

I also want to thank @mikhailshilkov for implementing the work around for this unfortunate issue with dotnet grpc! ❤️

gitfool on Dec 9, 2022

Thanks for sorting this @mikhailshilkov I can confirm it works as expected with the Alpha release, looking forward to getting it in once 2.8.1 is out!

sam-cogan on Nov 19, 2020

@gitfool

minimal repro yaml has 32 nested elements What am I missing?

So, for every layer of yaml, ReadMessage method would be called three times: once with message being a Struct, once with message being MapField<string, Value>, and once with Value. I guess that’s just how untyped dictionaries are represented in protobuf.

mikhailshilkov on Oct 30, 2020

I’ve got the same problem using dotnet/c# with cert manager. This is a showstopper! 😬

gitfool on Dec 9, 2022