pulumi-aws: Error running pulumi-aws in CI environment when assuming a role for credentials

Hi all

Based on the investigation work by @DavidHe1127 and @leezen, I was able to find that there was indeed a bug in named providers that stopped values being used that were not the default

I was able to fix that in https://github.com/pulumi/pulumi/pull/6496

This work has made it’s way down to the pulumi-aws provider as v3.32.1 and we believe this now means that you can set your values as expected and you should be able to authenticate in an EC2 environment with the iam role.

Your code would need to be something like this:

const prov = new aws.Provider("named-provider", {
    region: "us-west-2",
    skipMetadataApiCheck: false,
    skipGetEc2Platforms: false,
});

and it will pick up the creds

I am going to close this issue out BUT IF THERE ARE ANY FURTHER ANOMOLIES then please do comment and we can reopen and investigate further if needed

Thanks again for your patience here and we can’t thank you enough for helping us track down this bug

Paul

Hi @dferretti

If these config values work for you, then I will also add a guide to the README to suggest ### Running Pulumi-Aws in CI environments so that others don’t get bitten

P.

That is a bug! Great catch @hbollon - I will open an issue!

Hi @stack72 Little follow-up for our problem with @yann-soubeyrand With pulumi.Bool values we can’t set Skip* attributes of the aws.ProviderArgs struct because they are overridden by default values according to the pulumi preview output. For exemple with:

f.awsProvider, err = aws.NewProvider(ctx, "main",
		&aws.ProviderArgs{
			SkipMetadataApiCheck:      pulumi.Bool(false),
			SkipCredentialsValidation: pulumi.Bool(true),
			Region:                    args.Region,
		},
[...]

We have this:

6:20PM INFO  actions.preview._exec | #11 95.88         + pulumi:providers:aws: (create)
6:20PM INFO  actions.preview._exec | #11 95.88             [urn=urn:pulumi:dev::schweizmobil::foundations:Foundations$pulumi:providers:aws::main]
6:20PM INFO  actions.preview._exec | #11 95.88             region                   : "eu-west-1"
6:20PM INFO  actions.preview._exec | #11 95.88             skipCredentialsValidation: true
6:20PM INFO  actions.preview._exec | #11 95.88             skipGetEc2Platforms      : true
6:20PM INFO  actions.preview._exec | #11 95.88             skipMetadataApiCheck     : true
6:20PM INFO  actions.preview._exec | #11 95.88             skipRegionValidation     : true
6:20PM INFO  actions.preview._exec | #11 95.88             version                  : "5.10.0"

However, if we use pulumi.BoolPtr instead we have the expected result:

f.awsProvider, err = aws.NewProvider(ctx, "main",
		&aws.ProviderArgs{
			SkipMetadataApiCheck:      pulumi.BoolPtr(false),
			SkipCredentialsValidation: pulumi.BoolPtr(true),
			Region:                    args.Region,
		},
[...]

7:26AM INFO  actions.preview._exec | #11 105.2         + pulumi:providers:aws: (create)
7:26AM INFO  actions.preview._exec | #11 105.2             [urn=urn:pulumi:dev::schweizmobil::foundations:Foundations$pulumi:providers:aws::main]
7:26AM INFO  actions.preview._exec | #11 105.2             region                   : "eu-west-1"
7:26AM INFO  actions.preview._exec | #11 105.2             skipCredentialsValidation: true
7:26AM INFO  actions.preview._exec | #11 105.2             skipGetEc2Platforms      : true
7:26AM INFO  actions.preview._exec | #11 105.2             skipMetadataApiCheck     : false
7:26AM INFO  actions.preview._exec | #11 105.2             skipRegionValidation     : true
7:26AM INFO  actions.preview._exec | #11 105.2             version                  : "5.10.0"

My guess, after having taken a look at the code, is that you have some condition in the aws.NewProvider function which call IsZero utility func on those args (skip* and region) in order to set default values if none was provided:

if isZero(args.Region) {
   args.Region = pulumi.StringPtr(getEnvOrDefault("", nil, "AWS_REGION", "AWS_DEFAULT_REGION").(string))
}
if isZero(args.SkipCredentialsValidation) {
   args.SkipCredentialsValidation = pulumi.BoolPtr(false)
}
if isZero(args.SkipGetEc2Platforms) {
   args.SkipGetEc2Platforms = pulumi.BoolPtr(true)
}
if isZero(args.SkipMetadataApiCheck) {
   args.SkipMetadataApiCheck = pulumi.BoolPtr(true)
}
if isZero(args.SkipRegionValidation) {
   args.SkipRegionValidation = pulumi.BoolPtr(true)
}

And in this IsZero function you’re performing a test on the default value of the type of the argument:

// isZero is a null safe check for if a value is it's types zero value.
func isZero(v interface{}) bool {
	if v == nil {
		return true
	}
	return reflect.ValueOf(v).IsZero()
}

Knowing that, with a bool variable set to false as input it seems normal that this function return true and so override the arg (SkipMetadataApiCheck in our case) back to true. Is it an intended feature? In your exemples you also used pulumi.Bool and not the Ptr equivalent 🤔

We are seeing a slight variation of the error:

[2021-02-24T03:45:26Z] error: Preview failed: 1 error occurred:
--
  | [2021-02-24T03:45:26Z] 	* error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.
  | [2021-02-24T03:45:26Z]
  | [2021-02-24T03:45:26Z] Please see https://registry.terraform.io/providers/hashicorp/aws
  | [2021-02-24T03:45:26Z] for more information about providing credentials.
  | [2021-02-24T03:45:26Z]
  | [2021-02-24T03:45:26Z] Error: EC2RoleRequestError: no EC2 instance role found
  | [2021-02-24T03:45:26Z] caused by: RequestCanceled: EC2 IMDS access disabled via AWS_EC2_METADATA_DISABLED env var

With the

"@pulumi/aws": "3.28.1",
"@pulumi/awsx": "0.22.0",
"@pulumi/pulumi": "2.16.2",

I’m curious why this breaking change was made in a patch release