pulumi-azure: 'could not load plugin for azure provider' 'failed to locate compatible plugin'.

Also worth noting is that, although this bug masqueraded as an issue with our new plugin load behavior (and PULUMI_ENABLE_LEGACY_PLUGIN_SEARCH), this is actually not a new bug and both the old and new plugin loading behaviors are equally broken.

This is fixed with https://github.com/pulumi/pulumi/pull/2560 which is part of the 0.17.2 release.

Thanks to @adaniline-traderev’s example on slack (thank you!) I had a moment of inspiration and here’s what’s going on here and here is a set of repro steps.

1. Start with this program:

import * as aws from "@pulumi/aws";

const b = new aws.s3.Bucket("bucket");
export const name = b.id;

{
    "name": "aws-typescript",
    "devDependencies": {
        "@types/node": "latest"
    },
    "dependencies": {
        "@pulumi/pulumi": "0.17.0",
        "@pulumi/aws": "0.17.0"
    }
}

1. Create a stack and initialize config:

$ pulumi stack init dev
Created stack 'dev'
$ pulumi config set aws:region us-west-2

1. yarn install

$ yarn install
warning package.json: No license field
info No lockfile found.
warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.
warning aws-typescript: No license field
[1/4] 🔍   Resolving packages...
[2/4] 🚚   Fetching packages...
[3/4] 🔗   Linking dependencies...
[4/4] 🔨   Building fresh packages...
success Saved lockfile.
✨  Done in 16.38s.

1. Observe that a plugin did get installed, at version 0.17.0

$ pulumi plugin ls
NAME  KIND      VERSION  SIZE    INSTALLED       LAST USED
aws   resource  0.17.0   156 MB  28 seconds ago  28 seconds ago

TOTAL plugin cache size: 156 MB

1. Pulumi update:

▶ pulumi update
Previewing update (dev):

     Type                 Name      Plan
 +   pulumi:pulumi:Stack  node-dev  create
 +   └─ aws:s3:Bucket     bucket    create

Resources:
    + 2 to create

Do you want to perform this update? yes
Updating (dev):

     Type                 Name      Status
 +   pulumi:pulumi:Stack  node-dev  created
 +   └─ aws:s3:Bucket     bucket    created

Outputs:
    name: "bucket-4d0d87a"

Resources:
    + 2 created

Duration: 5s

Permalink: https://app.pulumi.com/swgillespie/node/dev/updates/1

1. Upgrade to version 0.17.1 of @pulumi/pulumi and @pulumi/aws:

7,8c7,8
<         "@pulumi/aws": "0.17.0",
<         "@pulumi/pulumi": "0.17.0"
---
>         "@pulumi/aws": "0.17.1",
>         "@pulumi/pulumi": "0.17.1"

$ yarn install

1. Observe that yarn installed v0.17.1 of the aws plugin:

NAME  KIND      VERSION  SIZE    INSTALLED       LAST USED
aws   resource  0.17.1   156 MB  24 seconds ago  24 seconds ago
aws   resource  0.17.0   156 MB  3 minutes ago   now

TOTAL plugin cache size: 312 MB

1. Pulumi update

$ pulumi update
Previewing update (dev):

     Type                 Name      Plan
     pulumi:pulumi:Stack  node-dev

Resources:
    2 unchanged

Do you want to perform this update? yes
Updating (dev):

     Type                 Name      Status
     pulumi:pulumi:Stack  node-dev

Outputs:
    name: "bucket-4d0d87a"

Resources:
    2 unchanged

Duration: 2s

Permalink: https://app.pulumi.com/swgillespie/node/dev/updates/2

1. Verify that v0.17.1 of the aws provider is in the snapshot:

$ pulumi stack export | jq '.deployment.resources[1].inputs.version'
"0.17.1"

1. Downgrade back to v0.17.0 of the aws and pulumi packages:

7,8c7,8
<         "@pulumi/aws": "0.17.1",
<         "@pulumi/pulumi": "0.17.1"
---
>         "@pulumi/aws": "0.17.0",
>         "@pulumi/pulumi": "0.17.0"

1. Delete all plugins:

$ pulumi plugin rm --all

1. Yarn install

$ yarn install
yarn install v1.13.0
warning package.json: No license field
warning aws-typescript: No license field
[1/4] 🔍   Resolving packages...
[2/4] 🚚   Fetching packages...
[3/4] 🔗   Linking dependencies...
[4/4] 🔨   Building fresh packages...
✨  Done in 16.56s.

1. The program is now hosed:

$ pulumi preview
Previewing update (dev):
Permalink: https://app.pulumi.com/swgillespie/node/dev/previews/6d827635-014d-44d1-8a62-8347261d5b65
error: could not load plugin for aws provider 'urn:pulumi:dev::node::pulumi:providers:aws::default': no resource plugin 'aws-v0.17.1' found in the workspace or on your $PATH, install the plugin using `pulumi plugin install resource aws v0.17.1`

1. Observe that running pulumi plugin install does not help:

$ pulumi plugin install --verbose
[resource plugin aws-0.17.0] installing
[resource plugin aws-0.17.0] skipping install (existing >= match)

$ pulumi preview
Previewing update (dev):
Permalink: https://app.pulumi.com/swgillespie/node/dev/previews/3041658b-f5b0-4886-a7b1-3ab6090865fc
error: could not load plugin for aws provider 'urn:pulumi:dev::node::pulumi:providers:aws::default': no resource plugin 'aws-v0.17.1' found in the workspace or on your $PATH, install the plugin using `pulumi plugin install resource aws v0.17.1`

I believe this repro covers everyone that’s posted about having issues in this thread. Step 11 and step 12 simulate being in CI environments, which is why I believe that everyone has seen this trouble in their CI environments while observing that they have no problems when run locally.

Workaround

If you are in this situation, you can run the command present in the error message and it will install the missing plugin:

$ pulumi plugin install resource aws v0.17.1
[resource plugin aws-0.17.1] installing
Downloading plugin:  43.14 MiB / 43.14 MiB [========================] 100.00% 9s


$ pulumi preview
Previewing update (dev):

     Type                 Name      Plan
     pulumi:pulumi:Stack  node-dev

Resources:
    2 unchanged

Permalink: https://app.pulumi.com/swgillespie/node/dev/previews/f8573b59-bc59-4fa8-ade7-9ee17d4d1d1c

Given that this most often happens in CI scenarios, a reasonable workaround for now would be to put this command in your CI script until Pulumi delivers a fix.

To anyone else having this issue:

You can set the environment variable PULUMI_ENABLE_LEGACY_PLUGIN_SEARCH=1 to get unblocked and load plugins exactly as before.