pulumi: Pulumi is stuck when running on GitHub Actions

What happened?

A simple pulumi preview command is stuck when running on GitHub actions. It is stuck even with no changes in the resources. Other commands such as pulumi up are also stuck.

The bug happens intermittently, so it is difficult to reproduce.

Always works fine when running on my machine (macOS). I tried to run on macOS in github actions but got the same problem.

Perhaps related to https://github.com/pulumi/pulumi/issues/6609

Steps to reproduce

Create a stack using

• pulumi 3.51.1
• s3 as stack backend
• github plugin github v5.1.0
• Around ~2k resources. We had ~4k resources before. I was hoping the number of resources would influence the bug, so I deleted a couple that could be deleted.
• Run a simple pulumi preview on GitHub actions, running on a ubuntu-20.04. You can apply the resources in your machine before running in a way you expect the preview to execute very fast.

Intermittently, the preview would be stuck forever.

Expected Behavior

The pulumi preview should not be stuck. It should run very fast.

Actual Behavior

The pulumi preview is taking forever to run. It lasted 5 hours in a workflow which we didn’t configure timeout.

Output of pulumi about

CLI          
Version      3.51.1
Go Version   go1.19.4
Go Compiler  gc

Plugins
NAME    VERSION
github  5.1.0
nodejs  unknown

Host     
OS       darwin
Version  12.6
Arch     arm64

This project is written in nodejs: executable='/Users/vinicius.vargas/.nvm/versions/node/v16.17.1/bin/node' version='v16.17.1'

Current Stack: loft-br.backstage-resources.gitsyncer

TYPE                                        URN
pulumi:pulumi:Stack                         urn:pulumi:loft-br.backstage-resources.gitsyncer::gitsyncer::pulumi:pulumi:Stack::gitsyncer-loft-br.backstage-resources.gitsyncer
pulumi:providers:github                     urn:pulumi:loft-br.backstage-resources.gitsyncer::gitsyncer::pulumi:providers:github::loft-br
github:index/team:Team                      urn:pulumi:loft-br.backstage-resources.gitsyncer::gitsyncer::github:index/team:Team::replaced-name
github:index/membership:Membership          urn:pulumi:loft-br.backstage-resources.gitsyncer::gitsyncer::github:index/membership:Membership::replaced@replaced.com
github:index/teamRepository:TeamRepository  urn:pulumi:loft-br.backstage-resources.gitsyncer::gitsyncer::github:index/teamRepository:TeamRepository::replaced-name
github:index/teamMembership:TeamMembership  urn:pulumi:loft-br.backstage-resources.gitsyncer::gitsyncer::github:index/teamMembership:TeamMembership::replaced-name

-- ATENTION: the number of resources above is way larger (around ~2k). I removed then because of the size limit on github issue body. --

Found no pending operations associated with loft-br.backstage-resources.gitsyncer

Backend        
Name           LOFT-M3786
URL            s3://loft-iac-replaced/.cloudify
User           vinicius.vargas
Organizations  

Pulumi locates its logs in /var/folders/m5/xwz1r9155jg2lcjlwqrh6p540000gq/T/ by default

Additional context

As you can guess from pulumi about, we use this stack to automate managing our github membership and team access to repositories. The resources are interpreted from .yaml files located at the same repo.

Everything was working fine since 2023/08 until last wednesday (2023/01/11).

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you’ve opened one already).