pulumi: failed to compute archive hash for "code": archive/tar: missed writing 2026611 bytes

What happened?

I am managing my AWS Lambda function using Pulumi, using pulumi.asset.AssetArchive to provide list of files to upload.

Two to three weeks ago, I started to observe problems when running pulumi up locally on my macOS M1 machine. The archive is uploaded to AWS, but some of the larger JavaScript files are truncated in the middle. (The file sizes are between 500KB to 5MB). Fortunately, this problem was not happening when running pulumi up from GitHub Actions.

Occasionally, I was receiving the following error on my machine, but again not in GitHub Actions:

Formatted for clarity:

Diagnostics:
  pulumi:pulumi:Stack (my-stack-name):
    error: Running program '/home/runner/work/my-project/my-project/infra'
      failed with an unhandled exception:
    Error: failed to register new resource 
      my-api-handler [aws:lambda/function:Function]: 2 
      UNKNOWN: failed to compute archive hash for "code": 
      archive/tar: missed writing 2026611 bytes

Full error:

Diagnostics:
  pulumi:pulumi:Stack (my-stack-name):
    error: Running program '/home/runner/work/my-project/my-project/infra' failed with an unhandled exception:
    Error: failed to register new resource my-api-handler [aws:lambda/function:Function]: 2 UNKNOWN: failed to compute archive hash for "code": archive/tar: missed writing 2026611 bytes
        at Object.registerResource (/home/runner/work/my-project/my-project/node_modules/.pnpm/@pulumi+pulumi@3.33.2/node_modules/@pulumi/runtime/resource.ts:295:27)
        at new Resource (/home/runner/work/my-project/my-project/node_modules/.pnpm/@pulumi+pulumi@3.33.2/node_modules/@pulumi/resource.ts:393:13)
        at new CustomResource (/home/runner/work/my-project/my-project/node_modules/.pnpm/@pulumi+pulumi@3.33.2/node_modules/@pulumi/resource.ts:772:9)
        at new Function (/home/runner/work/my-project/my-project/node_modules/.pnpm/@pulumi+aws@5.6.0/node_modules/@pulumi/lambda/function.ts:505:9)
        at new ZebraDashboardApiLambda (file:///home/runner/work/my-project/my-project/infra/dashboard-lambda.js:88:20)
        at new ZebraTenant (file:///home/runner/work/my-project/my-project/infra/tenant.js:129:31)
        at file:///home/runner/work/my-project/my-project/infra/infra.js:119:19
        at processTicksAndRejections (node:internal/process/task_queues:96:5)

Today, I started to get the error “missed writing N bytes” in GitHub Actions too.

Steps to reproduce

Am I the only person encountering this issue? If the log message does not offer any hints on what can be wrong, then I’ll trim down our project into an example that can be posted in public.

High-level steps to reproduce:

1. Create a new Pulumi project in JavaScript
2. Write a JavaScript lambda handler depending on pg and undici, run esbuild to create the source code file for AWS, configure Pulumi’s Lambda function code property using pulumi.asset.AssetArchive (see https://bajtos.net/posts/2022-05-bundling-nodejs-for-aws-lambda/)
3. Run pulumi up

Expected Behavior

Pulumi packs all provided assets into an archive and uploads the archive to AWS. The archive contains the full file content, nothing is truncated.

Actual Behavior

Sometimes, during the preview phase, I get the error failed to compute archive hash for "code": archive/tar: missed writing 2026611 bytes.

In other times, the Lambda code is updated and Lambda invocations fails at runtime.

ERROR Uncaught Exception 

{
  "errorType":"Runtime.UserCodeSyntaxError",
  "errorMessage": "SyntaxError: Unexpected end of input",
  "stack":[
    "Runtime.UserCodeSyntaxError: SyntaxError: Unexpected end of input",
    " at _loadUserApp (file:///var/runtime/index.mjs:724:17)",
    " at async Object.module.exports.load (file:///var/runtime/index.mjs:741:21)",
    " at async file:///var/runtime/index.mjs:781:15",
    " at async file:///var/runtime/index.mjs:4:1"
  ]
}

Versions used

CLI
Version      3.33.2
Go Version   go1.18.2
Go Compiler  gc

Plugins
NAME    VERSION
nodejs  unknown

Host
OS       darwin
Version  12.4
Arch     arm64

This project is written in nodejs (/usr/local/bin/node v16.15.0)

I am using pnpm which seems to be unsupported by pulumi about:

warning: Failed to get information about the Pulumi program's plugins: Could not find either (...)/yarn.lock or (...)/package-lock.json

Here is the output of pnpm ls "@pulumi/*":

dependencies:
@pulumi/aws 5.6.0
@pulumi/awsx 0.40.0
@pulumi/postgresql 3.4.0
@pulumi/pulumi 3.33.2
@pulumi/random 4.7.0

Additional context

Everything used to work fine for many months. The problems started 2-3 weeks ago. I am not aware of any change that would trigger the new behavior. When I observed the problem for the first time, I upgraded all Pulumi dependencies to the latest versions, but the problem persisted.

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you’ve opened one already).