playwright: [Bug]: Unable to extract large report.zip

I tried to reproduce with this workflow, but was not able to: https://github.com/mxschmitt/test/actions/runs/8397555420/job/23001044584

Some recent investigation on our end showed that this might be caused by the following:

• Run gets cancelled
• npx playwright test did not finish writing all the blobs
• if: always() kicks in and uploads a broken blob
• From there on things don’t work anymore.

as per here we should change

if: always()

to

if: ${{ !cancelled() }}

Which should fix this issue. Would appreciate if you could try testing it before we roll it out across docs/create-playwright. Its a very early investigation, so haven’t even tried it but looks promising. Thanks!

Maybe related to https://github.com/actions/download-artifact/issues/298

error is thrown here, in openReadStream, in playwright-core/lib/zipBundleImpl.js:

I have run into this exact issue, after upgrading to using upload/download-artifacts v4 and using merge-reports. I’m not able to share more code or the artifacts themselves.

These are the artifacts generated:

The workflow file runs a very specific scenario, where I shard into multiple jobs depending on a spec list (array of strings containing the paths to the spec), and then run max-parallel of 1, to guarantee that each block of specs runs sequentially, which contains the following steps:

jobs:
  test:
    timeout-minutes: 1440
    runs-on: self-hosted
    strategy:
      fail-fast: false
      max-parallel: 1
      matrix:
        shard: ${{ fromJSON(github.event.inputs.specs) }}
    steps:
    - uses: actions/checkout@v4
    - name: Setup Node
      uses: actions/setup-node@v4
      with:
        node-version: 20
    - name: Install dependencies
      run: npm ci
    - name: Install Playwright Browsers
      run: npx playwright install --with-deps
    - name: Run Playwright tests
      run: npx playwright test ${{ matrix.shard }} -c playwright.config.ts
    - name: Get random number to use in report
      if: always()
      id: generate_number
      run: echo "random_number=$(echo $RANDOM)" >> $GITHUB_OUTPUT
      shell: bash
    - name: Upload report blob
      uses: actions/upload-artifact@v4
      if: always()
      with:
        name: blob-reports-${{ steps.generate_number.outputs.random_number }}
        path: blob-report
        retention-days: 1
  merge-reports:
    runs-on: self-hosted
    if: always()
    needs: [test]
    steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-node@v4
      with:
        node-version: 20
    - name: Install dependencies
      run: npm ci
    - name: Download blob reports from GitHub Actions Artifacts
      uses: actions/download-artifact@v4
      with:
        pattern: blob-reports-*
        path: all-blob-reports
        merge-multiple: true
    - name: Merge into HTML Report
      run: npx playwright merge-reports --reporter html ./all-blob-reports # outputs to playwright-report folder
  ...

test job will run a matrix, that generates N blob-reports-RANDOM_NUMBER artifacts. merge-reports then downloads the artifacts with that pattern, with merge-multiple enabled.

But merging fails, with the exact same problem as other users mentioned:

I read that this error might occur when trying to unzip something that is not a zip file.

Downloading the files locally, extracting them into the all-blop-reports folder while renaming all of the “report.zip” files to “report.zip”, “report (1).zip”, “report (2).zip” and so on, then running npx playwright merge-reports --reporter HTML ./all-blob-reports works, just does not work in CI:

• downloaded artifacts:

• extracted report.zip files:

• merging reports, report working just fine locally:

These artifact sizes are normal, have opened reports before that were 3x their accumulated sizes.

While looking at https://github.com/microsoft/playwright/compare/v1.40.0...v1.41.2 it could be caused by dc8ecc3ca404b211815c5541dfea6e59dbd19b9a.

@jame-earnin how large is your zip / project? Would it be possible to share a reproduction?