build-push-action: GAR error "cannot reuse body, request must be retried"

Troubleshooting

Before submitting a bug report please read the Troubleshooting doc.

  • I have read TROUBLESHOOTING.md.

I wasn’t sure from the “Cannot push to a registry” section whether your preference is for issues to be opened here or in the linked tools’ repos.

Behaviour

Pushing image to GAR sometimes fails with error:

ERROR: failed commit on ref "manifest-sha256:0f17d0ea56a824cc2efa0ad1ded3e1dc540cdb3d0468feebb0b5506853c90f82": cannot reuse body, request must be retried

Steps to reproduce this issue

  1. Make and push a commit
  2. See build stage of build-push-action@v2 succeed
  3. See push stage of build-push-action@v2 fail with above error

It seems only to happen for commits that only touch files excluded from the Docker build context. However, the action does occasionally succeed this type of commit.

Expected behaviour

Push stage should succeed.

Actual behaviour

Push stage fails with error.

Configuration

name: CI/CD Pipeline

on:
  push:

jobs:
  publish-docker-image:
    if: github.ref == 'refs/heads/master'
    name: Publish Docker image
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
        with:
          version: latest
          buildkitd-flags: --debug

      # Caching strategy from: https://github.com/docker/build-push-action/blob/master/docs/advanced/cache.md#github-cache
      - name: Cache Docker layers for ${{ matrix.service }}
        uses: actions/cache@v2
        with:
          path: /tmp/.buildx-cache
          key: ${{ runner.os }}-buildx-fly-shortener-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-buildx-fly-shortener-

      - name: Get image tags
        id: image-tags
        run: |
          export GIT_SHA="${{ github.sha }}"
          export GIT_SHA_SHORT=${GIT_SHA:0:7}
          echo "::set-output name=sha::$GIT_SHA_SHORT"

      - name: Login to GAR
        uses: docker/login-action@v1
        with:
          registry: us-east4-docker.pkg.dev
          username: _json_key
          password: ${{ secrets.GCR_JSON_KEY }}

      - name: Build and push ${{ matrix.service }}
        id: docker_build
        uses: docker/build-push-action@v2
        with:
          push: true
          tags: |
            us-east4-docker.pkg.dev/assemble-services/apps/link-shortener:latest
            us-east4-docker.pkg.dev/assemble-services/apps/link-shortener:${{ steps.image-tags.outputs.sha }}
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,dest=/tmp/.buildx-cache-new

      # Temp fix
      # https://github.com/docker/build-push-action/issues/252
      # https://github.com/moby/buildkit/issues/1896
      - name: Move cache
        run: |
          rm -rf /tmp/.buildx-cache
          mv /tmp/.buildx-cache-new /tmp/.buildx-cache

Logs

1_Set up job.txt 2_Checkout.txt 3_Set up Docker Buildx.txt 4_Cache Docker layers for.txt 5_Get image tags.txt 6_Login to GAR.txt 7_Build and push.txt 12_Post Build and push.txt 13_Post Login to GAR.txt 15_Post Set up Docker Buildx.txt 16_Post Checkout.txt 17_Complete job.txt

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 17 (3 by maintainers)

Commits related to this issue

Most upvoted comments

Hi, I’ve been struggling with this problem and the only solution I’ve found is to add a label in docker buildx command. Example:

docker --context $DOCKER_CONTEXT buildx build  --platform linux/amd64,linux/arm64 -t $CONTAINER_BASE_IMAGE --push --label "git-commit=$CI_COMMIT_SHA" .

In my case I was having the problem in 2 different pipelines that didn’t touch the docker build context. (first one worked and second one failed)

Maybe you’ll need to change CI_COMMIT_SHA to something like a timestamp.

Hope this helps!

Happened just now, “Re run all jobs” is good enough

I also see this with Gituhub container archive, so it’s not just google; also on about 50% of runs. We are adding labels with the docker/metadata-action@v3 and it’s no help; I made sure to craft a label that guarantees uniqueness in case it’s some sort of collision.

name:                   tooklibs core Docker Image Build

concurrency:
  group:                        mearth-${{ github.workflow }}-${{ github.sha }}-docker_tooklibs_core
  cancel-in-progress:           true

on:
  workflow_call:
    secrets:
      AGENTTOKS_DOPTOK: { required: true }
    inputs:
      shatag:           { required: true, type: string }

env:
  REGISTRY:             ghcr.io
  IMAGE_NAME:           tookstock/tooklibs
  TOOKLIBS_SEED_IMAGE:  ghcr.io/tookstock/tooklibs:latest-seed

jobs:

  build:
    runs-on:            ubuntu-latest
    container:          ghcr.io/tookstock/buildbox:latest-base
    permissions:
      contents:         read
      packages:         write
      id-token:         write

    steps:

      - name:           Checkout repository
        uses:           actions/checkout@v3

      - name:                   Stub config
        shell:                  bash
        run: |
          git config --global --add safe.directory /__w/mearth/mearth || true
          yarn tk:fakesecs

      - name:                   Typescript
        shell:                  bash
        run: |
          yarn tsc --build --verbose

      - name:                   Agent Tokens
        env:
          DOPPLER_TOKEN:        ${{ secrets.AGENTTOKS_DOPTOK }}
          STAGE:                gh
          REALM:                took
        run: |
          DOPPLER_CONFIG=took_gh yarn tk:agenttoks

      - name:           Setup Docker buildx
        uses:           docker/setup-buildx-action@dc7b9719a96d48369863986a06765841d7ea23f6

      - name:           Log into registry ${{ env.REGISTRY }}
        if:             github.event_name != 'pull_request'
        uses:           docker/login-action@49ed152c8eca782a232dede0303416e8f356c37b
        with:
          registry:     ${{ env.REGISTRY }}
          username:     ${{ github.actor }}
          password:     ${{ secrets.GITHUB_TOKEN }}

      - name:           Extract Docker metadata
        id:             meta
        uses:           docker/metadata-action@69f6fc9d46f2f8bf0d5491e4aabe0bb8c6a4678a
        with:
          labels: |
            boxbranch=${{ env.IMAGE_NAME }}-core-${{ inputs.mearthbranch }}
            uniquely=${{ env.IMAGE_NAME }}-core-${{ inputs.shatag }}-${{ inputs.txbx_dt }}
            shatag=${{ inputs.shatag }}
            whim=${{ inputs.txbx_whim }}
            dt=${{ inputs.txbx_dt }}
          images: |
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=ref,event=branch,suffix=-core
            type=ref,event=pr,suffix=-core
            type=sha,suffix=-core

      - name:           Build and push Docker image
        id:             build-and-push
        uses:           docker/build-push-action@e551b19e49efd4e98792db7592c17c09b89db8d8
        with:
          build-args: |
            TOOKLIBS_SEED_IMAGE=${{ env.TOOKLIBS_SEED_IMAGE }}
          context:      .
          file:         Dockerfile-tooklibs-core
          cache-from:   type=gha
          cache-to:     type=gha,mode=max
          push:         ${{ github.event_name != 'pull_request' }}
          tags:         ${{ steps.meta.outputs.tags }}
          labels:       ${{ steps.meta.outputs.labels }

Another workaround for this is adding labels with timestamps using docker’s metadata-action https://github.com/docker/metadata-action

This was based on the @sanchezpaco comment

jobs:
  docker:
    runs-on: ubuntu-latest
    steps:
      -
        name: Checkout
        uses: actions/checkout@v2
      -
        name: Docker meta
        id: meta
        uses: docker/metadata-action@v3
        with:
          images: name/app
      -
        name: Login to DockerHub
        if: github.event_name != 'pull_request'
        uses: docker/login-action@v1
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      -
        name: Build and push
        uses: docker/build-push-action@v2
        with:
          context: .
          push: ${{ github.event_name != 'pull_request' }}
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha,mode=max