build-push-action: Possible incompatibility with AWS ECR

Troubleshooting

Before submitting a bug report please read the Troubleshooting doc.

Behaviour

Steps to reproduce this issue

Assuming you have the right permissions to push to an AWS ECR (in our case we do)

  1. Create the following action:
name: build-push

on:
  push:
    branches:
      - "test-branch"

env:
  TAG: test
  GH_TOKEN: ${{ github.token }}
  DOCKERFILE_PATH: ./Dockerfile
  IMAGE_NAME: test
  AWS_REGION: eu-west-1
  IAM_ROLE_ARN: arn:aws:iam::111111111111:role/ecr-role
  ECR: 111111111111.dkr.ecr.eu-west-1.amazonaws.com

jobs:
  build-push:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-region: ${{ env.AWS_REGION }}
          role-to-assume: ${{ env.IAM_ROLE_ARN }}
          role-session-name: test

      - name: Login to ECR
        uses: docker/login-action@v2
        with:
          registry: ${{ env.ECR }}

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Build and push container image
        if: github.event_name == 'push'
        uses: docker/build-push-action@v4
        with:
          context: .
          push: true
          file: ${{ env.DOCKERFILE_PATH }}
          tags: ${{ env.ECR }}/${{ env.IMAGE_NAME }}:${{ env.TAG }}
          # outputs: type=docker
    
      # The step below works with the configured credentials if the "outputs: type=docker" is uncommented
      - name: push manually
        run: docker push ${{ env.ECR }}/${{ env.IMAGE_NAME }}:${{ env.TAG }}
  1. Run the action.
  2. The above action returns the following error:
ERROR: failed to solve: failed to push 111111111111.dkr.ecr.eu-west-1.amazonaws.com/test:test: unexpected status: 403 Forbidden
Error: buildx failed with: ERROR: failed to solve: failed to push 111111111111.dkr.ecr.eu-west-1.amazonaws.com/test:test: unexpected status: 403 Forbidden

Expected behaviour

The docker/build-push-action@v4 should be able to upload the container image to the container registry.

Actual behaviour

The docker/build-push-action@v4 returns a 403 Forbidden error even when the action has the right credentials to push to the repository. That’s because setting push: false and using a separate docker push results in the image correctly pushed to the container registry.

Configuration

  • Repository URL (if public): Not public.
  • Build URL (if public): Not public.
name: build-push

on:
  push:
    branches:
      - "test-branch"

env:
  TAG: test
  GH_TOKEN: ${{ github.token }}
  DOCKERFILE_PATH: ./Dockerfile
  IMAGE_NAME: test
  AWS_REGION: eu-west-1
  IAM_ROLE_ARN: arn:aws:iam::111111111111:role/ecr-role
  ECR: 111111111111.dkr.ecr.eu-west-1.amazonaws.com

jobs:
  build-push:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-region: ${{ env.AWS_REGION }}
          role-to-assume: ${{ env.IAM_ROLE_ARN }}
          role-session-name: test

      - name: Login to ECR
        uses: docker/login-action@v2
        with:
          registry: ${{ env.ECR }}

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Build and push container image
        if: github.event_name == 'push'
        uses: docker/build-push-action@v4
        with:
          context: .
          push: true
          file: ${{ env.DOCKERFILE_PATH }}
          tags: ${{ env.ECR }}/${{ env.IMAGE_NAME }}:${{ env.TAG }}
          # outputs: type=docker
    
      # The step below works with the configured credentials if the "outputs: type=docker" is uncommented
      - name: push manually
        run: docker push ${{ env.ECR }}/${{ env.IMAGE_NAME }}:${{ env.TAG }}

Logs

Download the log file of your build and attach it to this issue.

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 7
  • Comments: 19 (4 by maintainers)

Most upvoted comments

While doing additional testing on the setup with AWS I noticed that if I updated the polices to allow the OIDC policy to perform ecr:* things started working for me.

We finally got it working using OIDC with this policy (some retractions made):

{
    "Statement": [
        {
            "Action": [
                "ecr:BatchGetImage",
                "ecr:BatchCheckLayerAvailability",
                "ecr:CompleteLayerUpload",
                "ecr:GetDownloadUrlForLayer",
                "ecr:InitiateLayerUpload",
                "ecr:PutImage",
                "ecr:UploadLayerPart"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:ecr:us-east-1:XXXXXXXXXXXX:repository/*"
            ],
            "Sid": "ecr"
        },
        {
            "Action": "ecr:GetAuthorizationToken",
            "Effect": "Allow",
            "Resource": "*",
            "Sid": "token"
        }
    ],
    "Version": "2012-10-17"
}

Note that we added the following compared to a policy that already worked for regular docker push.

"ecr:BatchGetImage",
"ecr:GetDownloadUrlForLayer",
+20
blockmar on May 3, 2023

Ok, the following is super weird.

If I use the following I get a 401 Unauthorized:

     - name: Configure AWS
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-region: ${{ env.AWS_REGION }}
          role-to-assume: ${{ env.IAM_ROLE_ARN }}
          role-session-name: test

      - name: Login to Amazon ECR
        uses: aws-actions/amazon-ecr-login@v1

      - name: Build and publish to Docker Hub and AWS ECR
        uses: docker/build-push-action@v3
        with:
          context: .
          file: ${{ env.DOCKERFILE_PATH }}
          tags: ${{ env.ECR }}/${{ env.IMAGE_NAME }}:${{ env.TAG }}
          push: true

But with the following I can push the images without any problem:

jobs:
  build-push-connector-container:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-region: ${{ env.AWS_REGION }}
          role-to-assume: ${{ env.IAM_ROLE_ARN }}
          role-session-name: test

      - name: Login to ECR
        uses: docker/login-action@v2
        with:
          registry: ${{ env.ECR }}

      # - name: Login to Amazon ECR
      #   id: login-ecr
      #   uses: aws-actions/amazon-ecr-login@v1

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        with:
          buildkitd-flags: --debug

      - name: Build and push container latest image
        uses: docker/build-push-action@v4
        with:
          push: false
          file: ${{ env.DOCKERFILE_PATH }}
          tags: ${{ env.ECR }}/${{ env.IMAGE_NAME }}:${{ env.TAG }}
          outputs: type=docker

      - name: Push manually
        run: docker push ${{ env.ECR }}/${{ env.IMAGE_NAME }}:${{ env.TAG }}

Do you have any idea of what to test? Definitely it is not a permissions issue whatsoever.

+5
jfagoagas on Mar 3, 2023

I’m still experiencing this issue, all actions are latest version. Commenting to highlight @halostatue is using user credentials (access and secret key), not OIDC (role-to-assume). Maybe that’s why it is working for them.

+2
CarlosBorroto on Apr 26, 2023

Ok, the following is super weird.

If I use the following I get a 401 Unauthorized:

(omitting your snippet for brevity)

Do you have any idea of what to test? Definitely it is not a permissions issue whatsoever.

I am experiencing the same thing with OCI-compliant (provenance:true) multiplatform images on self-hosted runners. And interestingly, not on all images (only really large ones?). Technically I can work around it by using type=oci, writing it out to a file, and in another step doing a docker import and docker push. But this will strip away CMD and ENTRYPOINT.

So for now type=docker and a subsequent push is the best workaround.

Here’s a gist I made for context.

What fails (same issue as yours)

The type=oci + import + push workaround

Mind you this isn’t even using the action. So it seems to be a buildx bug (or maybe some shortcoming on the ECR side?), rather than a build-push-action bug.

EDIT:

Your workaround won’t work for multiarch OCI-compliant images (see: https://github.com/docker/roadmap/issues/371)

...docker exporter does not currently support exporting manifest lists

SECOND EDIT (24/03/23): wanted to follow up on this:

A good workaround is to do outputs: type=image (which exports the image, including multiplatform OCI-compliant images to the buildx driver) and then push that image however you want. For me, podman manifest push works in all cases. But probably docker manifest push will work fine as well.

+2
korenyoni on Mar 24, 2023

I can confirm adding these two actions to the policy attached to the role we are assuming fixed the issue for us.

+1
CarlosBorroto on May 3, 2023
Read more comments on GitHub
← build-push-action: Not worked with ECR actions
build-push-action: raise `ghcr unexpected status: 403 Forbidden` when push ghcr image →