setup-buildx-action: setup-buildx-action hangs with 99% CPU when running on latest Github Action Runner v2.285.0
Behaviour
Our runners were auto-updated this morning from v2.284.0 to v2.285.0 and began to hang on the Download and Install Buildx step:
Steps to reproduce this issue
- Upgrade your runner to version v2.285.0
- Run the
docker/setup-buildx-action@v1in your workflow - You should see heavy CPU utilization that did not exist in v2.284.0
Expected behaviour
The step should complete in under 10s per our typical experience
Actual behaviour
The step is hanging indefinitely
Configuration
- Repository URL (if public): private
- Build URL (if public): private
Note: all steps prior to the failing step pass as expected
name: Test Environment
on:
pull_request:
types: [labeled, reopened, synchronize, ready_for_review]
workflow_dispatch:
env:
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
jobs:
build:
name: Build Image
runs-on: self-hosted
steps:
- name: Set outputs
id: vars
run: |
echo "::set-output name=head_sha::$(echo $PR_HEAD_SHA | cut -c1-7)"
env:
PR_HEAD_SHA: ${{ env.HEAD_SHA }}
- name: Login to DockerHub
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKER_HUB_USERNAME }}
password: ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }}
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: redacted
role-to-assume: redacted
- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v1
- name: Set up Docker Context for Buildx
id: buildx-context
run: |
docker context create test-builder
- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v1
with:
endpoint: test-builder
- name: Build and push Docker images
uses: docker/build-push-action@v2.7.0
Logs
Download the log file of your build and attach it to this issue.
Our logs are littered with private, implementation details. Could you help me isolate which part of the logs you need for diagnosis?
2021-12-01T19:25:54.2902593Z Found online and idle self-hosted runner in the current repository's enterprise account that matches the required labels: 'self-hosted'
2021-12-01T19:25:54.4048239Z Waiting for the runner to pick up this job...
2021-12-01T19:26:54.7363687Z Job was not picked up by the preferred runner, try requesting additional runners to run this job
2021-12-01T19:26:54.8282602Z Waiting for a self-hosted runner to pickup this job...
2021-12-01T19:32:40.3175224Z Job is about to start running on the runner: self-hosted-runner-group-6zjgq-67xpv (enterprise)
2021-12-01T19:32:44.8744015Z Current runner version: '2.285.0'
2021-12-01T19:32:44.8753670Z Runner name: 'self-hosted-runner-group-6zjgq-67xpv'
2021-12-01T19:32:44.8755541Z Runner group name: 'self-hosted'
2021-12-01T19:32:44.8758083Z Machine name: 'self-hosted-runner-group-6zjgq-67xpv'
...
2021-12-01T19:32:49.5211515Z ##[group]Download and install buildx
2021-12-01T19:32:49.5825561Z Downloading https://github.com/docker/buildx/releases/download/v0.7.1/buildx-v0.7.1.linux-amd64
2021-12-01T20:12:24.6445002Z ##[error]The operation was canceled.
2021-12-01T20:12:24.6564749Z Post job cleanup.
2021-12-01T20:12:24.7568406Z Post job cleanup.
2021-12-01T20:12:24.9055991Z Post job cleanup.
2021-12-01T20:12:24.9780442Z Post job cleanup.
2021-12-01T20:12:25.0473555Z [command]/usr/local/bin/docker logout
2021-12-01T20:12:25.1428510Z Removing login credentials for https://index.docker.io/v1/
2021-12-01T20:12:25.1518891Z Cleaning up orphan processes
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 12
- Comments: 61 (19 by maintainers)
Commits related to this issue
- Test fix for https://github.com/docker/setup-buildx-action/issues/117 — committed to ironcore-dev/ironcore by damyan 3 years ago
OK, what solved the issue for us was pinning the version to 287.1 and disabling the auto update of the runner:
@crazy-max If we add the extra
Install Docker Buildxstep the pipeline finished successfully.@jnsvd I’m really confused. Can you add this step before the
setup-buildx-actionstep to see if it’s linked to theactions/tool-cachepkg that downloads buildx?:I’d also like to confirm that when I switch to
v2.284.0this works.we are now running into this issue since our runners updated to
v2.288.0the extra
Install Docker Buildxstep described earlier does not help@crazy-max
Set-up job logs:
docker/setup-buildx-action@v1.5.0logs:I can confirm, that it is resolved with the new runner version 🙂
@maxnowack https://github.com/docker/setup-buildx-action/issues/117#issuecomment-984664730 should mitigate your issue in the meantime.
Yeah, unfortunately the runners self-update. I’m running a very hacky loop on my local machine to keep pipelines running for myself - fortunately, I’m the one triggering most of these pipelines so it’s only me inconvenienced.
It’s dirty and I’m not proud of it, but it is working.
Looks like this is an issue since node v12.19.0
v12.18.4 works fine…
base on https://nodejs.org/uk/blog/release/v12.19.0/ the highly suspicious change is
deps: upgrade to libuv 1.39.0 (Colin Ihrig) #34915which lines up with people discussed in https://github.com/nodejs/node/issues/40200
Github is slowly rolling out updates to their runners so it took many tries to get a run on an upgraded self-hosted runner…
Here are the logs:
There are no further logs. It’s just stuck at that spot
Update:
Would it be possible to add more logging for each of the steps? That might help future debugging
Yet another “same here”. Self-hosted k8s runners on a private repo stuck forever at the buildx step.
We’re seeing the same error on self-hosted runners. Version auto-changed overnight and now the job is timing out on the
uses: docker/setup-buildx-action@v1.5.0step:@dresselm
Interesting indeed, are you on GHE?