actions-runner-controller: Runner upgrade to 2.276.0 failed

Runner auto-upgrade to 2.276.0 failed

2021-01-20 01:59:58Z: Listening for Jobs
Runner update in progress, do not shutdown runner.
Downloading 2.276.0 runner
Waiting for current job finish running.
Generate and execute update script.
Runner will exit shortly for update, should back online within 10 seconds.
renamed '/runner/_diag/SelfUpdate-20210120-020013.log' -> '/runner/_diag/SelfUpdate-20210120-020013.log.failed'
Runner listener exited with error code 4
Runner listener exit with undefined return code, re-launch runner in 5 seconds.
Starting Runner listener with startup type: service
Started listener process
events.js:187
      throw er; // Unhandled 'error' event
      ^

Error: spawn /runner/bin/Runner.Listener ENOENT
    at Process.ChildProcess._handle.onexit (internal/child_process.js:264:19)
    at onErrorNT (internal/child_process.js:456:16)
    at processTicksAndRejections (internal/process/task_queues.js:80:21)
Emitted 'error' event on ChildProcess instance at:
    at Process.ChildProcess._handle.onexit (internal/child_process.js:270:12)
    at onErrorNT (internal/child_process.js:456:16)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  errno: 'ENOENT',
  code: 'ENOENT',
  syscall: 'spawn /runner/bin/Runner.Listener',
  path: '/runner/bin/Runner.Listener',
  spawnargs: [ 'run', '--startuptype', 'service', '--once' ]
}

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 10
Comments: 17 (4 by maintainers)

Commits related to this issue

Tested Self-update Fixes (#269) Adding fixes to #253 as confirmed and tested in https://github.com/summerwind/actions-runner-controller/issues/264#issuecomment-764549833 by @jolestar, @achedeuzot and... — committed to actions/actions-runner-controller by hfuss 3 years ago
Fix self-update failuers due to /runner/externals mount (#253) * Fix self-update failuers due to /runner/externals mount Fixes #252 * Tested Self-update Fixes (#269) Adding fixes to #253 as ... — committed to actions/actions-runner-controller by mumoshu 3 years ago
Fix self-update failuers due to /runner/externals mount (#253) * Fix self-update failuers due to /runner/externals mount Fixes #252 * Tested Self-update Fixes (#269) Adding fixes to #253 as ... — committed to neggles/actions-runners by mumoshu 3 years ago
Fix self-update failuers due to /runner/externals mount (#253) * Fix self-update failuers due to /runner/externals mount Fixes #252 * Tested Self-update Fixes (#269) Adding fixes to #253 as ... — committed to neggles/actions-runners by mumoshu 3 years ago

Most upvoted comments

Hi folks, we just tested @achedeuzot’s fixes on https://github.com/hfuss/actions-runner-controller/tree/testing-self-update-fix-2.276.0 and also had success. You can find the images here in the interim:

Patched to handled auto-update: brix4dayz/actions-runner:v2.275.1
To avoid auto-update altogether: brix4dayz/actions-runner:v2.276.0
Controller image with runner volumes needed by newer images: brix4dayz/actions-runner-controller:v0.16.1-prerelease

Example output from self-updating runner:

 No newline at end of file
.path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Starting Runner listener with startup type: service
Started listener process
Started running service
√ Connected to GitHub
2021-01-21 19:24:31Z: Listening for Jobs
Runner update in progress, do not shutdown runner.
Downloading 2.276.0 runner
Waiting for current job finish running.
Generate and execute update script.
Runner will exit shortly for update, should back online within 10 seconds.
Runner listener exited with error code 4
Runner listener exit with undefined return code, re-launch runner in 5 seconds.
Starting Runner listener with startup type: service
Started listener process
√ Connected to GitHub
2021-01-21 19:28:51Z: Listening for Jobs
2021-01-21 19:28:54Z: Running job: Tests
2021-01-21 19:29:17Z: Job Tests completed with result: Succeeded
Runner listener exited with error code 0

Opened #269 to get #253 updated and ready to :shipit:

hfuss on Jan 21, 2021

Hi !

@mumoshu I can confirm it works with the PR on the branch fix-self-update-failure with a few minor fixes.

This is what I did:

USER runner instead of USER runner/runner in the runner Dockerfile
adding sudo chown -R runner:docker /runner and mkdir ./externals in the runner/entrypoint.sh file
updating the controller image in my k8s cluster to my custom built one
updating my custom runner image to build from the patched runner image

➡️ everything is working again 😁

achedeuzot on Jan 21, 2021

@hfuss great

The actions-runner-controller image needs to be replaced with the image found in https://github.com/summerwind/actions-runner-controller/releases/latest/download/actions-runner-controller.yaml is that right?

kaykhancheckpoint on Jan 21, 2021

I am hitting the same error. I wasn’t able to resolve it by creating containers using the branch in #253. This error is blocking development work on a new CD workflow so I’m hoping to find a solution quickly. Is there something I can do to assist testing?

terry-yolabs on Jan 21, 2021

2.276.0 is not compatible with GitHub Enterprise Server - see https://github.com/actions/runner/pull/896#issuecomment-764016208

I would recommend upgrading to the latest version: https://github.com/actions/runner/pull/931

jonico on Jan 22, 2021