cml: `cml ` on cml runner instance fails

I have tried a few iterations of this:

jobs:
  setup:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: iterative/setup-cml@v1
      - name: deploy runner
        env:
          GOOGLE_APPLICATION_CREDENTIALS_DATA: ${{ secrets.GCP_CML_RUNNER_KEY }}
        run: |
          cml runner \
            --single \
            --labels=test \
            --token=${{ secrets.DACBD_PAT }} \
            --cloud=gcp \
            --cloud-region=us-west \
            --cloud-type=e2-highcpu-2
  test:
    needs: [setup]
    runs-on: [self-hosted, test]
    steps:           
      - uses: actions/checkout@v2
      - run: |
          which cml
          cml ci

Fails with an odd message: Error: Cannot find module '/tmp/tmp.njlS8oR0sf/.cml/cml-ehsjbxeja3/_work/cml-pulse/cml-pulse/ci'

and:

...
  test:
    needs: [setup]
    runs-on: [self-hosted, test]
    steps:           
      - uses: actions/checkout@v2
      - uses: iterative/setup-cml@v1
      - run: |
          which cml
          cml ci

Fails with a EEXIST: file already exists, symlink '../lib/node_modules/@dvcorg/cml/bin/cml.js' -> '/usr/bin/cml'

However using --cml-version=github:iterative/cml.git#master \ to trigger the npm install -g way of setting up cml on the runner instance works fine.

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 1
Comments: 16 (16 by maintainers)

Most upvoted comments

It looks like iterative/terraform-provider-iterative#458 might do the trick, but it’s just a huge hack to bypass the PKG_EXECPATH condition. 🪓

👀 🤔

As per https://github.com/vercel/pkg/issues/897#issuecomment-1076988215, I doubt that the current implementation of https://github.com/iterative/cml/pull/802 can not unset that environment variable, because it will be overridden by pkg again. 😅

I think what I have in mind shouldn’t care (if it works); as far as I have seen there is no issue with repeated invocations, just the “re-invocation” within the inherited context of the first pkg cml runner call which is the “parent” process of github actions runner and thus subsequent use of the cml cmd

Finished doing some reading and plan to try it out tomorrow evening 😴

dacbd on Mar 24, 2022

While “libelous” I feel validated with suspicion of child_process/spawn weirdness, also well done on the hunt! (https://github.com/vercel/pkg/issues/897#issuecomment-1076988215) I glanced at that chunk but, I missed the value of this in my initial grep of the codebase after I had found the culprit env.

To summarize, I agree I think the fix is solving #802 which I have an idea for beyond what is currently implemented in that branch which I intend to test out soon.

dacbd on Mar 24, 2022

libelous

🤣 /edit that was the one strikethrough I missed, I have not revisited the rabbit hole I dug trying to find the root of the issue

The cml standalone binary (#842 & iterative/terraform-provider-iterative#255) used for (2) is a pkg binary. As such, it sets the PKG_EXECPATH environment variable that propagates to all the child processes up to (5).

Does it set the env? In my basic docker reproduction running a cml command did not seem to set that ENV? … Though…

I don’t understand yet why setting PKG_EXECPATH causes binaries bundled with pkg to fail, but this can be mitigated by unsetting this variable before calling the GitHub runner process; see #802 for a stale draft.

I suspect you are correct that #802 would fix this.

dacbd on Mar 23, 2022

0x2b3bfa0 on Mar 24, 2022

Playing around more it seems it only fails if PKG_EXECPATH is set the actual executable

with Dockerfile:

FROM debian:buster

RUN apt-get update && apt-get install -y curl
RUN mkdir -p /opt/cml/
RUN curl --location --url https://github.com/iterative/cml/releases/download/v0.12.0/cml-linux --output /opt/cml/cml-linux
RUN chmod +x /opt/cml/cml-linux
RUN ln -s /opt/cml/cml-linux /usr/bin/cml

Which installs cml the same as tpi,

docker run --rm -it test:good /bin/bash
root@e3cab72f74e3:/# cml --help
# Good
root@e3cab72f74e3:/# PKG_EXECPATH=/random/garbage cml --help
# Good
root@e3cab72f74e3:/# PKG_EXECPATH=/opt/cml/cml-linux cml --help
# Bad

so if PKG_EXECPATH is set to the exact pkg binary cml will fail

dacbd on Mar 23, 2022

All the mentions to node-pseudoexec are, at least, libellous. 😄 Fortunately, this error doesn’t seem to have anything to do with the command plugin feature from #703.

When you (1) run cml runner --cloud=gcp it creates a cloud instance and (2) runs cml runner inside, which, in turn, (3) executes the GitHub Actions runner in charge of (4) running the workflow that (5) calls cml ci in the example above.

The cml standalone binary (#842 & https://github.com/iterative/terraform-provider-iterative/pull/255) used for (2) is a pkg binary. As such, it sets the PKG_EXECPATH environment variable that propagates to all the child processes up to (5).

I don’t understand yet why setting PKG_EXECPATH causes binaries bundled with pkg to fail, but this can be mitigated by unsetting this variable before calling the GitHub runner process; see #802 for a stale draft.

0x2b3bfa0 on Mar 23, 2022