cml: cml cloud runner doesn't receive runner token

Hi everybody, I saw some other issues like this, but the instructions there didn’t seem to help.

We are trying to set up a CML pipeline using azure VMs for the heavy lifting. We have adapted all the necessary things in the docker-machine command

docker-machine create -d azure
      --azure-subscription-id $(az account show --query "id" -o tsv)
      --azure-client-id ${AZURE_SP_APP_ID}
      --azure-client-secret ${AZURE_SP_PASSWORD}
      --azure-location westeurope
      --azure-ssh-user cml_runner
      --azure-size $MACHINE_SIZE
      --azure-resource-group $RESOURCE_GROUP
      --azure-vnet $VNET
      --azure-subnet $SUBNET
      --azure-open-port 6006 
      $MACHINE

Now this works and the subsequent setting up of the NVIDIA drivers (following the same as in your blogpost and that seems to work. But then when we try to run the docker container, the runner is never registered in gitlab. We use this command:

docker run --name runner --gpus all -d
        -v /docker_machine/machine:/root/.docker/machine
        -e DOCKER_MACHINE=$MACHINE
        -e repo_token=$repo_token
        -e RUNNER_LABELS=$RUNNER_LABELS
        -e RUNNER_REPO=$CI_PROJECT_URL
        -e RUNNER_IDLE_TIMEOUT=600
        dvcorg/cml-py3

When I tested the setup locally on my machine and run the docker container in the foreground, I get this output:

Unregistering runner
Runtime platform                                    arch=amd64 os=linux pid=17 revision=86ad88ea version=13.3.0
Running in system-mode.                            
                                                   
Shutting down docker machine
Error: RUNNER_TOKEN is needed to start the runner. Are you setting a runner?
    at run (/cml/bin/cml-cloud-runner-entrypoint.js:86:11)

So I guess the call for the runner token seems to fail somehow. Is there a possibility that azure somehow blocks the request to the gitlab api or are we missing something obvious here? We checked the repo_token’s permission, so that should be okay. Can some other repository settings interfere here?

If you need more info about our system, feel free to ask.