terrakube: executor is failed to download private terraform module from bitbucket

@alfespa17 i don’t find any documentation related to find workspace or any environment variable for executing custom scripts. basic flow:

* type: "customScripts"
  name: "Initializing"
  step: 100
  commands:
  
  * runtime: "BASH"
    priority: 200
    before: true
    script: |
    ls -la
    ./cloneforemanterraform.sh -b

* type: "approval"
  name: "Approve Plan from Terraform CLI"
  step: 150

If cloneforemanterraform script inside my workspace but when I initialize the jobs its not able to find it If you guide me that would be awesome

Ok let me explain a little bit how terrakube executes jobs internally.

• Terrakube Executor create a folder where it will download the workspace repository and extension tools.
  • The path is : /home/cnb/.terraform-spring-boot/executor/{{ORGANIZATION ID}}/{{WORKSPACE ID}}
• Terrakube will clone the workspace git repository to this directory
• Terrakube will also clone the “Extension Repository” to “/home/cnb/.terraform-spring-boot/executor/{{ORGANIZATION ID}}/{{WORKSPACE ID}}/.terrakube/toolsRepository”" if you are using the default it will be “https://github.com/AzBuilder/terrakube-extensions”
• After cloning the tools repository Terrakube will scan the repository for any “.sh” file, will make the script “executable” and add it to the linux “PATH” in your workspace.
  • You can find the internal logic here https://github.com/AzBuilder/terrakube/blob/main/executor/src/main/java/org/terrakube/executor/service/scripts/bash/BashEngine.java#L36
• Terrakube will inject a lot of information to the custom scripts(this will include env variables and terraform variables)
• You can check https://github.com/AzBuilder/terrakube-extensions#field-injection

If you want to execute some custom script inside your workspace

    type: "customScripts"
    name: "Initializing"
    step: 100
    commands:
        runtime: "BASH"
        priority: 200
        before: true
        script: |
        ls -la
        chmod +x /home/cnb/.terraform-spring-boot/executor/$organizationId/$workspaceId/cloneforemanterraform.sh 
        ./home/cnb/.terraform-spring-boot/executor/$organizationId/$workspaceId/cloneforemanterraform.sh -b
    type: "approval"
    name: "Approve Plan from Terraform CLI"
    step: 150

Basically your script “cloneforemanterraform” is not inside the linux PATH so you can not execute directly, you need to write the full path

This could be a new feature for terrakube import the “sh” files inside the “workspace” to the jobs execution environment and not only those inside the “extension repository” (you could create a new issue for this feature)

Finally you don’t see any information when you open the container directly because terrakube delete the folder every time the job is completed.

https://github.com/AzBuilder/terrakube/blob/main/executor/src/main/java/org/terrakube/executor/service/executor/ExecutorJobImpl.java#L87

I hope this information can help you.

Let me know if you have any additional question I will be happy to help 👍

Hello @thatsk

I got it working with docker-compose, you need to mount id_rsa and config

config file content:

StrictHostKeyChecking no

  terrakube-executor:
    image: azbuilder/executor:2.13.0
    ports: 
    - 8090:8090
    env_file:
      - executor.env
    volumes:
    - ./config:/home/cnb/.ssh/config
    - ./id_rsa:/home/cnb/.ssh/id_rsa

The container is running with user “cnb”

cnb@5d1ba02a71a9:~/.ssh$ whoami
cnb
cnb@5d1ba02a71a9:~/.ssh$ id
uid=1002(cnb) gid=1000(cnb) groups=1000(cnb)
cnb@5d1ba02a71a9:~/.ssh$ 

So you need to set the correct permission for the file config and id_rsa before running docker-compose up

sudo chown 1002 id_rsa
sudo chown 1002 config

Once you set the correct permission for cnb user, it should be able to read the files

cnb@5d1ba02a71a9:~/.ssh$ cat config 
StrictHostKeyChecking no

And when running the executor with a private repository from bitbucket it will work when using terraform

module "test" {
  source = "git@bitbucket.org:alfespa17/private-module.git"
}

terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - 
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Initializing the backend...
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - 
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Successfully configured the backend "s3"! Terraform will automatically
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - use this backend unless the backend configuration changes.
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Initializing modules...
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Downloading git::ssh://git@bitbucket.org/alfespa17/private-module.git for test...
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - - test in .terraform/modules/test
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - 
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Initializing provider plugins...
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - - Finding latest version of hashicorp/null...
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - - Finding latest version of hashicorp/time...
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - - Installing hashicorp/null v3.2.1...
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - - Installed hashicorp/null v3.2.1 (signed by HashiCorp)
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - - Installing hashicorp/time v0.9.1...
terrakube-api_1       | 2023-05-17 23:58:00.008  INFO 1 --- [ryBean_Worker-3] o.t.api.plugin.scheduler.ScheduleJob     : Checking Job 1 Status running
terrakube-api_1       | 2023-05-17 23:58:00.008  INFO 1 --- [ryBean_Worker-3] o.t.api.plugin.scheduler.ScheduleJob     : Job 1 running
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - - Installed hashicorp/time v0.9.1 (signed by HashiCorp)
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - 
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Terraform has created a lock file .terraform.lock.hcl to record the provider
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - selections it made above. Include this file in your version control repository
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - so that Terraform can guarantee to make the same selections by default when
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - you run "terraform init" in the future.
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - 
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Terraform has been successfully initialized!
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - 
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - You may now begin working with Terraform. Try running "terraform plan" to see
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - any changes that are required for your infrastructure. All Terraform commands
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - should now work.
terrakube-executor_1  | [ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - 
terrakube-executor_1  | [threadPoolTaskExecutor-1] WARN org.terrakube.executor.service.terraform.TerraformExecutorServiceImpl - No commands to run before terraform operation Job 1

The tricky part was the file access for id_rsa and config files 👍 when running the container