azure-pipelines-tasks: HelmDeploy@0 and Kubernetes@1 do not work with kubelogin

Question, Bug, or Feature? Type: bug

Task Name:

• HelmDeploy@0
• Kubernetes@1

Environment

• Server: Azure Pipelines

• Agent: Private, ubuntu 18.04, version 2.196.2

Issue Description

Both HelmDeploy@0 and Kubernetes@1 do not work with kubelogin while under the same conditions the plain helm and kubectl binaries do.

We run AKS with AAD integration enabled and avoid using local K8S service accounts. Instead we use an ARM Service Connection with MSI. If you try to use kubectl using an ARM Service Connection on an AKS cluster with AAD integration you will get a prompt for https://microsoft.com/devicelogin which does not work in a devops pipeline context. . The kubelogin utility is a client-go credential plugin for kubectl that implements Azure AD authentication. This works fine when using it from the command prompt, but not when using the HelmDeploy@0 and Kubernetes@1 tasks. Somehow there is a subtle difference in the process context where the kubectl and helm are used from the command line when compared to the invocation done by the tasks.

Below is an example pipeline to illustrate the issue:

• The first task does an az aks get-credentials, installs kubelogin and converts the kubeconfig file to use kubelogin.
• The seconds task uses kubectl and helm binaries directly from a bash shell. This works without issues.
• The third task uses HelmDeploy@0 task to list the deployed charts. This fails when calling the kubelogin credential plugin: getting credentials: exec: executable kubelogin failed with exit code 1.
• The fourth task uses Kubernetes@1 task to list pods in a namespace. This fails with similar errors as the helm task.

      - task: AzureCLI@2
        displayName: aks non-admin login using kubelogin
        inputs:
          azureSubscription: ${{parameters.serviceConnection}}
          scriptType: bash
          scriptLocation: inlineScript
          inlineScript: |
            az aks get-credentials -n <aks cluster> -g <resource group>
            # install kubelogin
            wget https://github.com/Azure/kubelogin/releases/download/v0.0.10/kubelogin-linux-amd64.zip
            unzip kubelogin-linux-amd64.zip
            mv bin/linux_amd64/kubelogin /usr/bin
            kubelogin convert-kubeconfig -l azurecli
 
      - task: AzureCLI@2
        displayName: test kubectl and helm from bash
        continueOnError: true 
        inputs:
          azureSubscription: ${{parameters.serviceConnection}}
          scriptType: bash
          scriptLocation: inlineScript
          inlineScript: |
            kubectl get pods -n <some namespace>
            helm list --namespace <some namespace>

      - task: HelmDeploy@0
        displayName: test HelmDeploy task
        continueOnError: true 
        inputs:
          connectionType: None
          namespace: <some namespace>
          command: list

      - task: Kubernetes@1
        displayName: test Kubernetes task
        continueOnError: true 
        inputs:
          connectionType: None
          command: get
          arguments: pods -n <some namespace>

Possibly this issue needs to be fixed in the kubelogin project but would like to understand the likely differences on how the tasks call the underlying kubectl and helm binaries when compared to invoking them from bash.

Task logs

See attached logs.zip

Troubleshooting

Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting

Error logs

2022-01-19T20:33:52.7909008Z ##[debug]set helmExitCode=1
2022-01-19T20:33:52.7910461Z Error: Kubernetes cluster unreachable: Get "https://<aks cluster>-a15872ed.hcp.westeurope.azmk8s.io:443/version?timeout=32s": getting credentials: exec: executable kubelogin failed with exit code 1
2022-01-19T20:33:52.7912489Z helm.go:88: [debug] Get "https://<aks cluster>-a15872ed.hcp.westeurope.azmk8s.io:443/version?timeout=32s": getting credentials: exec: executable kubelogin failed with exit code 1
2022-01-19T20:33:52.7915456Z ##[debug]Processed: ##vso[task.setvariable variable=helmExitCode;issecret=false;]1