azure-pipelines-agent: Azure subscription endpoint ID cannot be provided through a variable in build definition YAML file

There is an obsession here that closing issues because they don’t fit a timeline potentially makes GitHub almost a trawl of items that are closed prematurely and not being addressed or actioned, or are just closed items that refer to other closed items to other closed item etc, like a version of inception with no answer.

Very frustrating

Update on this? I got the same issue.

This issue still persists, please reopen the issue.

Who’s here in 2020 and still can’t believe this is a thing?

To be more precise. What is not working are variables referenced in variable groups not hard coded in yaml files but in Azure DevsOps variable groups. This is the recommended way to reuse variables over pipelines.

I’ve posted over half a year ago that its working, if its not for you - you misconfigured something.

Issue still exists

Input

  - job: ARMDeployment
    displayName: Create ARM Deployment
    pool:
      vmImage: ubuntu-latest
    variables:
      service_connection: My Service Connection
    steps:
      - task: AzureResourceGroupDeployment@2
        inputs:
          azureSubscription: $(service_connection)

Expected: Task AzureResourceGroupDeployment@2 uses service connection name My Service Connection

Actual: Task AzureResourceGroupDeployment@2 fails as variable is not expanded

The pipeline is not valid. Job ARMDeployment: Step AzureResourceGroupDeployment input ConnectedServiceName references service connection $(service_connection) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."

Closing this specific issue. The re-use scenarios are on our backlog.

Using template parameters is not the issue, but using e.g. variables from variable groups with macro syntax $(serviceConnection)

Hi all, I think there are two separate issues here. It’s great @mathieu-benoit’s issue has been worked around though.

@ericsciple - your suggestion of using a condition to switch between two versions of the same task, but with different Azure service ID references, isn’t really a long-term solution. It’ll get us through for now, but ultimately we’d surely want the build.yaml file to be agnostic to the GUIDs used within VSTS - and for these to be able to be provided in variables. This will become very important when we get VSTS release definitions as YAML.

I understand that the endpoints are authorised on push of the definition, but it feels like this is a legitimate use case. It may not be a bug per se, but could this be tracked as a feature improvement request?

any news on this? the template approach does not work, it still does not resolve variables.

variable-docker-login.yml:

parameters:
  azureSubscriptionEndpoint: ''
  azureContainerRegistry: ''

steps:
- task: Docker@1
  displayName: Container registry login
  inputs:
    command: login
    azureSubscriptionEndpoint: ${{ parameters.azureSubscriptionEndpoint }}
    azureContainerRegistry: ${{ parameters.azureContainerRegistry }}.azurecr.io

in the build pipeline i add a variable group, then

  - template: steps/variable-docker-login.yml@pipelines
    parameters:
      azureSubscriptionEndpoint: $(AZURE_SUBSCRIPTION)
      azureContainerRegistry: $(AZURE_CONTAINER_REGISTRY)

the pipeline fails immediately with:

The pipeline is not valid. Job Build: Step Docker input azureSubscriptionEndpoint references service connection $(AZURE_SUBSCRIPTION) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz.

same behavior inline without templates.

Please reopen this. It is a general problem and we need to be able to use variable groups in templates to ease developer experience

It’s pretty frustrating that the developer community ticket was closed quickly too… 😦

I suspect most people are just making a template for each environment and moving on, but it’s not exactly the best experience. How can we find the right people to talk to and solve this? Do we need to open a support request?

@ericsciple

@lqueryvg templates is the answer

How can templates be the answer ?

I hope you are not suggesting that people write a separate template for each possible service endpoint.

This goes against the whole idea of using variables.

This issue quite clearly needs to be fixed.

So when is this likely to get fixed, and where is this being tracked ? Also, if this is not considered to be a bug, where is this behaviour documented ?

Forcing users to hard-code configuration information like this is always BAD.

HERE IS MY USE CASE:

1. We have 2 organisations (with separate VSTS domains / projects) pointing at YAML in git in a shared repo. I.e. two VSTS, one git repo.
2. Each organisation wants to use it’s own names for the service endpoints.
3. But, since Microsoft are forcing us to hard-code the service endpoint name into our YAML code (because we can’t use variables), both organisations are forced to use the same names - which in some case breaks our naming standards.

@bryanmacfarlane or @ericsciple Any github issue or entry on the roadmap we can refer to in order to know when this will be available? Is it plan for GA?

The value for “SubscriptionA” in my example is listed in my pipeline as a variable. The pipeline is calling the template “deploy-webapp.yaml” as one of its steps, and passing the parameter into it.

This gets even weirder with current multi stage pipelines and environments. Conditions can only be expressed on stage, step or job-level. So I decided to work around this issue by having a template containing 3 jobs with a single task only. Each job is not a plain job-definition but a deployment-definition with an environment:

jobs:
- deployment: 'onDev'
  environment: 'dev'
  condition: eq('${{ parameters.environmentName }}', 'dev')
  strategy:
    runOnce:
      deploy:
       steps:
       - task: AzurePowerShell@3
         inputs:
           azureSubscription: 'azure-dev' # neither $(Environment.Name) nor ${{ parameters. }} does work here
- deployment: 'onInt'
  environment: 'int'
  condition: eq('${{ parameters.environmentName }}', 'int')
  strategy:
  # same job as above
  # and again for prod

It works and runs only the job matching the condition but requires approval according to all 3 environments, no matter which environment it runs on. Executing for dev needs approval for the two others, same in int and prod.

My intention was only to have 3 stages, each for its own environment running the single PowerShell task only switching azureSubscription based on ${{ parameters.environmentName }} which would have been set to $(Environment.Name).

@ericsciple, I think specific scenarios are confusing the issue.

It’s as simple as this…

I want to be able to pass the service endpoint as a variable. This is exactly what this ticket is supposed to address.

Without this, I am unable to separate code (in git) from configuration in VSTS which in turn makes it impossible to re-use my code for different configurations.

I should not have to add or change any yaml code (in git) every time I change or add a new service endpoint.

Without this feature, VSTS does not allow me to follow the DevOps mantra of “code all the things” and is not a viable tool for true DevOps.

@ericsciple , I’m sorry my wording was incorrect. Your solution is not multiple templates, it’s one. But it’s still multiple yaml files - one for each service endpoint (this is what I meant). I.e. you are moving the values (i.e. the endpoint name) into code, when it should really be in variables.

My point still stands.

nice, the variable was set as secret, now as plain text it works. This changes our workflow, but… Thank you for the help @copdips

my solution is to use a single azureSubscription for all the repos within the same project, then I set the value staticly in clear text in the pipeline, you can use a variable, but the value should be in clear text.

1. This is currently operational with a task template.
2. First two belong in the template
3. Second two are part of the pipeline.

Think of it as a variable per stage, if you have different subscriptions for each stage. Different variables for different stages are pretty commonplace, but through further interpolation could also be made simpler.

Below is a full example:

Pipeline:

trigger: none



variables:
- group: Global-Variables
- name: buildName
  value: Build1
- name: customerName
  value: MyCustomer
- name: customerNameAbbreviation
  value: myc
- name: pipelinenumber
  value: 15

name: $(Myapp-Version)

stages:
- stage: Dev
  displayName: Deploy To Dev
  jobs:  
  - deployment: Deploy_to_Dev
    displayName: Deploy To Dev
    variables:
      regionPrefix: eg
      environment: dev
      environmentAbbreviation: dev
      serviceConnectionName: SubscriptionA # Service Connection Name
    pool:
      name: 'Windows'
    environment: Dev
    workspace:
      clean: all
    strategy: 
      runOnce:
        deploy:
          steps:
          - template: ../templates/deploy-webapp.yaml  # Template reference
            parameters:
              regionPrefix: '$(regionPrefix)'
              customerName: '$(customerName)'
              customerNameAbbreviation: '$(customerNameAbbreviation)'
              environment: '$(environment)'
              environmentAbbreviation: '$(environmentAbbreviation)'
              pipelinenumber: '$(pipelinenumber)'
              serviceConnectionName: ${{ variables.serviceConnectionName }} # Passing in service connection name here

Template:

parameters:
- name: 'regionPrefix'  # defaults for any parameters that aren't specified
  type: string
  default: false
- name: 'customerName'  # defaults for any parameters that aren't specified
  type: string
  default: false
- name: 'customerNameAbbreviation'  # defaults for any parameters that aren't specified
  type: string
  default: false
- name: 'environment'  # defaults for any parameters that aren't specified
  type: string
  default: false
- name: 'environmentAbbreviation'  # defaults for any parameters that aren't specified
  type: string
  default: false
- name: 'pipelinenumber'  # defaults for any parameters that aren't specified
  type: string
  default: false
- name: 'serviceConnectionName'  # defaults for any parameters that aren't specified
  type: string
  default: false

steps:
- task: PowerShell@2
  displayName: List Directory
  enabled: true
  inputs:
    targetType: 'inline'
    script: |
      cd $(Pipeline.Workspace)
      pwd
      dir
      tree /F
- task: AzureRmWebAppDeployment@4
  displayName: 'Deploy App'
  inputs:
    ConnectionType: 'AzureRM'
    azureSubscription: ${{ parameters.serviceConnectionName }}
    appType: 'webApp'
    WebAppName: '$(regionPrefix)-$(customerNameAbbreviation)$(environmentAbbreviation)myapp-wa'
    deployToSlotOrASE: true
    ResourceGroupName: '$(regionPrefix)-$(customerNameAbbreviation)$(environmentAbbreviation)-rg'
    SlotName: 'staging'
    packageForLinux: '$(Pipeline.Workspace)\release-$(customerName)-$(environment)-app.zip'
    enableCustomDeployment: true
    DeploymentType: 'webDeploy'
    RemoveAdditionalFilesFlag: true
- task: AzureAppServiceManage@0
  displayName: 'Swap Slots'
  inputs:
    azureSubscription: ${{ parameters.serviceConnectionName }}
    Action: 'Swap Slots'
    WebAppName: '$(regionPrefix)-$(customerNameAbbreviation)$(environmentAbbreviation)myapp-wa'
    ResourceGroupName: '$(regionPrefix)-$(customerNameAbbreviation)$(environmentAbbreviation)-rg'
    SourceSlot: 'staging'
    TakeAppOfflineFlag: true

Azure Devops need to check all permissions at the beginning of the run of the pipeline.

…and could add evaluation of variable expressions to first pass.

Nevertheless I’ll give the global variable definition a try, if I can convince the others to avoid typos in variable names or copy&paste-induced issues because of wrong variable. My intention is to avoid both issues by using/ incoporating the EnvironmentName.

Main issue with complex (=real world) pipelines is typos/ wrong environment. And it is not very comfortable since undefined variables evaluate to empty string instead of warning about an undefined variable.

Does this feel like the revival of Visual Basic with all its »features«? I waste my time fixing errors I didn’t cause but otherwise couldn’t proceed as fast as I supposed to. Why is this YAML-thing an advantage over autotools/ make or why do I need minutes to let Jenkins parse existing pom.xml and derive build graphs from dependencies but hours to figure out how to ingest Environment.Name in a variable?

@4c74356b41 the best way to find the guid is save it on a designer build, then browse to the History tab, and diff the history of the designer definition. The task input will contain the guid.

I’m not surprised by your skepticism. I just left a comment on the other issue, detailing the bug that is causing a very confusing experience.

@ericsciple

Right, so you are suggesting that we write a separate template for each service endpoint. As I said, this goes against the whole idea of using variables and the problem still stands.

As as DevOps engineer I understand the importance of being able to separate code from configuration. Creating separate templates (code) for every service endpoint (configuration) is obviously not a proper solution - it’s a compromise in order to work around a feature which is missing from VSTS.

I get the impression (ie this ticket is “Closed”) that Microsoft does not see this as a problem ?

@lqueryvg templates is the answer

I agree. We are thinking about multiple types of process-reuse scenarios and ways to pass inputs to a re-usable process.

@mathieu-benoit if you make a trivial edit to the web definition and save, the act of saving the web definition will trigger resource authorization to run again (loads file from master)

Endpoint authorization for yaml definitions is performed on-push of the file and on-edit of the definition. It will not work if you use a variable.

If you want to be variable, you could add two tasks, and use conditions to drive the runtime-decision which one is executed. You can use queue-time variables in your condition.