apiops: [BUG] Application Insights YAML Override doesn't work as expected

Release version

v3.0.1

Describe the bug

I tried using YAML configuration to replace the application insights for stage and production environment. But it seems like the YAML config doesn’t replace the value as expected.

I got this error from the Azure DevOps publish pipeline which cannot find the instrumentation key and use the incorrect application insight:

System.InvalidOperationException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/dataops/providers/Microsoft.ApiManagement/service/dataops-st/loggers/dataops-appinsights?api-version=2021-12-01-preview failed with status code 400. Content is '{"error":{"code":"ValidationError","message":"Validation failed for logger-id 'dataops-appinsights'. Exception Received 'Invalid instrumentation key for Application Insights Logger'","details":null}}'.

The actual resourceId that should be used in this step should be the application insights with the name of dataops-appinsights-stage and not dataops-appinsights.

My YAML configuration looks like to this configuration.staging.yaml

apimServiceName: dataops-st
namedValues:
  - name: dataops-adf-key
    properties:
      displayName: dataops-adf-key
      keyVault: 
        secretIdentifier: "https://xxx-dataops-stage.vault.azure.net/secrets/dataops-adf-key"
backends:
  - name: dataops-adf
    properties:
      url: "https://dataops-adf.azurewebsites.net/api"
      resourceid: "https://management.azure.com/subscriptions/xxx/resourceGroups/<rg-name>/providers/Microsoft.Web/sites/dataops-adf"
loggers:
  - name: dataops-appinsights
    properties:
      resourceId: "/subscriptions/xxxx/resourceGroups/<rg-name>/providers/microsoft.insights/components/dataops-appinsights-stage"
      credentials:
        instrumentationKey: "application-insights-instrumentation-key"

I have the similar file with the name configuration.production.yaml, which replacing the value for production environment.

Currently, I use the Azure DevOps extract pipeline to change the folder name of loggers and named values of the generated instrumentation key.

The added stage for changing the directory name looks like this.

- task: Bash@3 
            displayName: Change Logger folder name
            inputs:
              targetType: 'inline'
              script: |
                mv "$(Build.ArtifactStagingDirectory)/${{ parameters.API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH }}/loggers/dataops-appinsights-dev" "$(Build.ArtifactStagingDirectory)/${{ parameters.API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH }}/loggers/dataops-appinsights"
              failOnStderr: false
          - task: Bash@3 
            displayName: Change InstrumentationKey folder name
            inputs:
              targetType: 'inline'
              script: |
                if [[ -d "$(Build.ArtifactStagingDirectory)/${{ parameters.API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH }}/named values/64872de0217d2004e4c3567f" ]]
                then
                  mv "$(Build.ArtifactStagingDirectory)/${{ parameters.API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH }}/named values/64872de0217d2004e4c3567f" "$(Build.ArtifactStagingDirectory)/${{ parameters.API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH }}/named values/application-insights-instrumentation-key"
                fi
              failOnStderr: false

The final directory names are as follows:

artifacts/loggers/dataops-appinsights
artifacts/named values/application-insights-instrumentation-key

the loggerInformation.json in artifacts/loggers/dataops-appinsights folder looks like this.

{
  "properties": {
    "credentials": {
      "instrumentationKey": "{{Logger-Credentials--64872de0217d2004e4c35680}}"
    },
    "isBuffered": true,
    "loggerType": "applicationInsights",
    "resourceId": "/subscriptions/xxx/resourceGroups/<rg-name>/providers/microsoft.insights/components/dataops-appinsights-dev"
  }
}

Expected behavior

The publish pipeline should use the correct application insight resource Id and instrumentation key based on the YAML config override.

Actual behavior

crit: Publisher[0]
      System.InvalidOperationException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/dataops/providers/Microsoft.ApiManagement/service/dataops-st/loggers/dataops-appinsights?api-version=2021-12-01-preview failed with status code 400. Content is '{"error":{"code":"ValidationError","message":"Validation failed for logger-id 'dataops-appinsights'. Exception Received 'Invalid instrumentation key for Application Insights Logger'","details":null}}'.
         at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri)
         at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken)
         at publisher.Program.<>c__DisplayClass12_0.<<GetPutRestResource>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at publisher.Logger.PutLogger(LoggerName loggerName, JsonObject json, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
         at publisher.Logger.<>c__DisplayClass8_0.<<ProcessArtifactsToPut>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
      --- End of stack trace from previous location ---
         at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken)
         at publisher.Logger.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
         at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken)
         at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken)
         at publisher.Publisher.Run(CancellationToken cancellationToken)
         at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken)
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...
fail: Microsoft.Extensions.Hosting.Internal.Host[9]
      BackgroundService failed
      System.InvalidOperationException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/dataops/providers/Microsoft.ApiManagement/service/dataops-st/loggers/dataops-appinsights?api-version=2021-12-01-preview failed with status code 400. Content is '{"error":{"code":"ValidationError","message":"Validation failed for logger-id 'dataops-appinsights'. Exception Received 'Invalid instrumentation key for Application Insights Logger'","details":null}}'.
         at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri)
         at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken)
         at publisher.Program.<>c__DisplayClass12_0.<<GetPutRestResource>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at publisher.Logger.PutLogger(LoggerName loggerName, JsonObject json, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
         at publisher.Logger.<>c__DisplayClass8_0.<<ProcessArtifactsToPut>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
      --- End of stack trace from previous location ---
         at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken)
         at publisher.Logger.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
         at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken)
         at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken)
         at publisher.Publisher.Run(CancellationToken cancellationToken)
         at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken)
         at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
dbug: Microsoft.Extensions.Hosting.Internal.Host[3]

Reproduction Steps

Already explained above.

About this issue

Original URL
State: closed
Created a year ago
Reactions: 1
Comments: 24 (13 by maintainers)

Most upvoted comments

@guythetechie I got the pipeline working now. Here is something that we can add to the wiki

In diagnostics block, you must name it applicationinsights otherwise it will not work. (https://github.com/Azure/apiops/blob/7e5216b5d57ab34ac048f8c15193837a4e805f0a/configuration.prod.yaml#L35)
The way to grab the logger Id is using this API (https://learn.microsoft.com/en-us/rest/api/apimanagement/current-ga/logger/list-by-service?tabs=HTTP). Then, using the try it functionality on the page works great for me.
Make sure that the APIM has permission to access the key vault if you use one for named values before running the pipeline.
You can add a step on the extractor pipeline to rename logger folder name so the name matches your logger block on YAML configuration. Something like this:

- task: Bash@3 
    displayName: Change Logger folder name
    inputs:
      targetType: 'inline'
      script: |
        mv "$(Build.ArtifactStagingDirectory)/${{ parameters.API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH }}/loggers/dataops-appinsights-dev" "$(Build.ArtifactStagingDirectory)/${{ parameters.API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH }}/loggers/dataops-appinsights"

Add displayName on the namedValue block makes it easier to match the configuration.

namedValues:
  - name: application-insights-instrumentation-key
    properties:
      # This value should be passed to loggers[*].properties.credentials.instrumentationKey.
      # Override in configuration here is optional
      displayName: my-display-name
      value: "781e4eac-0445-4b6e-a6db-78dfe19adf6d"

pleelapr on Jul 15, 2023

Looks like the logger worked. Your error is now on the diagnostic resource (LoggerId must be specified). You have to specify the resource ID of your logger, as indicated in our sample configuration.

guythetechie on Jul 12, 2023

@pleelapr let me get back to you on this tomorrow eod. I’m having a weekly sync up with the lead developer.

waelkdouh on Jul 6, 2023

Thanks @waelkdouh! In my case, the issue was that the name of the logger in the configuration file was different from the logger in the repo. Once I fixed the name of the logger in the configuration file to deploy to the second instance, it worked like a charm!

thallesnoce on Jul 6, 2023

App insights can get tricky, so we dedicated a section in our wiki on how to handle it. You can find it here.

waelkdouh on Jul 5, 2023