azure-cli: ARM template should not delete and replace application settings for a function app

Please consider getting a fix in a upcoming milestone, this issue is blocking us from using ARM templates to deploy resources to Azure in our CI/CD pipeline. I started creating ARM templates for our Azure Functions and planned to have the entire infrastructure deployed via ARM template. It is discouraging to see that my first ARM template is running into this issue and I’m worried that every other resource I try to automate with ARM templates will run into similar problems where things just arent idempotent. We have several more App Services with application settings and I imagine we will be running into this problem.

The application deployment should be able to set App Settings without worrying about them being wiped out by the next ARM infrastructure deployment. That is my underlying problem with the current approach.

Also the fact that this isn’t well documented, and that the implementation unexpectedly differs between ‘kinds’ of Microsoft.Web/sites resource deployments (app/functionapp)

Incremental, unfortunately, is a misleading term.

Incremental mode does not apply to properties of a resource, it applies to the resource itself. If a deployment is done in complete mode, that means it will delete resources not declared in the template, but it does not affect how the properties within a resource are handled. That is up to the RP itself (in this case the web RP).

@bim-msft or @Juliehzl - is there someone from the web team we can bring into this?

Same issue here. App Settings in an ARM template are not respecting the incremental nature of ARM template deployments.

@bmoore-msft - in this case my main frustration is around slots.

I have separate pipelines to deploy infrastructure and code (which includes App Settings). When the infrastructure pipeline creates a Function App with no App Settings FUNCTIONS_EXTENSION_VERSION is set to ~1. However when the code pipeline carries out a slot swap the value of FUNCTIONS_EXTENSION_VERSION doesn’t swap.

So this means that I need to set App Settings for the same Function App in two separate pipelines.

At the moment the infrastructure pipeline includes tasks to set that value through a PowerShell script, as the PowerShell cmdlet will PATCH the App Settings. So it’s doable, but more work.

Going forward the dream is to be able to deploy the infrastructure pipeline with some App Settings in the ARM template that are unlikely to change (e.g. APPINSIGHTS_INSTRUMENTATIONKEY, FUNCTIONS_EXTENSION_VERSION, etc.). And then with the code pipeline I can include an ARM template for any other App Settings.

Hope that makes sense.

So the workaround for this issue for me currently is (as suggested by others above): using union to merge existing/runtime settings with the “defaults” provided by resource deployment (in my case via bicep).

Bicep example:

// file: modules\function_app_settings.bicep
param functionAppName string
param currentAppSettings object
param appSettings object

resource siteconfig 'Microsoft.Web/sites/config@2020-06-01' = {
  name: '${functionAppName}/appsettings'
  properties: union(currentAppSettings, appSettings)
}

// file: main.bicep
module functionAppSettings 'modules/function_app_settings.bicep' = {
  name: 'functionAppSettingsDeployment'
  params: {
    functionAppName: functionAppName
    currentAppSettings: list('Microsoft.Web/sites/${functionAppName}/config/appsettings', '2020-06-01').properties
    appSettings: {
      AZURE_KEY_VAULT_URI: 'https://${keyVaultName}${environment().suffixes.keyvaultDns}'
      AzureWebJobsStorage: storage.outputs.storageConnectionString
      FUNCTIONS_EXTENSION_VERSION: '~3'
    }
  }
}

I put together a solution/workaround for this in an ARM template.

1. Remove the appsettings property from the siteConfig in the Microsoft.Web/sites resource
2. Create a nested deployment, which depends on the Microsoft.Web/sites resource, which just has an Outputs section with:

"outputs": {
    "AppSettings": {
        "type": "object",
        "value": "[list(format('{0}/config/appsettings', resourceId('Microsoft.Web/sites', parameters('FunctionAppNameInner'))), '2020-12-01').properties]"
    }
}

1. Add a Microsoft.Web/sites/config resource to the main template which does a union() on the output of the nested template plus whatever default settings you want to have in the template.

Here is the Function app portion of my ARM template:

{
    "apiVersion": "2019-08-01",
    "name": "[parameters('FunctionAppName')]",
    "type": "Microsoft.Web/sites",
    "kind": "functionapp",
    "location": "[parameters('location')]",
    "tags": {},
    "dependsOn": [
        "[concat('Microsoft.Web/serverfarms/', variables('hostingPlanName'))]",
        "[concat('Microsoft.Storage/storageAccounts/', variables('AzureWebJobsStorageAccountName'))]",
        "[parameters('AppInsightsName')]"
    ],
    "identity": {
        "type": "SystemAssigned"
    },
    "properties": {
        "name": "[parameters('FunctionAppName')]",
        "httpsOnly": true,
        "serverFarmId": "[resourceId('Microsoft.Web/serverfarms', variables('hostingPlanName'))]",
        "clientAffinityEnabled": false,
        "siteConfig": {
            "use32BitWorkerProcess": true,
            "powerShellVersion": "~7"
        }
    }
},
{
    "type": "Microsoft.Resources/deployments",
    "apiVersion": "2020-10-01",
    "name": "GetAppSettings",
    "dependsOn": [ "[parameters('FunctionAppName')]" ],
    "properties": {
        "mode": "Incremental",
        "expressionEvaluationOptions": {
            "scope": "Inner"
        },
        "parameters": {
            "FunctionAppNameInner": {
                "value": "[parameters('FunctionAppName')]"
            }
        },
        "template": {
            "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
            "contentVersion": "1.0.0.0",
            "parameters": {
                "FunctionAppNameInner": {
                    "type": "string"
                }
            },
            "resources": [],
            "outputs": {
                "AppSettings": {
                    "type": "object",
                    "value": "[list(format('{0}/config/appsettings', resourceId('Microsoft.Web/sites', parameters('FunctionAppNameInner'))), '2020-12-01').properties]"
                }
            }
        }
    }
},
{
    "type": "Microsoft.Web/sites/config",
    "name": "[concat(parameters('FunctionAppName'), '/appsettings')]",
    "apiVersion": "2019-08-01",
    "dependsOn": [ "GetAppSettings" ],
    "properties": "[union(reference('GetAppSettings').outputs.AppSettings.value, createObject(
                'FUNCTIONS_EXTENSION_VERSION', '~3',
                'FUNCTIONS_WORKER_RUNTIME', 'powershell',
                'APPINSIGHTS_INSTRUMENTATIONKEY', reference(parameters('AppInsightsName'), '2020-02-02-preview').InstrumentationKey,
                'AzureWebJobsStorage', concat('DefaultEndpointsProtocol=https;AccountName=',variables('AzureWebJobsStorageAccountName'),';AccountKey=',listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('AzureWebJobsStorageAccountName')), '2019-06-01').keys[0].value,';EndpointSuffix=','core.windows.net'),
                'WEBSITE_CONTENTAZUREFILECONNECTIONSTRING', concat('DefaultEndpointsProtocol=https;AccountName=',variables('AzureWebJobsStorageAccountName'),';AccountKey=',listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('AzureWebJobsStorageAccountName')), '2019-06-01').keys[0].value,';EndpointSuffix=','core.windows.net'),
                'WEBSITE_CONTENTSHARE', toLower(parameters('FunctionAppName')),
                'FUNCTIONS_WORKER_PROCESS_COUNT', '1',
                'PSWorkerInProcConcurrencyUpperBound', '10',
                'WEBSITE_RUN_FROM_PACKAGE', '1'
                    ))]"
},

What is the state of this issue? Will there be an option so that patching appsettings without wiping all other appsettings work in the near future?

In my case and I assume in others during deployment not all configuration settings are known. Thats why it is impossible to put them into the ARM template.

Hey @ajklotz, do you think this is how the product works by design?

See this: https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/deployment-modes#incremental-mode

In the note there you can read that:

…"When redeploying an existing resource in incremental mode, all properties are reapplied. The properties aren’t incrementally added. A common misunderstanding is to think properties that aren’t specified in the template are left unchanged. If you don’t specify certain properties, Resource Manager interprets the deployment as overwriting those values. Properties that aren’t included in the template are reset to the default values. Specify all non-default values for the resource, not just the ones you’re updating. The resource definition in the template always contains the final state of the resource. It can’t represent a partial update to an existing resource.

In rare cases, properties that you specify for a resource are actually implemented as a child resource. For example, when you provide site configuration values for a web app, those values are implemented in the child resource type Microsoft.Web/sites/config. If you redeploy the web app and specify an empty object for the site configuration values, the child resource isn’t updated. However, if you provide new site configuration values, the child resource type is updated…"

Please let me know if my statement above reflects the issue you are having or if I am confused. Thanks.

@alex-frankel I would like to confirm with you the key issues: What you mean is that the incremental mode and complete mode of the ARM template only affects whether the undefined resources in template are deleted, but they do not take effect on the property settings in resources defined in the template. These original property settings will be directly overwritten by new properties in the parameter file when deployed in either mode, right?

I have the same problem. It is not possible to create an arm template for a function app with a event triggered function, which needs event topic or event subscription related app settings. This is because the event topic and subscription needs to be created before the function app, which in turn needs to be created before the event topic and subscription to add the event specific app settings… infinite loop.

My proposal is that Microsoft.Web/sites/config appsettings dont overwrite the complete appsettings list, but rather add the specified appsettings from the nested resource. That would solve the problem, and would also follow the same approach as key vault access policies do (which only adds new policies rather than overwriting the existing list).

Example key vault access policy resource:

        {
            "type": "Microsoft.KeyVault/vaults/accessPolicies",
            "apiVersion": "2021-10-01",
            "name": "[concat(variables('keyVaultName'), '/add')]",
            "dependsOn": [
                "[resourceId('Microsoft.KeyVault/vaults', variables('keyVaultName'))]",
                "[resourceId('Microsoft.Web/sites', variables('functionAppName'))]"
            ],
            "properties": {
                "accessPolicies": [
                    {
                        "tenantId": "[parameters('tenantId')]",
                        "objectId": "[reference(resourceId('Microsoft.Web/sites', variables('functionAppName')),'2020-06-01', 'Full').identity.principalId]",
                        "permissions": {
                            "secrets": [
                                "Get"
                            ]
                        }
                    }
                ]
            }
        }

Sorry to butt-in with something tangential but I’ve been burned by this, too (expecting a diff to be applied). I think partly because a) I assumed such behaviour as “obvious” b) I assumed any behaviour would be universal c) my mental model is from my previous life in IT configuring Windows GPOs, and the Get-Test-Set model of PowerShell DSC.

What I think could help is if, like Windows, the Azure portal were to either lock or warn the user that settings have been set by infra-as-code and changing them in the portal may mean they’ll be overwritten. This would prevent accidents but also help sign-post how it works.

For example, if some parts of a resource are overwritten by the template as a “block unit” then that whole editor UI could be locked to indicate this behaviour. And where individual settings are used and unspecified settings are retained as-is, then this too can be communicated by locking just those individual settings.

This is still an issue for us. Any updates to resolve?

@bmoore-msft - We have two pipelines and MUST set some App Settings in both due to our use of slots. Slot swapping is working, but using slots complicates our procedure because some App Settings are treated as slot-settings even thought they are not identified as such, e.g. FUNCTIONS_EXTENSION_VERSION.

(Apologies to anyone who falls asleep attempting to read this…)

How we do things at the moment

1. We run a pipeline to deploy our Function App resource (production and slot), and we MUST set FUNCTIONS_EXTENSION_VERSION to ~3 for production. If we don’t, it’s set to ~1 by default, and as mentioned above, a slot swap does not swap the value of that setting even though we don’t identify it as a slot-setting.
2. We run a pipeline to deploy our code to the Function App slot. At the same time we update all App Setting in the slot. Finally we swap production and slot.

The issue with this is that in step 1, we only really need to set FUNCTIONS_EXTENSION_VERSION in production on the first deployment, but as there is no way to test if a resource exists through an ARM template that was always unknown and so we always had to set FUNCTIONS_EXTENSION_VERSION just in case. This mean that all other App Settings were wiped out due to the PUT nature of appSettings in an ARM template, necessitating a deployment of the code after any infrastructure update.

How we can (hopefully) now do things

Now that I know of the list() function, we can make things simpler. In both steps we can use list() to pull back the existing App Settings and then using union() to merge in new/updated settings. This means that an update of our infrastructure won’t necessitate a redeployment of our code.

How things can be even easier

If we could chose to PATCH rather than PUT App Settings, we wouldn’t even have to worry about the list() and union() functions.

Hope that helps.

To work a bit around this you can use the ‘union’ function in ARM to combine sets of configuration settings. First a param to provide an array with settings:

      "appProperties": { 
            "type": "Array",
            "metadata": {
                "description": "Settings."
            }
        }

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.1",
    "parameters": {
        "appProperties": {
            "value": [
                {
                    "name": "Setting3",
                    "value": "Value for setting3"
                },
                {
                    "name": "Setting4",
                    "value": "Value for setting4"
                }
            ]
        }
    }
}

These params can be passed without a param file as well.

Next: in variables the default section and use the ‘union’ function to combine the param settings and the variable settings.

"variables": {
        "defaultProperties": [
            {
                "name": "Setting1",
                "value": "[parameters('Setting1')]"
            },
            {
                "name": "Setting2",
                "value": "Value for Setting2"
            }
        ],
        "appProperties": "[union(variables('defaultProperties'), parameters('appProperties'))]"

Deploy :

"siteConfig": {
                    "appSettings": "[variables('appProperties')]"
                

These original property settings will be directly overwritten by new properties in the parameter file when deployed in either mode, right?

In most cases that is correct, but some RPs will treat this as a PATCH request which will do a strategic merge instead of overwriting. This is totally up to the RP teams though.

The rest of what you said is correct.

@bmoore-msft I don’t get any errors. After the deployment completes, I just notice that under the deployment tab for the resource group the module functionAppSettingsDeployment didn’t run (or even shows up).

if I comment the currentAppSettings in the module (and set a default value for currentAppSettings in the modules\function_app_settings.bicep file) then it works.

@bkarstaedt I’m using the same bicep version. The only difference that I can think of is that I’m deploying at subscription level: az deployment sub create

edit: I managed to get it working. it is indeed the target scope difference. I needed to add the full resource id into the list function to make it work

Which brings me to another problem. I cannot use any referencing in the list function… list('${webapp.id}/config/appsettings', '2020-06-01').properties is not possible.

The subscription reference works. list('/subscriptions/${subscription().subscriptionId}/resourceGroups/<nameOfRG>/providers/Microsoft.Web/sites/<nameOfSite>/config/appsettings', '2020-06-01').properties

But that is the only one. Which means I need to hard code the resource id in my bicep file. Which is a bit of a bummer

edit2: This also gives issues when your web application doesn’t exist yet. As the list function checks the resource id at the start of the deployment and deploying both the site + the app settings will always fail the first time mentioning that the resource ID could not be found.

@dgard1981 I agree that there needs to be a way to do incremental updates without wiping the existing settings. But, as for the example you give for FUNCTIONS_EXTENSION_VERSION, it should never be swapped. We have to go the extra mile for our deployments as well and make sure to include it on our config settings in our ARM template. It really should be a property within Microsoft.Web/sites/properties/siteConfig rather an appSetting.

Good shout @alex-frankel. There is a list endpoint for Microsoft.Web/sites/config that I can use to get App Settings.

It works in an ARM template -

"appSettings": {
  "type": "object",
  "value": "[list(format('{0}/config/appsettings', resourceId('Microsoft.Web/sites', parameters('functionApp').name)), '2020-06-01')]"
}

But currently fails in Bicep. I created an issue in that repo.

Error BCP082: The name “list” does not exist in the current context. Did you mean “last”?

output appSettings object = list(format('{0}/config/appsettings', functionAppResource.id), functionAppResource.apiVersion)

I think the issue in this particular case @dgard1981 is that the Web RP treats appSettings as a secret, so it doesn’t support simple GETs (which is what the reference() function is doing).

There may be a list* POST API for retrieving this, but I don’t know off hand. Worth looking more closely here: https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/template-functions-resource?tabs=json#list

@bmoore-msft may also know a better way to deal with this

Resource Provider - the first part of the resourceType (e.g. Web, Compute, Sql, etc.).

I’m observing back and forth conversation. Can you define “RP”? Thanks

The same goes for connection strings.