terraform-provider-azurerm: `azurerm_linux_function_app` loses external `WEBSITE_RUN_FROM_PACKAGE` setting, causing the Function App to break

Is there an existing issue for this?

  • I have searched the existing issues

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave “+1” or “me too” comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.2.1

AzureRM Provider Version

3.8.0

Affected Resource(s)/Data Source(s)

azurerm_linux_function_app

Terraform Configuration Files

resource "azurerm_linux_function_app" "this" {
  name                       = "func-example-dev"
  resource_group_name        = var.resource_group_name
  location                   = var.location
  service_plan_id            = var.service_plan_id
  storage_account_name       = var.storage_account_name
  storage_account_access_key = var.storage_account_access_key
  https_only                 = true
  builtin_logging_enabled    = false
  tags                       = var.tags

  identity {
    type = "SystemAssigned"
  }

  site_config {
    application_insights_connection_string = var.application_insights_connection_string
    http2_enabled                          = true

    application_stack {
      node_version = 16
    }
  }

  app_settings = merge(
    { "WEBSITE_MOUNT_ENABLED" : "1" },
    var.app_settings
  )

  lifecycle {
    ignore_changes = [
      # We tried with and without ignoring changes here, but in both cases adding/removing entries to/from app_settings
      # causes WEBSITE_RUN_FROM_PACKAGE to be removed entirely
      app_settings["WEBSITE_RUN_FROM_PACKAGE"],
    ]
  }
}

Debug Output/Panic Output

n/a (please let me know if I can provide any helpful output)

Expected Behaviour

Making changes to azurerm_linux_function_app.app_settings has no effect on any external WEBSITE_RUN_FROM_PACKAGE entry.

Actual Behaviour

Whenever there are changes to app_settings within the terraform configuration, the WEBSITE_RUN_FROM_PACKAGE setting gets removed on the next terraform apply. This renders the Function App broken, because the deployment package cannot be retrieved.

In particular this happens when deploying with the Azure Function Core Tools using func azure functionapp publish, since this will publish the package to a storage account and set the WEBSITE_RUN_FROM_PACKAGE in the Function App’s app_settings. Upon the next changes in terraform, this setting will be removed and require another explicit deployment using the Azure Function Core Tools to be repaired. In between the Function App is broken.

Unlike mentioned in issue https://github.com/hashicorp/terraform-provider-azurerm/issues/16396 the WEBSITE_RUN_FROM_PACKAGE setting does not appear in the terraform plan or state at all, but it will be removed whenever there are any other changes in app_settings.

Steps to Reproduce

  1. Create the Function App func-example-dev using terraform apply
  2. Deploy a function using func azure functionapp publish func-example-dev (here the WEBSITE_RUN_FROM_PACKAGE will be set on the Function App’s settings)
  3. Add an entry to the app_settings of the azurerm_linux_function_app resource in terraform
  4. Run terraform apply to apply the changes to the app_settings (note that the plan will show no changes to WEBSITE_RUN_FROM_PACKAGE)
  5. The Function App loses its WEBSITE_RUN_FROM_PACKAGE setting and becomes unusable

Important Factoids

No response

References

A similar issue has been reported here: https://github.com/hashicorp/terraform-provider-azurerm/issues/16396

The issue was supposedly fixed in this pull request: https://github.com/hashicorp/terraform-provider-azurerm/pull/16641

However, we are still running into the issue, although the behavior seems to have changed a bit (see Actual Behavior)

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 52
  • Comments: 41 (6 by maintainers)

Commits related to this issue

Most upvoted comments

I can also confirm this issue, and I think it’s understated how bad this is. Any changes to app_settings silently removes the WEBSITE_RUN_FROM_PACKAGE setting, unless it is explicitly stated (which for a lot of users it is not), subsequently breaking the function app. Further I can report that it is not only explicit changes to app_settings that results in the removal, implicit changes, like setting application_insights_connection_string in site_config produces the same behavior.

Terraform will perform the following actions:

  # azurerm_linux_function_app_slot.xxxxxxxxxxxxxxx will be updated in-place
  ~ resource "azurerm_linux_function_app_slot" "xxxxxxxxxxxxxx" {
        id                                 = "xxxxxxxxxxxxxxxxxxxxxxxxxxx"
        name                               = "staged"
        tags                               = {
            "environment" = "stage"
        }
        # (25 unchanged attributes hidden)



      ~ site_config {
          - application_insights_connection_string        = (sensitive value)
            # (33 unchanged attributes hidden)

            # (1 unchanged block hidden)
        }
        # (2 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

image

This behavior is not present in the deprecated azurerm_function_app resource.

Looking at the terraform states, I can see that WEBSITE_RUN_FROM_PACKAGE is not tracked, which I guess is why it doesn’t show up in the diff, and also why adding the ignore_changes makes no difference.

Note that I’m using the latest provider release at the time of writing 3.29.1

I can confirm this is still not working and I would agree this is a major bug. We are deploying function code using DevOps AzureFunctionApp@1 action, yet Function App itself is deployed using Terraform. We are not specifying WEBSITE_RUN_FROM_PACKAGE parameter in there and we also include app_settings in ignore_changes.

  lifecycle {
    ignore_changes = [
      app_settings,
      sticky_settings,
      tags["hidden-link: /app-insights-conn-string"],
      tags["hidden-link: /app-insights-instrumentation-key"],
      tags["hidden-link: /app-insights-resource-id"]
    ]
  }

What I tried today is:

  • deploy function code using DevOps action (this adds WEBSITE_RUN_FROM_PACKAGE = 1, I checked it in activity log withing Azure portal once DevOps pipeline was finished)
  • run terraform apply
  • check changes to function - only reported change was following
~ site_config {
            # (34 unchanged attributes hidden)

          + cors {
              + support_credentials = false
            }

            # (1 unchanged block hidden)
        }
  • check Function app activity log within Azure portal, where you can see that properties.appSettings["WEBSITE_RUN_FROM_PACKAGE"] has been deleted
  • deployed function is gone

When it comes to versions, I am using following:

  • Terraform v1.3.7
  • Azurerm Terraform provider v3.41.0

Pull request was closed but this is still a critical issue for many of us. Please fix!

Can we have this prioritized guys, please ? We unfortunately hit that bugged yesterday and wiped function on prod environment

ps: linking @tombuildsstuff

This is still open, we currently implementing this provider would be sad if this still breaks the function app

I agree, the issue is quite critical, maybe I was not stern enough in my communication.

We recently updated our terraform and azurerm versions to v1.3.4 and 3.30.0 respectively, and now the previous workaround also does not work reliably anymore, because setting up the Function App for the first time now produces the error that I expected previously:

│ Error: Linux Function App: (Site Name "func-foo" / Resource Group "rg-foo") not found
│
│   in data "azurerm_linux_function_app" "existing":
│   33: data "azurerm_linux_function_app" "existing" {
│
│ Linux Function App: (Site Name "func-foo" / Resource Group "rg-foo") not found

I mentioned in an earlier comment that I didn’t understand how it was working in the first place, but @xiaxyi mentioned that it would work. This is no longer the case, so now there isn’t even a workaround (which was not really a solution in the first place, to be honest).

I can confirm experiencing the same issue mentioned here.

We also tried the lifecycle ignore approach but it does not work as expected.

e.g.

app_settings = {
    WEBSITE_ENABLE_SYNC_UPDATE_SITE = "true"
    WEBSITE_RUN_FROM_PACKAGE        = "" # This value is irrelevant and will be populated correctly by subsequent Function app code deployments (github) which is why we need to init it here and ignore it below
  }

  lifecycle {
    ignore_changes = [
      app_settings["WEBSITE_RUN_FROM_PACKAGE"]
    ]
  }

When terraform initially creates the function app it correctly sets an empty string Subsequent github deploy sets the string value to a url Next Terraform change containing an update to app_settings causes the setting to disappear

In other words it’s not being tracked at all that point whereas it should be tracked and ignoring future changes. This is definitely a bug

Regarding the workaround we have in place now: We had to adapt it even further, because in case of an existing Function App that has not been deployed yet, we would still run into error (because WEBSITE_RUN_FROM_PACKAGE was not set in the data source’s app_settings). For anybody else facing this issue, this is what we currently use:

data "azurerm_linux_function_app" "existing" {
  name                = "func-example"
  resource_group_name = var.resource_group_name
}

locals {
  existing_app_settings = coalesce(data.azurerm_linux_function_app.existing.app_settings, {})
}

resource "azurerm_linux_function_app" "example" {
  name                       = "func-example"
  ...
  app_settings = merge(
    {
      "WEBSITE_MOUNT_ENABLED"    = lookup(local.existing_app_settings, "WEBSITE_MOUNT_ENABLED", null)
      "WEBSITE_RUN_FROM_PACKAGE" = lookup(local.existing_app_settings, "WEBSITE_RUN_FROM_PACKAGE", null)
    },
    var.app_settings,
  )
}

It somehow works, but it still makes me wonder if that is how it’s supposed to be, because it seems quite tedious and error prone. In the terraform plan there still is no mention that any of the app settings would be removed, so without knowing about this issue it can and will break Function App deployments without any notice. We ran into several broken deployments ourselves before we figured out what was actually happening. And we are just using the regular approach of deployment via Azure Functions Core Tools as documented and officially recommended, nothing fancy either.

Hi @xiaxyi , thank you for the suggestion. Adding the property manually is not really feasible, because the value of the setting will change with every publish, pointing to a new deployment package. You mentioned that terraform will change/remove settings that aren’t managed, but usually these at least show up when running terraform plan - this is not the case in this scenario, the WEBSITE_RUN_FROM_PACKAGE setting does not appear as an outside change or in any other way to be updated/removed.

I had thought of using a data source as well, but that would break the module when attempting to create the resources from scratch, since the data source import will fail in that case (right?). There might be a workaround for using data sources conditionally, but I haven’t tried that yet to be honest, and it seems a bit hacky for something that I thought was expected behavior. I can try this out earliest on Thursday, and get back to you with some details.

Apart from the workaround solution: The source code for the linux_function_app_resource does have a special case for the WEBSITE_RUN_FROM_PACKAGE setting, and the comment suggests to cover this exact use case, but I think it’s not working as intended. There was another similar issue a few weeks back, that was supposedly fixed, but I think the behavior just changed a bit with the underlying issue still being present.

On a side note: There are several other app_settings that are not manually managed inside the app_settings block, and they are still present even after running terraform apply. I assume this is because they are somehow managed in another way, but couldn’t that be an option for the WEBSITE_RUN_FROM_PACKAGE, too? Some examples of other properties that our Function App has, that are not part of our app_settings: grafik

Can we have this prioritized guys, please ? We unfortunately hit that bugged yesterday and wiped function on prod environment

Over a year later and multiple proposed and submitted solutions it is quite clear that Hashicorp are not interested in supporting us or Azure.

According to @jackofallops

Removal of lines https://github.com/hashicorp/terraform-provider-azurerm/blob/v3.73.0/internal/services/appservice/linux_function_app_resource.go#L1291-L1295 (version tagged) should fix the issue.

I guess this works in the azurerm_windows_function_app. So that could be a possible intermediate alternative.

I have dozens of functions and I just changed a variable used as an app setting in all of them. The TF run and I really need to emphasise the danger here because only showed the change to the intended setting and yet it took down every single function because it silently removed the WEBSITE_RUN_FROM_PACKAGE. As others here have said, it was explicitly in the ignore_changes.

  • It should ABSOLUTELY show in the plan that it’s going to remove it
  • When ignore_changes is present it should ignore the setting

The PR seems to have gone dark… Am eagerly awaiting the fix please! @jackofallops @xiaxyi

We’ve been facing this issue for a while and thought of using the azapi provider just for the app_settings part.

And we stumbled upon this example which is indeed fixing the issue.

app_settings part is now covered by azapi provider and any change there does not break the existing function code.

Some remarks on the linked example code:

resource_id = "${azurerm_linux_function_app.test.id}/config/appsettings"
  • The properties might contain secrets which would be disclosed on a terraform plan; so I would suggest using sensitive function here.
  • Similar issue is on the response_export_values which is basically showing the whole app_settings including secrets; I would suggest to replace ["*"] with [] here to not show any values at all.

Hope this helps …

Actually it’s any changes to the function at all that trigger this.

Also I don’t understand the above discussion. How can it be “by-design” that azurerm removes a vital app setting without expressing that in the plan, nor giving us the ability to ignore changes on that setting?

And if that is “by-design” why does it do that only with the “WEBSITE_RUN_FROM_PACKAGE”? We have several app settings that are set by our code deployment that we can ignore and it works just fine.

We tried setting WEBSITE_RUN_FROM_PACKAGE to “” in app settings and had lifecycle ignore for the setting. Still removed when we applied changes (not even to function) so the ignore failed.

Using the workaround above (merge settings) for now. I would expect the ignore to work at least but it did not so maybe there is abug?

Thank you @athisun, unfortunately I don’t think it’s possible to specify the name of the deployment package when publishing the Function App via the Functions Core Tools, see https://github.com/Azure/azure-functions-core-tools/blob/db2ff784941c3f6241aea996818ba370ea441bd2/src/Azure.Functions.Cli/Actions/AzureActions/PublishFunctionAppAction.cs#L614.

The DevOps task (which we are not using btw) also doesn’t seem to have that option.

Even if it were possible, wouldn’t that restrict us to use a static name for the deployment package? Not sure what the official recommendations are, but at least all Azure tools seem to generate new package names for each deployment, instead of overriding an existing deployment package.

@xiaxyi yes, exactly. In our case the WEBSITE_RUN_FROM_PACKAGE setting is managed by Azure Functions Core Tools, which we use to publish our projects to the Function App.

Running terraform plan will not show any outside WEBSITE_RUN_FROM_PACKAGE changes at all (e.g. to apply it with a -refresh-only), and it will also not show the setting to be removed. I believe this is due to the changes in PR #16641 , but I am not sure.

However, when there are any other changes to app_settings in terraform, the next terraform apply will remove the WEBSITE_RUN_FROM_PACKAGE setting, which is required to keep the Function App functional.