terraform-provider-azurerm: `azurerm_linux_function_app` loses external `WEBSITE_RUN_FROM_PACKAGE` setting, causing the Function App to break
Is there an existing issue for this?
- I have searched the existing issues
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave “+1” or “me too” comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Terraform Version
1.2.1
AzureRM Provider Version
3.8.0
Affected Resource(s)/Data Source(s)
azurerm_linux_function_app
Terraform Configuration Files
resource "azurerm_linux_function_app" "this" {
name = "func-example-dev"
resource_group_name = var.resource_group_name
location = var.location
service_plan_id = var.service_plan_id
storage_account_name = var.storage_account_name
storage_account_access_key = var.storage_account_access_key
https_only = true
builtin_logging_enabled = false
tags = var.tags
identity {
type = "SystemAssigned"
}
site_config {
application_insights_connection_string = var.application_insights_connection_string
http2_enabled = true
application_stack {
node_version = 16
}
}
app_settings = merge(
{ "WEBSITE_MOUNT_ENABLED" : "1" },
var.app_settings
)
lifecycle {
ignore_changes = [
# We tried with and without ignoring changes here, but in both cases adding/removing entries to/from app_settings
# causes WEBSITE_RUN_FROM_PACKAGE to be removed entirely
app_settings["WEBSITE_RUN_FROM_PACKAGE"],
]
}
}
Debug Output/Panic Output
n/a (please let me know if I can provide any helpful output)
Expected Behaviour
Making changes to azurerm_linux_function_app.app_settings
has no effect on any external WEBSITE_RUN_FROM_PACKAGE
entry.
Actual Behaviour
Whenever there are changes to app_settings
within the terraform configuration, the WEBSITE_RUN_FROM_PACKAGE
setting gets removed on the next terraform apply
. This renders the Function App broken, because the deployment package cannot be retrieved.
In particular this happens when deploying with the Azure Function Core Tools using func azure functionapp publish
, since this will publish the package to a storage account and set the WEBSITE_RUN_FROM_PACKAGE
in the Function App’s app_settings
. Upon the next changes in terraform, this setting will be removed and require another explicit deployment using the Azure Function Core Tools to be repaired. In between the Function App is broken.
Unlike mentioned in issue https://github.com/hashicorp/terraform-provider-azurerm/issues/16396 the WEBSITE_RUN_FROM_PACKAGE
setting does not appear in the terraform plan or state at all, but it will be removed whenever there are any other changes in app_settings
.
Steps to Reproduce
- Create the Function App
func-example-dev
usingterraform apply
- Deploy a function using
func azure functionapp publish func-example-dev
(here theWEBSITE_RUN_FROM_PACKAGE
will be set on the Function App’s settings) - Add an entry to the
app_settings
of theazurerm_linux_function_app
resource in terraform - Run
terraform apply
to apply the changes to theapp_settings
(note that the plan will show no changes toWEBSITE_RUN_FROM_PACKAGE
) - The Function App loses its
WEBSITE_RUN_FROM_PACKAGE
setting and becomes unusable
Important Factoids
No response
References
A similar issue has been reported here: https://github.com/hashicorp/terraform-provider-azurerm/issues/16396
The issue was supposedly fixed in this pull request: https://github.com/hashicorp/terraform-provider-azurerm/pull/16641
However, we are still running into the issue, although the behavior seems to have changed a bit (see Actual Behavior
)
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 52
- Comments: 41 (6 by maintainers)
Commits related to this issue
- double the check timeout of deletion of nic to fix #17081 — committed to lonegunmanb/terraform-provider-azurerm by lonegunmanb 2 years ago
I can also confirm this issue, and I think it’s understated how bad this is. Any changes to app_settings silently removes the
WEBSITE_RUN_FROM_PACKAGE
setting, unless it is explicitly stated (which for a lot of users it is not), subsequently breaking the function app. Further I can report that it is not only explicit changes to app_settings that results in the removal, implicit changes, like setting application_insights_connection_string in site_config produces the same behavior.This behavior is not present in the deprecated azurerm_function_app resource.
Looking at the terraform states, I can see that
WEBSITE_RUN_FROM_PACKAGE
is not tracked, which I guess is why it doesn’t show up in the diff, and also why adding the ignore_changes makes no difference.Note that I’m using the latest provider release at the time of writing 3.29.1
I can confirm this is still not working and I would agree this is a major bug. We are deploying function code using DevOps
AzureFunctionApp@1
action, yet Function App itself is deployed using Terraform. We are not specifyingWEBSITE_RUN_FROM_PACKAGE
parameter in there and we also includeapp_settings
inignore_changes
.What I tried today is:
WEBSITE_RUN_FROM_PACKAGE = 1
, I checked it in activity log withing Azure portal once DevOps pipeline was finished)terraform apply
properties.appSettings["WEBSITE_RUN_FROM_PACKAGE"]
has been deletedWhen it comes to versions, I am using following:
Pull request was closed but this is still a critical issue for many of us. Please fix!
Can we have this prioritized guys, please ? We unfortunately hit that bugged yesterday and wiped function on prod environment
ps: linking @tombuildsstuff
This is still open, we currently implementing this provider would be sad if this still breaks the function app
I agree, the issue is quite critical, maybe I was not stern enough in my communication.
We recently updated our
terraform
andazurerm
versions to v1.3.4 and 3.30.0 respectively, and now the previous workaround also does not work reliably anymore, because setting up the Function App for the first time now produces the error that I expected previously:I mentioned in an earlier comment that I didn’t understand how it was working in the first place, but @xiaxyi mentioned that it would work. This is no longer the case, so now there isn’t even a workaround (which was not really a solution in the first place, to be honest).
I can confirm experiencing the same issue mentioned here.
We also tried the lifecycle ignore approach but it does not work as expected.
e.g.
When terraform initially creates the function app it correctly sets an empty string Subsequent github deploy sets the string value to a url Next Terraform change containing an update to app_settings causes the setting to disappear
In other words it’s not being tracked at all that point whereas it should be tracked and ignoring future changes. This is definitely a bug
Regarding the workaround we have in place now: We had to adapt it even further, because in case of an existing Function App that has not been deployed yet, we would still run into error (because
WEBSITE_RUN_FROM_PACKAGE
was not set in the data source’sapp_settings
). For anybody else facing this issue, this is what we currently use:It somehow works, but it still makes me wonder if that is how it’s supposed to be, because it seems quite tedious and error prone. In the terraform plan there still is no mention that any of the app settings would be removed, so without knowing about this issue it can and will break Function App deployments without any notice. We ran into several broken deployments ourselves before we figured out what was actually happening. And we are just using the regular approach of deployment via Azure Functions Core Tools as documented and officially recommended, nothing fancy either.
Hi @xiaxyi , thank you for the suggestion. Adding the property manually is not really feasible, because the value of the setting will change with every
publish
, pointing to a new deployment package. You mentioned that terraform will change/remove settings that aren’t managed, but usually these at least show up when runningterraform plan
- this is not the case in this scenario, theWEBSITE_RUN_FROM_PACKAGE
setting does not appear as an outside change or in any other way to be updated/removed.I had thought of using a data source as well, but that would break the module when attempting to create the resources from scratch, since the data source import will fail in that case (right?). There might be a workaround for using data sources conditionally, but I haven’t tried that yet to be honest, and it seems a bit hacky for something that I thought was expected behavior. I can try this out earliest on Thursday, and get back to you with some details.
Apart from the workaround solution: The source code for the linux_function_app_resource does have a special case for the
WEBSITE_RUN_FROM_PACKAGE
setting, and the comment suggests to cover this exact use case, but I think it’s not working as intended. There was another similar issue a few weeks back, that was supposedly fixed, but I think the behavior just changed a bit with the underlying issue still being present.On a side note: There are several other
app_settings
that are not manually managed inside theapp_settings
block, and they are still present even after runningterraform apply
. I assume this is because they are somehow managed in another way, but couldn’t that be an option for theWEBSITE_RUN_FROM_PACKAGE
, too? Some examples of other properties that our Function App has, that are not part of ourapp_settings
:Over a year later and multiple proposed and submitted solutions it is quite clear that Hashicorp are not interested in supporting us or Azure.
According to @jackofallops
Removal of lines https://github.com/hashicorp/terraform-provider-azurerm/blob/v3.73.0/internal/services/appservice/linux_function_app_resource.go#L1291-L1295 (version tagged) should fix the issue.
I guess this works in the
azurerm_windows_function_app
. So that could be a possible intermediate alternative.I have dozens of functions and I just changed a variable used as an app setting in all of them. The TF run and I really need to emphasise the danger here because only showed the change to the intended setting and yet it took down
every
single
function
because it silently removed the WEBSITE_RUN_FROM_PACKAGE. As others here have said, it was explicitly in the ignore_changes.The PR seems to have gone dark… Am eagerly awaiting the fix please! @jackofallops @xiaxyi
We’ve been facing this issue for a while and thought of using the azapi provider just for the
app_settings
part.And we stumbled upon this example which is indeed fixing the issue.
app_settings
part is now covered byazapi
provider and any change there does not break the existing function code.Some remarks on the linked example code:
terraform plan
; so I would suggest using sensitive function here.app_settings
including secrets; I would suggest to replace["*"]
with[]
here to not show any values at all.Hope this helps …
Actually it’s any changes to the function at all that trigger this.
Also I don’t understand the above discussion. How can it be “by-design” that azurerm removes a vital app setting without expressing that in the plan, nor giving us the ability to ignore changes on that setting?
And if that is “by-design” why does it do that only with the “WEBSITE_RUN_FROM_PACKAGE”? We have several app settings that are set by our code deployment that we can ignore and it works just fine.
We tried setting WEBSITE_RUN_FROM_PACKAGE to “” in app settings and had lifecycle ignore for the setting. Still removed when we applied changes (not even to function) so the ignore failed.
Using the workaround above (merge settings) for now. I would expect the ignore to work at least but it did not so maybe there is abug?
Thank you @athisun, unfortunately I don’t think it’s possible to specify the name of the deployment package when publishing the Function App via the Functions Core Tools, see https://github.com/Azure/azure-functions-core-tools/blob/db2ff784941c3f6241aea996818ba370ea441bd2/src/Azure.Functions.Cli/Actions/AzureActions/PublishFunctionAppAction.cs#L614.
The DevOps task (which we are not using btw) also doesn’t seem to have that option.
Even if it were possible, wouldn’t that restrict us to use a static name for the deployment package? Not sure what the official recommendations are, but at least all Azure tools seem to generate new package names for each deployment, instead of overriding an existing deployment package.
@xiaxyi yes, exactly. In our case the
WEBSITE_RUN_FROM_PACKAGE
setting is managed by Azure Functions Core Tools, which we use to publish our projects to the Function App.Running
terraform plan
will not show any outsideWEBSITE_RUN_FROM_PACKAGE
changes at all (e.g. to apply it with a-refresh-only
), and it will also not show the setting to be removed. I believe this is due to the changes in PR #16641 , but I am not sure.However, when there are any other changes to
app_settings
in terraform, the nextterraform apply
will remove theWEBSITE_RUN_FROM_PACKAGE
setting, which is required to keep the Function App functional.