azure-functions-host: Restarted host is not picking up newly-deployed assemblies
Spawning this from #1690, as it is a separate issue.
While investigating #1690, I noticed that a customer was seeing ~20 simultaneous requests to shutdown the host, yet the process never recycled. The newly deployed assembly was never loaded so the site had to be restarted to pick up the changes.
Details on finding site name:
| where PreciseTimeStamp between (datetime(2018-03-08 11:00) .. datetime(2018-03-08 12:10))
| where Summary contains "c67d3f04-1134-4b2d-a4c3-266878710784"
And then for seeing the full timeframe of this occurrence:
| where PreciseTimeStamp between (datetime(2018-03-08 11:00) .. datetime(2018-03-08 12:10))
| where AppName == "{sitenamefoundabove}"
| project PreciseTimeStamp, Pid, Details, Summary, FunctionName
You’ll see that an environment restart was requested around 11:05, yet the process id never changed. That didn’t happen until the next request around 12:05, at which point the host shuts down and the process id changes.
@simonness, FYI
About this issue
- Original URL
- State: open
- Created 6 years ago
- Reactions: 1
- Comments: 37 (21 by maintainers)
@petro2050 Thanks for this additional info. It is possible for multiple workers to be assigned, even when load is low. This sometimes happens when we are running a platform upgrade. We start running your app on another machine so that we can unload it from the existing machine, and we do this with overlap so there is no downtime. I think what was happening here is as you said, an older version of the app running in parallel with the new version.
The fact that even after waiting 10 minutes, the old version was still running and you had to do a restart is suspicious. It sounds like something did not work right. But this is different to the problem I was worried about that can happen very occasionally with deployments that are not based on a package, where the app picks up only some of the file changes.
As an update to my last comment, this seems to be happening pretty consistently on our end (function deployment requires a manual restart in order for new code to run).
Just a heads up that we will probably be submitting a formal support request for this today to see if we can get some traction as perhaps it’s something we are doing wrong since no one else appears to be having this issue.
@jimmcslim could you please open a new issue with those details? It would be good to track this separately as it is targeting V2 and the root cause is not related to the what this issue is originally tracking.
Yeah I don’t think your restart went through. I can’t find a record of it in our logs. I think this deployment shows that our 500ms “wait for more files” logic needs to be relaxed slightly. I filed a separate issue for that: https://github.com/Azure/azure-functions-host/issues/3818
I’m re-titling this as I misunderstood how that API worked. It looks like the shutdown is happening – but rather than recycling the process, it recycles the AppDomain. But that doesn’t explain why we see apps occasionally using old assemblies after an AppDomain recycle.
We’ve had one internal report of this as well, and there are likely others that just don’t realize it.
There’s something fishy going on there.