azure-functions-host: ServiceBus trigger with managed identity doesn't wake up consumption plan function apps when Connection isn't explicitly defined

This is for in-process v4 function apps, I’m unsure whether this also applies to the isolated variant.

When a Service Bus trigger is defined with a connection via managed identity, the function app doesn’t get woken up by Azure when the Connection property is not defined in the trigger attribute, despite the trigger working fine when the function app is active.

Investigative information

Timestamp: 2022-03-22T02:58:56.0115521Z
Invocation ID: 6309186b-c3f9-43f1-abeb-6f6385cd7c10 (one of many)
Region: West Europe

Repro steps

Create a function app with a service bus trigger via managed identity without specifying the Connection property
Publish to Azure
Wait for the function app to go inactive
Post a message to the service bus queue

Expected behavior

The function app should get woken up and the service bus triggered function should get triggered. As far as I know, the Connection property was never mandatory for triggers that use connection strings, so this feels like an invisible breaking change when moving to managed identity based connections.

Actual behavior

The function app does not get woken up, and the service bus triggered function is not triggered.

Known workarounds

Define the Connection property that points to the name of the configuration (and gets resolved to <ConnectionName>__fullyQualifiedNamespace).

Related information

I’ve noticed that the generated function.json only includes the connection field if the Connection property is defined. This makes me believe that the default value “ServiceBus” is only considered in the functions runtime, and not in the Azure scale controller for function apps.

Source

[FunctionName("ServiceBusTest")]
public Task Run([ServiceBusTrigger("queue-name")] QueueMessage message)
{
    // This doesn't get triggered if the function app is inactive
}

The second trigger generates a function.json file with the connection property, because it’s now explicitly defined:

[FunctionName("ServiceBusTest2")]
public Task Run([ServiceBusTrigger("queue-name", Connection = "ServiceBus")] QueueMessage message)
{
    // This does get triggered if the function app is inactive
}

About this issue

Original URL
State: open
Created 2 years ago
Comments: 21 (8 by maintainers)

Most upvoted comments

I’ve tried to summarize my experiences (and this GitHub issue) in a blog post. Might be of use to someone stumbling into this issue.

hbrotan on Aug 22, 2022

Hi @fabiocav I have a PR for the internal repo and work item for tracking this work item. Once the PR has been merged and new Scale Controller has been released that includes the fix, I’ll let you guys know.

TsuyoshiUshio on Nov 2, 2022

I believe I have resolved my issue. Details below.

The scale controller (which us end users have no control over from what I can tell) is responsible for waking the JobHost up to consume messages when a service bus queue has any messages.

I’ve added the Azure Service Bus Data Owner role (along w/ Azure Service Bus Data Reader) to my function app’s managed identity (which it didn’t have before). I am hypothesizing that the scale controller component runs under this managed identity and needs visibility into the queue length to determine whether to spin the function app up or not. Queue length cannot be looked at without the Azure Service Bus Data Owner role. Can anyone confirm that this is how the scale controller works? I’m not 100% on if this is part of what resolved my situation but I did add this role to my function app.
This one I’m sure about. My binding for the queue name on my ServiceBusTrigger uses a hierarchical config setting.

public async Task Run([ServiceBusTrigger("%ServiceBus:QueueName:Email%", Connection = "ServiceBusConnection")] string queueMessage)

but my app settings use different hierarchical separators:

ServiceBus__QueueName__Email (no colons)

The scale controller was trying to resolve the queue name by scanning the ServiceBusTrigger attribute in my code (via assembly scanning I’m guessing) and seeing “ServiceBus:QueueName:Email” in that binding, and then trying to resolve “ServiceBus:QueueName:Email” in my app settings but since my app settings use different separators the latter fails.

Long story short: Using consistent hierarchical config separators for both app settings and ServiceBusTrigger binding allows the scale controller to resolve the queue name and start waking up the JobHost again to consume messages.

Super helpful note if you’re experiencing issues with your function app waking up for ServiceBusTrigger functions.

Scale controller logging can be enabled using the following as an app setting for the function app. See here for more info on configuring scale controller logging.

SCALE_CONTROLLER_LOGGING_ENABLED=AppInsights:Verbose

After that, you can see these emitted logs by opening log analytics for your app insights instance w/ this query (making sure the scope of your log query is the app insights instance connected to your function app)

traces
| where customDimensions.Category == "ScaleControllerLogs"

This might reveal exceptions happening when the scale controller is trying to parse out your ServiceBusTrigger function binding. If it can’t parse your binding it won’t know what queue to look at in order to decide whether to wake your JobHost up to consume received messages.

Could be solution for #7762 as well

boylec on Aug 4, 2022

@TsuyoshiUshio any updates on the fix for this?

liliankasem on Mar 8, 2023

@boylec It seems to be working now, using User-assigned Managed Identity 🙏
I haven’t done any changes, so a bit unsure why it didn’t work three days ago, but I’ll take it.

Seems like the solution was like you said: Adding the Azure Service Bus Data Owner for the Managed Identity (I added it on the namespace /root level). Now, messages are consumed after the function goes idle. It takes about a minute before the message is consumed, which seems a bit long, but I guess this is a known issue for consumption plans.

NB! It did not work when adding the Azure Service Bus Data Owner for the Managed Identity on the queue itself, it needed to be on the Service Bus level.

@brettsam These docs vaguely describes the role requirements, but no mention of the scale controller or why/where the Data Owner role is required. Also, there’s no mentioning of roles or required config for User-assigned Managed Identity in the SB extension docs.

hbrotan on Aug 11, 2022

@fabiocav It would be really useful to get a comment from MS about this issue. A function not picking up messages from ServiceBus after going idle is a big showstopper for us.

hbrotan on Aug 8, 2022

@boylec Still doesn’t work for me, using User-Assigned Managed Identity. The MI has Azure Service Bus Data Receiver for the queue it’s suppose to read from, and Azure Service Bus Data Owner for the service bus, so that should be sufficient.

hbrotan on Aug 8, 2022

@boylec I’m experiencing the same issue as you:

my function idles and JobHost goes to sleep (because I’m on Consumption plan) the next message in the queue does not wake the JobHost back

I’m trying to use an User Assigned Managed Identity, with connection string: "ServiceBusConnection__fullyQualifiedNamespace"

@boylec Did you use an User Assigned Managed Identity (UAMI) or System Assigned Managed Identity (SAMI)?
While using the UAMI, the scale controller gives the following warning: [ManagedIdentity] SystemAssigned Managed Identity not found, but the chosen connection method requires one. If I add ServiceBusConnection__credential and ServiceBusConnection__clientId to application settings, the ScaleController understands that it needs to use the UAMI.
I’ve given the UAMI the role Azure Service Bus Data Owner on the subsciption level.

The scale controller logs: [ManagedIdentity] Created NamespaceManager with ManagedIdentity [ManagedIdentity] Created QueueClient with ManagedIdentity This seems legit.

But the problem still exists -> The messages received when the function is idle will not be processed.

According to the docs (Azure WebJobs Service Bus client library for .NET) this should be straightforward, but there’s no mention of scale controller, roles etc.

Can anyone shed a light on this?

hbrotan on Aug 4, 2022

I am experiencing the same issue, but the work around does not work for me. Any word on a resolution to the issue?

Xodust on Jun 15, 2022

Hi @Archomeda , Thank you for your feedback! We will check for the possibilities internally and update you with the findings.

v-bbalaiagar on Apr 19, 2022