azure-webjobs-sdk: Support triggering a single function on multiple queues (wildcard queue name)

Example scenario: A webjob that monitors all *-poison queues in a storage account, with a single generic action taken on all (e.g. write queue message to blob for inspection, email/Slack/SMS to ops team to alert of failure).

I’ve looked into implementing this as an extension, but rapidly ended up with 90% of the code being duplication of the core SDK due to much of the infrastructure behind QueueTriggerAttribute being marked as internal.

Would you be willing to consider a pull request that creates a MultipleQueueTriggerAttribute that accepts a regex to match against queue names, creates a listener monitoring all matching queues and adds a queueName binding in addition to the existing bindings?

public static void HandlePoisonQueues(
    [MultipleQueueTriggerAttribute(".*-poison")] string message,
    string queueName,
    [Blob("poison/{queueName}/{id}.txt")] TextWriter blob,
    TextWriter logger)
{
    logger.WriteLine($"Poison message received on {queueName}")
    blob.Write(message);
}

About this issue

Original URL
State: open
Created 8 years ago
Reactions: 31
Comments: 15

Most upvoted comments

Has anyone from Microsoft had a look at this, and is it on the backlog?

For massive scale and better tenancy isolation, 1 queue per tenant would be ideal - but that would also require one function binding per queue too, which is far from ideal (and likely to hit some scaling limitation, which are rife throughout all Azure services)

cocowalla on Jan 4, 2020

Bueller? At 5 years and ticking, this would be nice to have.

dotnetnate on Jul 5, 2021

After five years, any progress or update? At least a workaround?

onurleblebici on Mar 15, 2021

Any updates? Has anyone found a workaround for this?

VictorGazzinelli on Mar 27, 2021

I need a function to be triggered by multiple queues. Any progress here?

alexgman on Jul 9, 2019

I think if you need this it’s an indication of poor design

On Wed, Jan 25, 2023, 7:46 AM Aaron Schmid @.***> wrote:

Would be kinda cool if we can have this 😃

— Reply to this email directly, view it on GitHub https://github.com/Azure/azure-webjobs-sdk/issues/808#issuecomment-1403649243, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAIE336I4YAEPARR4WJYWLWUEVFFANCNFSM4COKFDKA . You are receiving this because you commented.Message ID: @.***>

alexgman on Jan 25, 2023

Another interesting and useful scenario:

To overcome the Azure Storage Queue scalability limits more easily.

A single queue can process approximately 2,000 messages (1KB each) per second (each AddMessage, GetMessage, and DeleteMessage count as a message here). If this is insufficient for your application, you should use multiple queues and spread the messages across them.

Defining a Regex on the queue name: [MultipleQueueTriggerAttribute(“orders[0-9]”)] string message

Passsing several queue names: [MultipleQueueTriggerAttribute(“orders1”, “orders2”, “orders3”)] string message

CarlosSardo on Dec 28, 2016