azure-webjobs-sdk: Support triggering a single function on multiple queues (wildcard queue name)

Example scenario: A webjob that monitors all *-poison queues in a storage account, with a single generic action taken on all (e.g. write queue message to blob for inspection, email/Slack/SMS to ops team to alert of failure).

I’ve looked into implementing this as an extension, but rapidly ended up with 90% of the code being duplication of the core SDK due to much of the infrastructure behind QueueTriggerAttribute being marked as internal.

Would you be willing to consider a pull request that creates a MultipleQueueTriggerAttribute that accepts a regex to match against queue names, creates a listener monitoring all matching queues and adds a queueName binding in addition to the existing bindings?

public static void HandlePoisonQueues(
    [MultipleQueueTriggerAttribute(".*-poison")] string message,
    string queueName,
    [Blob("poison/{queueName}/{id}.txt")] TextWriter blob,
    TextWriter logger)
    logger.WriteLine($"Poison message received on {queueName}")

About this issue

  • Original URL
  • State: open
  • Created 8 years ago
  • Reactions: 31
  • Comments: 15

Most upvoted comments

Has anyone from Microsoft had a look at this, and is it on the backlog?

For massive scale and better tenancy isolation, 1 queue per tenant would be ideal - but that would also require one function binding per queue too, which is far from ideal (and likely to hit some scaling limitation, which are rife throughout all Azure services)

Bueller? At 5 years and ticking, this would be nice to have.

After five years, any progress or update? At least a workaround?

Any updates? Has anyone found a workaround for this?

I need a function to be triggered by multiple queues. Any progress here?

I think if you need this it’s an indication of poor design

On Wed, Jan 25, 2023, 7:46 AM Aaron Schmid @.***> wrote:

Would be kinda cool if we can have this 😃

— Reply to this email directly, view it on GitHub, or unsubscribe . You are receiving this because you commented.Message ID: @.***>

Another interesting and useful scenario:

To overcome the Azure Storage Queue scalability limits more easily.

A single queue can process approximately 2,000 messages (1KB each) per second (each AddMessage, GetMessage, and DeleteMessage count as a message here). If this is insufficient for your application, you should use multiple queues and spread the messages across them.

Defining a Regex on the queue name: [MultipleQueueTriggerAttribute(“orders[0-9]”)] string message

Passsing several queue names: [MultipleQueueTriggerAttribute(“orders1”, “orders2”, “orders3”)] string message