dapr: pubsub retry not working

evnrionment machine: mac M1 docker image: daprio/daprd:edge, build by docker-compose component: rabbitmq

problem when a service subscribe the topic message and return the status code 500, i expect the dapr sidecar send the message to the service again, until it reach the backOffMaxRetries。 however,dapr sidecar only send the message once, never retry again. that confuse me. maby this is not a bug, just caused by wrong configuration, can any one help me?

configuration pubsub.yaml

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: rabbitmq-pubsub
  namespace: default
spec:
  type: pubsub.rabbitmq
  version: v1
  metadata:
    - name: host
      value: "amqp://root:123456@rabbitmq:5672"
    - name: durable
      value: "true"
    - name: deletedWhenUnused
      value: "false"
    - name: autoAck
      value: "false"
    - name: deliveryMode
      value: "2"
    - name: requeueInFailure
      value: "false"
    - name: prefetchCount
      value: "5"
    - name: reconnectWait
      value: "0"
    - name: concurrencyMode
      value: parallel
    - name: backOffPolicy
      value: "constant"
    - name: backOffDuration
      value: "3s"
    - name: backOffMaxRetries
      value: 10

subscription.yaml

apiVersion: dapr.io/v1alpha1
kind: Subscription
metadata:
  name: rabbitmq-subscription
spec:
  topic: daprTopic1
  route: /dapr/test_pubsub_sub
  pubsubname: rabbitmq-pubsub

service response

http code: 500
body: {"status": "RETRY"}

logs

svc1-dapr                       | time="2022-10-11T09:18:56.029212918+08:00" level=warning msg="retriable error returned from app while processing pub/sub event fe48034c-ef85-40f2-8d3a-a76108a4f2df, topic: daprTopic1, body: {\"status\":\"RETRY\"}. status code returned: 500" app_id=svc1 instance=43081fb6f22a scope=dapr.runtime type=log ver=edge
svc1-dapr                       | time="2022-10-11T09:18:56.029278084+08:00" level=error msg="rabbitmq pub/sub error: handling message from topic 'daprTopic1', retriable error returned from app while processing pub/sub event fe48034c-ef85-40f2-8d3a-a76108a4f2df, topic: daprTopic1, body: {\"status\":\"RETRY\"}. status code returned: 500" app_id=svc1 instance=43081fb6f22a scope=dapr.contrib type=log ver=edge

from the logs, i think the sidecar knows that it will retry after some time, but i never receive a subscription call again。

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 4
  • Comments: 16 (5 by maintainers)

Most upvoted comments

What is the commitment to retries moving forward in resiliency - Having this logic built-in to pubsub was a big draw to us and we moved our production over to this. Moving to a preview feature for this is concerning for us.

Also, it is working fine. If you’re interested, this is our TerraForm-based dapr deployment with resiliency enabled: https://github.com/microsoft/farmvibes-ai/blob/main/resources/terraform/local/modules/kubernetes/dapr.tf#L88

(We’re using 1.9.6)

Is resiliency still a preview feature? Not seeing it in the list of preview features.

No longer preview since 1.10