dapr: pubsub retry not working
evnrionment machine: mac M1 docker image: daprio/daprd:edge, build by docker-compose component: rabbitmq
problem
when a service subscribe the topic message and return the status code 500, i expect the dapr sidecar send the message to the service again, until it reach the backOffMaxRetries。
however,dapr sidecar only send the message once, never retry again. that confuse me. maby this is not a bug, just caused by wrong configuration, can any one help me?
configuration pubsub.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: rabbitmq-pubsub
namespace: default
spec:
type: pubsub.rabbitmq
version: v1
metadata:
- name: host
value: "amqp://root:123456@rabbitmq:5672"
- name: durable
value: "true"
- name: deletedWhenUnused
value: "false"
- name: autoAck
value: "false"
- name: deliveryMode
value: "2"
- name: requeueInFailure
value: "false"
- name: prefetchCount
value: "5"
- name: reconnectWait
value: "0"
- name: concurrencyMode
value: parallel
- name: backOffPolicy
value: "constant"
- name: backOffDuration
value: "3s"
- name: backOffMaxRetries
value: 10
subscription.yaml
apiVersion: dapr.io/v1alpha1
kind: Subscription
metadata:
name: rabbitmq-subscription
spec:
topic: daprTopic1
route: /dapr/test_pubsub_sub
pubsubname: rabbitmq-pubsub
service response
http code: 500
body: {"status": "RETRY"}
logs
svc1-dapr | time="2022-10-11T09:18:56.029212918+08:00" level=warning msg="retriable error returned from app while processing pub/sub event fe48034c-ef85-40f2-8d3a-a76108a4f2df, topic: daprTopic1, body: {\"status\":\"RETRY\"}. status code returned: 500" app_id=svc1 instance=43081fb6f22a scope=dapr.runtime type=log ver=edge
svc1-dapr | time="2022-10-11T09:18:56.029278084+08:00" level=error msg="rabbitmq pub/sub error: handling message from topic 'daprTopic1', retriable error returned from app while processing pub/sub event fe48034c-ef85-40f2-8d3a-a76108a4f2df, topic: daprTopic1, body: {\"status\":\"RETRY\"}. status code returned: 500" app_id=svc1 instance=43081fb6f22a scope=dapr.contrib type=log ver=edge
from the logs, i think the sidecar knows that it will retry after some time, but i never receive a subscription call again。
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 4
- Comments: 16 (5 by maintainers)
What is the commitment to retries moving forward in resiliency - Having this logic built-in to pubsub was a big draw to us and we moved our production over to this. Moving to a preview feature for this is concerning for us.
Also, it is working fine. If you’re interested, this is our TerraForm-based dapr deployment with resiliency enabled: https://github.com/microsoft/farmvibes-ai/blob/main/resources/terraform/local/modules/kubernetes/dapr.tf#L88
(We’re using 1.9.6)
No longer preview since 1.10