dapr: Create/delete reminder throws "possible etag mismatch" after enabling resiliency

In what area(s)?

/area runtime

Ask your question here

We are using dapr actors and our application is very heavy on actor reminders. After enabling resiliency feature we started experiencing a lot of exceptions with error deleting actor reminder: possible etag mismatch when creating and deleting reminders. We also see that these operations write into a single key actors||ActorName||metadata and since we are triggering many reminders from many actors that could cause ETag mismatch if used on state requests:

Dapr.DaprApiException: error deleting actor reminder: possible etag mismatch. 
error from state store: ERR Error running script @user_script:14:  failed to set key actors||ActorName||metadata
 at Dapr.Actors.DaprHttpInteractor.SendAsyncHandleUnsuccessfulResponse(Func`1 requestFunc, String relativeUri, CancellationToken cancellationToken)\n 

Is there any way we could solve this problem and still keep the resiliency feature enabled? Thanks!

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 2
  • Comments: 21 (12 by maintainers)

Commits related to this issue

Most upvoted comments

Update from this morning’s maintainer sync: we are looking at getting this into a hotfix for Dapr 1.10.4. No ETA yet, but likely not this week.

Thanks for sharing the code. It will help investigating further, although I have an idea on what could be causing this and opened #6050 with some initial mitigations.

If storeReminder is not concurrency-safe, is it recommended to use distributed locks before registering reminders?

Let us see how we can fix this in Dapr first. If this is a concurrency error, we need to fix this in Dapr first.

No, my application does not scale horizontally, but has 2 replicas for testing purposes.

Having 2 replicas in this case has the same effect as scaling horizontally and could trigger the other kinds of race conditions (across processes).