dapr: Create/delete reminder throws "possible etag mismatch" after enabling resiliency
In what area(s)?
/area runtime
Ask your question here
We are using dapr actors and our application is very heavy on actor reminders. After enabling resiliency feature we started experiencing a lot of exceptions with error deleting actor reminder: possible etag mismatch when creating and deleting reminders. We also see that these operations write into a single key actors||ActorName||metadata and since we are triggering many reminders from many actors that could cause ETag mismatch if used on state requests:
Dapr.DaprApiException: error deleting actor reminder: possible etag mismatch.
error from state store: ERR Error running script @user_script:14: failed to set key actors||ActorName||metadata
at Dapr.Actors.DaprHttpInteractor.SendAsyncHandleUnsuccessfulResponse(Func`1 requestFunc, String relativeUri, CancellationToken cancellationToken)\n
Is there any way we could solve this problem and still keep the resiliency feature enabled? Thanks!
About this issue
- Original URL
- State: closed
- Created a year ago
- Reactions: 2
- Comments: 21 (12 by maintainers)
Commits related to this issue
- Cascading contexts in actor reminder state store methods Also increased information returned in errors See: #6018 Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com> — committed to ItalyPaleAle/dapr by ItalyPaleAle a year ago
- Cascading contexts in actor reminder state store methods Also increased information returned in errors See: #6018 Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com> — committed to ItalyPaleAle/dapr by ItalyPaleAle a year ago
- Use a transaction to store reminder partition and metadata See #6018 Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com> — committed to ItalyPaleAle/dapr by ItalyPaleAle a year ago
- Actor reminders improvements Fixes race conditions for the most part (fixes #6018) Includes improvements to perf: - Update actor reminders data structures - Switch data field to be json.RawMessage ... — committed to ItalyPaleAle/dapr by ItalyPaleAle a year ago
- Actor reminders improvements Fixes race conditions for the most part (fixes #6018) Includes improvements to perf: - Update actor reminders data structures - Switch data field to be json.RawMessage ... — committed to ItalyPaleAle/dapr by ItalyPaleAle a year ago
- Actor reminders improvements (#6039) Fixes race conditions for the most part (fixes #6018) Includes improvements to perf: - Update actor reminders data structures - Switch data field to be jso... — committed to dapr/dapr by ItalyPaleAle a year ago
- [release-1.10] Use a lock around all operations to store reminders Should fix the remaining race conditions causing operations to fail (even if without data loss) Fixes #6018 (whatever is left) Sig... — committed to ItalyPaleAle/dapr by ItalyPaleAle a year ago
- [release-1.10] Use a lock around all operations to store reminders Should fix the remaining race conditions causing operations to fail (even if without data loss) Fixes #6018 (whatever is left) Sig... — committed to ItalyPaleAle/dapr by ItalyPaleAle a year ago
- Squashed commit of the following: commit 5c060c1741c476e457e56a34ecc857273a09a265 Merge: ba4d618a 770d4e51 Author: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com> Date: Thu Ma... — committed to artursouza/dapr by artursouza a year ago
- Squashed commit of the following: commit 5c060c1741c476e457e56a34ecc857273a09a265 Merge: ba4d618a 770d4e51 Author: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com> Date: Thu Ma... — committed to artursouza/dapr by artursouza a year ago
- Squashed commit of the following: commit 5c060c1741c476e457e56a34ecc857273a09a265 Merge: ba4d618a 770d4e51 Author: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com> Date: Thu Ma... — committed to artursouza/dapr by artursouza a year ago
- Squashed commit of the following: commit 5c060c1741c476e457e56a34ecc857273a09a265 Merge: ba4d618a 770d4e51 Author: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com> Date: Thu Ma... — committed to artursouza/dapr by artursouza a year ago
- Squashed commit of the following: commit 5c060c1741c476e457e56a34ecc857273a09a265 Merge: ba4d618a 770d4e51 Author: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com> Date: Thu Ma... — committed to artursouza/dapr by artursouza a year ago
- Squashed commit of the following: commit 5c060c1741c476e457e56a34ecc857273a09a265 Merge: ba4d618a 770d4e51 Author: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com> Date: Thu Ma... — committed to artursouza/dapr by artursouza a year ago
- E2E test to check eTag metadata race + fix (#6325) * E2E test to check eTag metadata race. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Try with 1.9.6 sidecar image. Signed-off-by: Ar... — committed to dapr/dapr by artursouza a year ago
Update from this morning’s maintainer sync: we are looking at getting this into a hotfix for Dapr 1.10.4. No ETA yet, but likely not this week.
Thanks for sharing the code. It will help investigating further, although I have an idea on what could be causing this and opened #6050 with some initial mitigations.
Let us see how we can fix this in Dapr first. If this is a concurrency error, we need to fix this in Dapr first.
Having 2 replicas in this case has the same effect as scaling horizontally and could trigger the other kinds of race conditions (across processes).