orleans: InconsistentStateException with ADO.NET SQL Server grain storage
We’ve been getting this exception for a while in various grains. Particularly one which is activated in a StartupTask
that should never be deactivated. If I’ve understood correctly this can happen when there are duplicate activations of a grain (this happens perhaps during deployments) - so question is what do you do when that happens? Do you have to catch it and DeactivateOnIdle
? Do you call ReadStateAsync
and lose that data? We had a grain that during the lifetime of the activation never again could write to state so I’m wondering how you recover from it. I couldn’t find info on this in docs, maybe I missed it
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 1
- Comments: 37 (30 by maintainers)
They are. Timers are the only interleaving bug that has become a feature that some people now depend on.
Thanks for your suggestions, I’ll keep monitoring the Silo closely so far the trick was done using my own store to keep state and moving all grains to stateless, I’ll keep it that way in the meantime.
Thanks again for you time and help. Have an excellent day.
Sent from my Email account 😉.
Interesting, that definitely solves one of my issues, thanks. I would think stream messages are queued like normal, if not I have som refactoring to do
Timers execute interleaved. Not sure about streams.
I also wanna add that this issue has been discussed before. #2565 A possible workaround via self-invocation for timers was posted here: #2574
Recently I had to deal with this issue as well. For me the trouble came from interleaving code in stateful grains. In particular orleans timers can be a source of this problem.
As soon as 2 calls end up on WriteStateAsync() at the same time, you have a good chance for this error to happen.
In my opinion ETags should prevent different activations from writing to the same state but the current design doesn’t even allow the same grain to write multiple times.