orleans: When does OrleansBrokenTransactionLockException occur?
Currently I am attempting to perform a transaction while in a timer. Occasionally i would get a broken lock exception with the following stack trace
[13:31:04 WRN] Break-lock timeout for transactions 5882be80-e0d3-47ef-a25e-e55363fb01b6. 3ms late
[13:31:04 WRN] Break-lock timeout for transactions 4b4acd05-0f33-40fa-8627-bf5ddf5ba34f. 2ms late
[13:31:04 WRN] Break-lock timeout for transactions 5882be80-e0d3-47ef-a25e-e55363fb01b6. 3ms late
[13:31:04 ERR] Caught and ignored exception: Orleans.Transactions.OrleansBrokenTransactionLockException with message: Transaction 4b4acd05-0f33-40fa-8627-bf5ddf5ba34f aborted because a broken lock was detected, when trying to re-enter lock thrown from timer callback GrainTimer. TimerCallbackHandler:MyGrainWithTimer->System.Threading.Tasks.Task <ReceiveReminder>b__10_0(System.Object)
Orleans.Transactions.OrleansBrokenTransactionLockException: Transaction 4b4acd05-0f33-40fa-8627-bf5ddf5ba34f aborted because a broken lock was detected, when trying to re-enter lock
at Orleans.OrleansTaskExtentions.<ToTypedTask>g__ConvertAsync|4_0[T](Task`1 asyncTask)
I have a setup using some sort of reduce pattern, where I have subtotal grains that have timers to transfer amount into the total grain using a transaction to perform the transfer. Seems like on a random chance, a broken lock exception will occur, which will cause the transaction to fail
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 16 (8 by maintainers)
Broken locks typically occur if a transaction takes too long to complete. The time from when a grain is accessed in a transaction until the transaction completes cannot exceed
TransactionalStateOptions.LockTimeout, which is 8 seconds by default.Maybe
broken lockshould have been calledlock timeout.Why would a transaction take so long?
Perhaps your system has very high latencies, runs very long transactions, or is overloaded. In that case transactions simply don’t work. You will need workflows, or sagas, or something like that.
Another (and more easily fixable) reason of why locks time out is that there may be deadlocks going on: If you have multiple transactions that are accessing the same grains they can get into a deadlock situation (each transaction holds some locks that another transaction needs). For example, if you have grains A and B, and transaction T1 accesses A first, then B, while transaction T2 accesses B first, then A, you can get into the situation where T1 is waiting for T2 to release the lock on B, while T2 is waiting for T1 to release the lock on A. This deadlock then causes the locks to time out.
Deadlocks can be avoided by designing transactions so they follow a consistent ordering when accessing grain. For example, say you have three grains A, B, C. By enforcing that the order of grain access inside a transaction follows alphabetic order, you can avoid deadlocks and the corresponding broken locks.