node-redlock: LockError: Unable to fully release the lock on resource

I finally found out the problem and I resolved it myself. I will have some plan to use another component in order to get rid of such issues.

I now also see the bug in 4.2.0, I want to know which version should I use to avoid this bug?

Now, perhaps this isn’t a scenario that you would consider an error (and initially I didn’t either, given this library’s earlier behavior). However, given that the purpose of this library is to guarantee exclusivity of a particular resource, I would consider it an error where the specified expiration breaks this guarantee.

@mike-marcacci could the error returned get more details? Right now it says Unable to fully release the lock on resource but if it’s due to the resource being gone (or something else) it’d be great if the error said so instead of a more generic “failed”.

(mostly asking since we’re seeing these errors now when we didn’t before, and more context on what is failing would help us narrow down the issue 🙂)

I only increased the lock time to 60000 milliseconds and it is now working fine.

Same issue when upgrading to the latest.

@mike-marcacci Here is a script for the bug reproduction:

'use strict';

const Redis = require('ioredis');
const Redlock = require('redlock');

const TIMEOUT_SEC = 2000;
const ADDITIONAL_TIMEOUT = 100;
const KEY = 'key';

const redis = new Redis();
const redlock = new Redlock([redis]);

redlock.lock(KEY, TIMEOUT_SEC)
    .then(lock => {
        console.log(`resource is locked, key=[${KEY}]`);

        setTimeout(() => {
            lock.unlock()
                .then(() => {
                    console.log(`resource is unlocked, key=[${KEY}]`);
                    process.exit(0);
                })
                .catch(error => {
                    console.log(`error on resource unlocking, error=[${error}]`);
                    process.exit(1);
                })
            
        }, TIMEOUT_SEC + ADDITIONAL_TIMEOUT);
    });

You can try run it with different library tags and make sure it works for 3.1.2 but not for 4.1.0.

I personally used Redis server version 3.2.12 into a docker container and ioredis 4.14.2.

My suggestion here is that you have broken that functionality by changing Lua scripts for locking and unlocking.

Seems like a critical bug since an application code throws an error on an attempt to unlock a resource after a timeout has exceeded.

This error occurs when we attempt to release an already expired lock.

A condition like below while releasing the lock would help

if (lock.expiration < Date.now() && lock.expiration != 0) {
  lock.unlock()
}

The only input I can provide is that, you should not simply increase the lock/unlock time/timeout blindly.

This is not really a solution and it can only mask other problems of the library or your system. If there is something wrong with your system, this can result in lock time exceptions. Either acquiring or releasing. For example, if your lengthy processes take more time than the total lock acquiring time, then any other attempt to acquire the lock will expire and throw an error.

Such cases should be considered as false positives as regards to this issue I initially opened. As they are related with misconfiguration or bugs of the system using the lock mechanism.

For my use-case, the system was tested; with each operation reaching a total lock time of 1-2 seconds as a maximum. After upgrading we started hitting the error, consistently. The only workaround so far was to roll back to redlock version 3.1.2.

I believe that there is something wrong with the changes introduced here

For me this goes away if going back to 3.1.2. But having a look at the timing it appears that this “new” error message pops up when the lock expired. Eg. if locking for 250 ms and the actual processing while the lock is acquired takes 300 ms an error gets thrown. Maybe this was enforced with 4.0.0 and before was handled silently?

@gnitsenko93 perhaps I’m missing something here (I’m still on my first coffee), but in your reproduction the lock has already expired and the keys have been evicted by redis… so calling unlock should error. This way your application can alert you to the fact that the lock expired and became re-acquirable before you unlocked it.

If this is the scenario that is being described in this issue, this is expected behavior in v4 as described in CHANGELOG.md. This would make sense given my inability to reproduce an issue that folks are clearly seeing in the wild.

Now, perhaps this isn’t a scenario that you would consider an error (and initially I didn’t either, given this library’s earlier behavior). However, given that the purpose of this library is to guarantee exclusivity of a particular resource, I would consider it an error where the specified expiration breaks this guarantee.

Hey folks, sorry for the slow replies: I’ve been unbelievably busy this year so far. I think that @b00tsy may be onto something here, especially given @amirzandi’s workaround.

Version 3 should have rejected with the same error, but perhaps it wasn’t? Unlocking was originally a best-effort operation that wouldn’t report an error, but that was changed quite early on.

For those of you who have reverted to version 3, can you watch your redis instances for key expirations? Something like this:

config set notify-keyspace-events Ex
psubscribe __keyevent@0__:expired

If all is well and you are acquiring or extending locks with adequate durations, you shouldn’t see any keys that match your locks expiring.