bullmq: [Bug]: moveToFailed throws an exception when using Elasticache serverless

Version

5.12.0

Platform

NodeJS

What happened?

We are using Elasticache Serverless instance (redis v7.1) When adding a job to queue and the job fails an exception is thrown -

ReplyError: EXECABORT Transaction discarded because of previous errors.
    at parseError (bull-monitor/node_modules/redis-parser/lib/parser.js:179:12)
    at parseType (bull-monitor/node_modules/redis-parser/lib/parser.js:302:14) {
  command: { name: 'exec', args: [] },
  previousErrors: [
    ReplyError: ERR command not supported inside transaction
        at parseError (bull-monitor/node_modules/redis-parser/lib/parser.js:179:12)
        at parseType (bull-monitor/node_modules/redis-parser/lib/parser.js:302:14) {
      command: [Object]
    },
    ReplyError: ERR command not supported inside transaction
        at parseError (bull-monitor/node_modules/redis-parser/lib/parser.js:179:12)
        at parseType (bull-monitor/node_modules/redis-parser/lib/parser.js:302:14) {
      command: [Object]
    }
  ]
}

How to reproduce.

replace some-serverless-host with a relevant redis instance

import {Worker, Queue, UnrecoverableError} from 'bullmq';
import Redis from 'ioredis';

const clusterQueue = new Queue('test-queue', {
    prefix: '{bullMQ}',
    connection: new Redis.Cluster([
        {host: 'some-serverless-host', port: 6379},
    ], {
        dnsLookup: (address, callback) => callback(null, address),
        redisOptions: {
            tls: true,
        }
    })
})

export async function renderQueue() {
    await clusterQueue.add('name:some-name', 'some-job-data')
}

const WorkerQueue = new Worker('test-queue', async (job) => {
    throw new UnrecoverableError('test cluster exception')
}, {
    connection: new Redis.Cluster([
        {host: 'some-serverless-host', port: 6379},
    ], {
        dnsLookup: (address, callback) => callback(null, address),
        redisOptions: {
            tls: true,
        }
    }),
    prefix: '{bullMQ}'
})

WorkerQueue.on('waiting', () => console.log('waiting completed'))
WorkerQueue.on('completed', () => console.log('jobs completed'))
WorkerQueue.on('failed', () => console.log('failed completed'))

Relevant log output

ReplyError: EXECABORT Transaction discarded because of previous errors.
    at parseError (bull-monitor/node_modules/redis-parser/lib/parser.js:179:12)
    at parseType (bull-monitor/node_modules/redis-parser/lib/parser.js:302:14) {
  command: { name: 'exec', args: [] },
  previousErrors: [
    ReplyError: ERR command not supported inside transaction
        at parseError (bull-monitor/node_modules/redis-parser/lib/parser.js:179:12)
        at parseType (bull-monitor/node_modules/redis-parser/lib/parser.js:302:14) {
      command: [Object]
    },
    ReplyError: ERR command not supported inside transaction
        at parseError (bull-monitor/node_modules/redis-parser/lib/parser.js:179:12)
        at parseType (bull-monitor/node_modules/redis-parser/lib/parser.js:302:14) {
      command: [Object]
    }
  ]
}


### Code of Conduct

- [X] I agree to follow this project's Code of Conduct

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 4
  • Comments: 29 (10 by maintainers)

Most upvoted comments

@bowenzhou222 EVAL and EVALSHA works, but what does not work is calling these commands within a multi/exec Redis transaction. However I cannot find where this is stated, nor where it could be reported so that they could implement it in the future. For now I am trying to eliminate the use of multi + eval in the most used code paths of BullMQ, but there will be some features that will not work as they are too complicated to fix, such as flows and adding jobs in bulks.

hi @Omers-Frontegg we are also working on removing multi usage when upserting a job scheduler. It would take 1 or 2 weeks of work still. An alternative would be to use v5.39.2.

Should be fixed from version 7.26.5.

Good to know, we should support it.

@Omers-Frontegg @madolson even though we may succeed removing multi/eval for upsertJobScheduler (which is not trivial to do either by the way), we will not be able to remove all multi/evals we have in the rest of the codebase. So in general I cannot recommend using Elasticache Serverless to be used with BullMQ at the time being, as we cannot guarantee that it will work stable enough. Until we can run the whole test suite in serverless you should not use BullMQ in production with this Redis alternative. You could on the other hand use standard Elasticache non serverless version, or Upstash as a serverless solution that does support multi/eval.

@madolson Yep, here is the caught error.

Image

@Omers-Frontegg This is something we will need to fix on our (AWS) side. Can you validate it’s the same error that manast mentioned above with the ERR command not supported inside transaction.

@tobiasviehweger I will look into this asap or @roggervalf if you are faster than me 😃

@manast Yeah, we are looking into it on our end. We manually marked a bunch of commands as blocked for multi that cause issues with consistency, specifically EVALSHA is problematic for us, and some other reasons like head of line blocking. We didn’t find any issue yet with EVAL though, so it might have just been an oversight for us to block it. We have someone looking into fixing it.

@tobiasviehweger yes, I am not sure if this is the same because there are other places where we combine MULTI with EVAL, such as in batches, flows, schedulers, so depending on what you are using you can trigger this error.

@madolson I am pinging you in case you are not following this issue yet 😃