node-redis: SocketClosedUnexpectedlyError - Unhandled Error

Description

Hi,

We are experiencing issue with the client. Many times a day we have this unhandled error that is thrown away from the redis client.

We have properly set the “Error” handler but it is not catched in it. As a consequence our process is crashing multiple times per day. It is severly impacting our customers.

We already dig into the issue list where this error was mentionned, we tried the PingInterval / SetInterval stuff as we are on Azure but it doesn’t fix the issue.

I suspect this is another edge case than the load balancer that drop the connection. If you can see how to catch properly this error so the reconnect system can reopen smoothlty the connection it will save our day.

Thanks,

Node.js Version

Node.js v18.17.1

Redis Server Version

Node.js v18.17.1

Node Redis Version

No response

Platform

Linux

Logs

2023-10-04T09:34:16.491974249Z node:events:492
2023-10-04T09:34:16.492029849Z       throw er; // Unhandled 'error' event
2023-10-04T09:34:16.492037249Z       ^
2023-10-04T09:34:16.492041649Z 
2023-10-04T09:34:16.492045549Z SocketClosedUnexpectedlyError: Socket closed unexpectedly
2023-10-04T09:34:16.492049849Z     at Socket.<anonymous> (/home/site/wwwroot/node_modules/@redis/client/dist/lib/client/socket.js:194:118)
2023-10-04T09:34:16.492054649Z     at Object.onceWrapper (node:events:629:26)
2023-10-04T09:34:16.492058649Z     at Socket.emit (node:events:514:28)
2023-10-04T09:34:16.492062649Z     at TCP.<anonymous> (node:net:323:12)
2023-10-04T09:34:16.492066948Z Emitted 'error' event on Commander instance at:
2023-10-04T09:34:16.492071048Z     at RedisSocket.<anonymous> (/home/site/wwwroot/node_modules/@redis/client/dist/lib/client/index.js:396:14)
2023-10-04T09:34:16.492075148Z     at RedisSocket.emit (node:events:514:28)
2023-10-04T09:34:16.492082148Z     at RedisSocket._RedisSocket_onSocketError (/home/site/wwwroot/node_modules/@redis/client/dist/lib/client/socket.js:217:10)
2023-10-04T09:34:16.492086248Z     at Socket.<anonymous> (/home/site/wwwroot/node_modules/@redis/client/dist/lib/client/socket.js:194:107)
2023-10-04T09:34:16.492090248Z     at Object.onceWrapper (node:events:629:26)
2023-10-04T09:34:16.492094048Z     at Socket.emit (node:events:514:28)
2023-10-04T09:34:16.492097948Z     at TCP.<anonymous> (node:net:323:12)

About this issue

  • Original URL
  • State: open
  • Created 9 months ago
  • Comments: 16

Most upvoted comments

@TDola its fine not awaiting a promise, but you’ll have to catch the errors, otherwise it’ll crash with “unhandled promise rejection” error…

I tried reproducing it using

import { createClient } from 'redis';

const client = await createClient()
  .on('error', (err) => console.log('client error', err))
  .on('ready', () => console.log('client is ready'))
  .connect();

function createKeys() {
  for (let i = 0; i < 10000; i++) {
    client.set(`text-key${i.toString()}`, i.toString()).catch(err => {
      console.error(`Error in SET ${i}`, err);
    });
  }
}

async function removeKeys() {
  for (const key of await client.keys('test-key:*')) {
    client.del(key).catch(err => {
      console.error(`Error in DEL ${key}`, err);
    });
  }
}

function runner() {
  console.log('starting runner loop');
  createKeys();

  console.log('done setting keys');
  removeKeys();

  console.log('done deleting keys, waiting 1000');
  setTimeout(runner, 1000);
}

runner();

but… it works…

The application(nest.js) is hosted on render, and this error is often reported. Redis uses the vercel kv service.

CacheModule.registerAsync({
      isGlobal: true,
      useFactory: async () => {
        const store = await redisStore({
          socket: {
            host: process.env.KV_HOST,
            port: +process.env.KV_PORT,
            tls: true,
          },
          username: process.env.KV_USERNAME,
          password: process.env.KV_PASSWORD,
          ttl: 60,
        })
        return {
          store: store as unknown as CacheStore,
        }
      },
    }),
    ThrottlerModule.forRootAsync({
      useFactory: () => ({
        throttlers: [{ limit: 10, ttl: seconds(60) }],
        storage: new ThrottlerStorageRedisService({
          host: process.env.KV_HOST,
          port: +process.env.KV_PORT,
          username: process.env.KV_USERNAME,
          password: process.env.KV_PASSWORD,
          tls: true,
          maxRetriesPerRequest: 20,
        } as unknown as RedisOptions),
      }),
    }),
node:events:495
      throw er; // Unhandled 'error' event
      ^
SocketClosedUnexpectedlyError: Socket closed unexpectedly
    at TLSSocket.<anonymous> (/opt/render/project/src/node/node_modules/.pnpm/@redis+client@1.5.14/node_modules/@redis/client/dist/lib/client/socket.js:194:118)
    at Object.onceWrapper (node:events:632:26)
    at TLSSocket.emit (node:events:529:35)
    at node:net:350:12
    at TCP.done (node:_tls_wrap:657:7)
Emitted 'error' event on Commander instance at:
    at RedisSocket.<anonymous> (/opt/render/project/src/node/node_modules/.pnpm/@redis+client@1.5.14/node_modules/@redis/client/dist/lib/client/index.js:412:14)
==> Requesting node version 18
==> Using Node version 18.19.1 via /opt/render/project/src/node/.nvmrc
==> Docs on specifying a Node version: https://render.com/docs/node-version
==> Running 'pnpm start'
> homing-pigeon-workspace@ start /opt/render/project/src/node
> pnpm --filter server start
> server@1.0.0 start /opt/render/project/src/node/server
> cross-env NODE_ENV=production node dist/main
    at RedisSocket.emit (node:events:517:28)
    at RedisSocket._RedisSocket_onSocketError (/opt/render/project/src/node/node_modules/.pnpm/@redis+client@1.5.14/node_modules/@redis/client/dist/lib/client/socket.js:218:10)
    at TLSSocket.<anonymous> (/opt/render/project/src/node/node_modules/.pnpm/@redis+client@1.5.14/node_modules/@redis/client/dist/lib/client/socket.js:194:107)
    at Object.onceWrapper (node:events:632:26)
    [... lines matching original stack trace ...]
    at TCP.done (node:_tls_wrap:657:7)
Node.js v18.19.1
/opt/render/project/src/node/server:
 ERR_PNPM_RECURSIVE_RUN_FIRST_FAIL  server@1.0.0 start: `cross-env NODE_ENV=production node dist/main`
Exit status 1
 ELIFECYCLE  Command failed with exit code 1.

@TDola I’ve fixed the example, please LMK if you find anything…

@bossajie are you listening to errors on the cluster? i.e.:

import { createCluster } from 'redis';

const cluster = await createCluster(...)
  .on('error', err => console.error('Redis Cluster Error', err)
  .connect();

In my case, it is not catch inside the error handler 😦

If I shutdown the redis server, the error is properly managed: 2023-10-06T13:18:21.252432723Z Redis error: Error: read ECONNRESET 2023-10-06T13:18:21.252493923Z at TCP.onStreamRead (node:internal/stream_base_commons:217:20) { 2023-10-06T13:18:21.252499722Z errno: -104, 2023-10-06T13:18:21.252502922Z code: ‘ECONNRESET’, 2023-10-06T13:18:21.252505922Z syscall: ‘read’ 2023-10-06T13:18:21.252508922Z } 2023-10-06T13:18:21.252566422Z client is reconnecting

@leibale Any tips ? Is there a possibility that it happens when 2 command are sent at the same time? thanks,