node-redis: SocketClosedUnexpectedlyError: Socket closed unexpectedly

Good day! The socket connection closes unexpectedly and doesn’t reconnect, my older version of node-redis(2.8.0) was reconnecting fine and still does. After reconnect it doesn’t print ready anymore. We are using AWS ElastiCache. I can only reproduce this on production env. We have a package that wraps node-redis and is used in all our apps, so I don’t think is related to version difference. I tried to comment out next code and it doesn’t close the connection at all (from @socket.io/redis-adapter).

this.subClient.pSubscribe(this.channel + "*", (msg, channel) => {
    this.onmessage(null, channel, msg);
}, true);

Redis connected
Redis ready!
Redis connected
Redis ready!
fetch error Error: timeout reached while waiting for fetchSockets response
    at Timeout._onTimeout (C:\work\test-socket\node_modules\@socket.io\redis-adapter\dist\index.js:615:28)
    at listOnTimeout (node:internal/timers:568:17)
    at processTimers (node:internal/timers:510:7)
List with sockets connected to hello room: 0
List with sockets connected to hello room: 5
SocketClosedUnexpectedlyError: Socket closed unexpectedly
    at Socket.<anonymous> (C:\work\test-socket\node_modules\@node-redis\client\dist\lib\client\socket.js:195:118)
    at Object.onceWrapper (node:events:510:26)
    at Socket.emit (node:events:390:28)
    at TCP.<anonymous> (node:net:687:12)
Redis reconnecting
Redis connected
ParserError: Protocol error, got "n" as reply type byte
    at handleError (C:\work\test-socket\node_modules\redis-parser\lib\parser.js:190:15)
    at parseType (C:\work\test-socket\node_modules\redis-parser\lib\parser.js:304:14) {
  offset: 304,
  buffer: '....'

Here is where I catch the parser error: client/socket.ts

if (this.#initiator) {
            try {
                await this.#initiator();
            } catch (err) {

List with buffers and offsets:

[{
        offset: 405,
        buffer: '{"type":"Buffer","data":[42,52,13,10,36,56,13,10,112,109,101,115,115,97,103,101,13,10,36,49,50,13,10,105,110,116,101,114,97,99,116,35,47,35,42,13,10,36,54,52,13,10,105,110,116,101,114,97,99,116,35,47,35,104,105,115,82,111,111,109,114,104,122,78,49,49,74,80,98,111,113,107,54,120,78,103,80,52,100,121,71,102,106,84,120,75,86,115,121,98,88,106,107,108,56,56,45,50,66,119,76,65,112,73,99,35,13,10,36,49,53,53,13,10,147,166,119,65,102,98,82,66,131,164,116,121,112,101,2,164,100,97,116,97,146,173,110,111,116,105,102,105,99,97,116,105,111,110,115,130,165,99,111,117,110,116,205,5,184,164,116,121,112,101,181,110,111,116,105,102,105,99,97,116,105,111,110,115,95,99,111,117,110,116,101,114,163,110,115,112,161,47,131,165,114,111,111,109,115,145,217,52,104,105,115,82,111,111,109,114,104,122,78,49,49,74,80,98,111,113,107,54,120,78,103,80,52,100,121,71,102,106,84,120,75,86,115,121,98,88,106,107,108,56,56,45,50,66,119,76,65,112,73,99,166,101,120,99,101,112,116,144,165,102,108,97,103,115,128,13,10,42,52,13,10,36,56,13,10,112,109,101,115,115,97,103,101,13,10,36,49,50,13,10,105,110,116,101,114,97,99,116,35,47,35,42,13,10,36,54,52,13,10,105,110,116,101,114,97,99,116,35,47,35,104,105,115,82,111,111,109,100,76,57,116,118,98,120,51,84,118,52,114,56,87,114,109,56,83,78,89,50,71,53,110,98,80,114,97,112,85,70,116,71,50,107,74,76,116,90,45,50,66,57,105,85,35,13,10,36,49,53,51,13,10,147,166,119,65,102,98,82,66,131,164,116,121,112,101,2,164,100,97,116,97,146,173,110,111,116,105,102,105,99,97,116,105,111,110,115,130,165,99,111,117,110,116,110,164,116,121,112,101,181,110,111,116,105,102,105,99,97,116,105,111,110,115,95,99,111,117,110,116,101,114,163,110,115,112,161,47,131,165,114,111,111,109,115,145,217,52,104,105,115,82,111,111,109,100,76,57,116,118,98,120,51,84,118,52,114,56,87,114,109,56,83,78,89,50,71,53,110,98,80,114,97,112,85,70,116,71,50,107,74,76,116,90,45,50,66,57,105,85,166,101,120,99,101,112,116,144,165,102,108,97,103,115,128,13,10,42,52,13,10,36,56,13,10,112,109,101,115,115,97,103,101,13,10,36,49,50,13,10,105,110,116,101,114,97,99,116,35,47,35,42,13,10,36,54,50,13,10,105,110,116,101,114,97,99,116,35,47,35,104,105,115,82,111,111,109,120,89,112,113,90,54,101,111,89,48,107,120,90,105,81,86,113,97,111,89,90,99,69,119,121,72,50,100,70,122,78,115,108,52,82,50,114,122,111,109,85,113,107,35,13,10,36,49,53,49,13,10,147,166,119,65,102,98,82,66,131,164,116,121,112,101,2,164,100,97,116,97,146,173,110,111,116,105,102,105,99,97,116,105,111,110,115,130,165,99,111,117,110,116,113,164,116,121,112,101,181,110,111,116,105,102,105,99,97,116,105,111,110,115,95,99,111,117,110,116,101,114,163,110,115,112,161,47,131,165,114,111,111,109,115,145,217,50,104,105,115,82,111,111,109,120,89,112,113,90,54,101,111,89,48,107,120,90,105,81,86,113,97,111,89,90,99,69,119,121,72,50,100,70,122,78,115,108,52,82,50,114,122,111,109,85,113,107,166,101,120,99,101,112,116,144,165,102,108,97,103,115,128,13,10,42,52,13,10,36,56,13,10,112,109,101,115,115,97,103,101,13,10,36,49,50,13,10,105,110,116,101,114,97,99,116,35,47,35,42,13,10,36,54,50,13,10,105,110,116,101,114,97,99,116,35,47,35,104,105,115,82,111,111,109,53,111,74,87,66,49,70,85,50,108,109,86,73,54,55,70,106,68,99,83,103,71,118,121,68,48,76,66,83,90,67,83,97,67,113,103,97,69,115,86,119,120,85,35,13,10,36,49,53,50,13,10,147,166,119,65,102,98,82,66,131,164,116,121,112,101,2,164,100,97,116,97,146,173,110,111,116,105,102,105,99,97,116,105,111,110,115,130,165,99,111,117,110,116,204,203,164,116,121,112,101,181,110,111,116,105,102,105,99,97,116,105,111,110,115,95,99,111,117,110,116,101,114,163,110,115,112,161,47,131,165,114,111,111,109,115,145,217,50,104,105,115,82,111,111,109,53,111,74,87,66,49,70,85,50,108,109,86,73,54,55,70,106,68,99,83,103,71,118,121,68,48,76,66,83,90,67,83,97,67,113,103,97,69,115,86,119,120,85,166,101,120,99,101,112,116,144,165,102,108,97,103,115,128,13,10,42,52,13,10,36,56,13,10,112,109,101,115,115,97,103,101,13,10,36,49,50,13,10,105,110,116,101,114,97,99,116,35,47,35,42,13,10,36,54,52,13,10,105,110,116,101,114,97,99,116,35,47,35,104,105,115,82,111,111,109,53,85,74,71,49,53,87,110,116,103,79,88,115,116,106,115,49,45,50,66,82,66,76,106,72,102,82,68,65,109,115,122,112,98,114,82,86,82,69,102,73,50,105,53,85,35,13,10,36,49,53,52,13,10,147,166,119,65,102,98,82,66,131,164,116,121,112,101,2,164,100,97,116,97,146,173,110,111,116,105,102,105,99,97,116,105,111,110,115,130,165,99,111,117,110,116,204,155,164,116,121,112,101,181,110,111,116,105,102,105,99,97,116,105,111,110,115,95,99,111,117,110,116,101,114,163,110,115,112,161,47,131,165,114,111,111,109,115,145,217,52,104,105,115,82,111,111,109,53,85,74,71,49,53,87,110,116,103,79,88,115,116,106,115,49,45,50,66,82,66,76,106,72,102,82,68,65,109,115,122,112,98,114,82,86,82,69,102,73,50,105,53,85,166,101,120,99,101,112,116,144,165,102,108,97,103,115,128,13,10]}'
    },
    {
        offset: 136,
        buffer: '{"type":"Buffer","data":[42,51,13,10,36,57,13,10,115,117,98,115,99,114,105,98,101,13,10,36,49,57,13,10,105,110,116,101,114,97,99,116,45,114,101,113,117,101,115,116,35,47,35,13,10,58,49,13,10,42,51,13,10,36,57,13,10,115,117,98,115,99,114,105,98,101,13,10,36,50,48,13,10,105,110,116,101,114,97,99,116,45,114,101,115,112,111,110,115,101,35,47,35,13,10,58,50,13,10,42,51,13,10,36,57,13,10,115,117,98,115,99,114,105,98,101,13,10,36,50,55,13,10,105,110,116,101,114,97,99,116,45,114,101,115,112,111,110,115,101,35,47,35,88,115,52,75,106,79,35,13,10,58,51,13,10,42,51,13,10,36,57,13,10,115,117,98,115,99,114,105,98,101,13,10,36,50,52,13,10,105,110,116,101,114,97,99,116,45,114,101,113,117,101,115,116,35,47,97,100,109,105,110,35,13,10,58,52,13,10,42,51,13,10,36,57,13,10,115,117,98,115,99,114,105,98,101,13,10,36,50,53,13,10,105,110,116,101,114,97,99,116,45,114,101,115,112,111,110,115,101,35,47,97,100,109,105,110,35,13,10,58,53,13,10,42,51,13,10,36,57,13,10,115,117,98,115,99,114,105,98,101,13,10,36,51,50,13,10,105,110,116,101,114,97,99,116,45,114,101,115,112,111,110,115,101,35,47,97,100,109,105,110,35,104,65,112,66,90,76,35,13,10,58,54,13,10,42,51,13,10,36,49,48,13,10,112,115,117,98,115,99,114,105,98,101,13,10,36,49,50,13,10,105,110,116,101,114,97,99,116,35,47,35,42,13,10,58,55,13,10,42,51,13,10,36,49,48,13,10,112,115,117,98,115,99,114,105,98,101,13,10,36,49,55,13,10,105,110,116,101,114,97,99,116,35,47,97,100,109,105,110,35,42,13,10,58,56,13,10]}'
    },
    {
        offset: 12,
        buffer: '{"type":"Buffer","data":[42,52,13,10,36,56,13,10,112,109,101,115,115,97,103,101,13,10,36,49,50,13,10,105,110,116,101,114,97,99,116,35,47,35,42,13,10,36,54,56,13,10,105,110,116,101,114,97,99,116,35,47,35,104,105,115,82,111,111,109,90,71,86,107,121,121,84,115,98,77,84,45,50,66,111,76,53,69,104,82,106,69,121,81,78,116,48,100,69,57,99,66,52,45,50,70,54,67,90,76,113,81,113,45,50,66,117,81,119,35,13,10,36,49,53,56,13,10,147,166,119,65,102,98,82,66,131,164,116,121,112,101,2,164,100,97,116,97,146,173,110,111,116,105,102,105,99,97,116,105,111,110,115,130,165,99,111,117,110,116,204,131,164,116,121,112,101,181,110,111,116,105,102,105,99,97,116,105,111,110,115,95,99,111,117,110,116,101,114,163,110,115,112,161,47,131,165,114,111,111,109,115,145,217,56,104,105,115,82,111,111,109,90,71,86,107,121,121,84,115,98,77,84,45,50,66,111,76,53,69,104,82,106,69,121,81,78,116,48,100,69,57,99,66,52,45,50,70,54,67,90,76,113,81,113,45,50,66,117,81,119,166,101,120,99,101,112,116,144,165,102,108,97,103,115,128,13,10]}'
    },
    {
        offset: 88,
        buffer: '{"type":"Buffer","data":[42,52,13,10,36,56,13,10,112,109,101,115,115,97,103,101,13,10,36,49,50,13,10,105,110,116,101,114,97,99,116,35,47,35,42,13,10,36,54,56,13,10,105,110,116,101,114,97,99,116,35,47,35,104,105,115,82,111,111,109,89,111,97,115,90,54,72,53,80,45,50,66,71,45,50,66,80,45,50,66,80,49,90,106,82,87,74,87,87,83,54,113,72,74,50,86,120,80,77,68,78,51,117,81,101,120,75,87,81,35,13,10,36,49,53,56,13,10,147,166,104,55,52,78,66,121,131,164,116,121,112,101,2,164,100,97,116,97,146,173,110,111,116,105,102,105,99,97,116,105,111,110,115,130,165,99,111,117,110,116,204,194,164,116,121,112,101,181,110,111,116,105,102,105,99,97,116,105,111,110,115,95,99,111,117,110,116,101,114,163,110,115,112,161,47,131,165,114,111,111,109,115,145,217,56,104,105,115,82,111,111,109,89,111,97,115,90,54,72,53,80,45,50,66,71,45,50,66,80,45,50,66,80,49,90,106,82,87,74,87,87,83,54,113,72,74,50,86,120,80,77,68,78,51,117,81,101,120,75,87,81,166,101,120,99,101,112,116,144,165,102,108,97,103,115,128,13,10]}'
    }]

Code for creating the client

const Redis = require('redis');
Redis.debug_mode = true;

const getClient = async (url) => {

    const client = Redis.createClient({
        socket: {
            host: url
            port: 6379,
        }
    });
    client.on('error', (err) => {
        console.error(err);
    });
    client.on('connect', () => {
        console.log('Redis connected');
    });
    client.on('reconnecting', () => {
        console.log('Redis reconnecting');
    });
    client.on('ready', () => {
        console.log('Redis ready!');
    });

    await client.connect();
    return client;
};

module.exports = {
    getClient,
};

Environment:

Node.js Version: v17.3.1
Redis Server Version: 6.2
Node Redis Version: 4.0.3
Platform: Ubuntu and Windows 10

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 15
Comments: 61 (2 by maintainers)

Commits related to this issue

fix crash if redis disconnects https://github.com/redis/node-redis/issues/2032 — committed to transloadit/uppy by mifi 2 years ago
fix crash if redis disconnects (#3954) https://github.com/redis/node-redis/issues/2032 — committed to transloadit/uppy by mifi 2 years ago
Fixing Redis reconnect It seems Redis randomly fails to reconnect. This seems to be linked to a new behaviour in the Redis 4.x release. See comments here: https://github.com/redis/node-redis/issues/... — committed to workadventure/workadventure by moufmouf 2 years ago
Fixing Redis reconnect It seems Redis randomly fails to reconnect. This seems to be linked to a new behaviour in the Redis 4.x release. See comments here: https://github.com/redis/node-redis/issues/... — committed to workadventure/workadventure by moufmouf 2 years ago
Fixing Redis reconnect It seems Redis randomly fails to reconnect. This seems to be linked to a new behaviour in the Redis 4.x release. See comments here: https://github.com/redis/node-redis/issues/... — committed to workadventure/workadventure by moufmouf 2 years ago
leaving limit.server.ts out until i figure out the redis bug https://github.com/redis/node-redis/issues/2032 — committed to rooteco/tweetscape-streams by ntorba 2 years ago
Add event handlers to redis client If certain event handlers aren't defined on the client before connecting, the app will crash on disconnect event (e.g., timeout). Issue comment regarding issue det... — committed to Shopify/shopify-app-js by mkevinosullivan a year ago
Add event handlers to redis client If certain event handlers aren't defined on the client before connecting, the app will crash on disconnect event (e.g., timeout). Issue comment regarding issue det... — committed to Munavvar-Haydarov/shopify-app-js by Cool-Dev121 a year ago
fix crash if redis disconnects (#3954) https://github.com/redis/node-redis/issues/2032 — committed to docsend/uppy by mifi 2 years ago

Most upvoted comments

@rpong Could you please update the tentative ETA for this issue as it is a blocker for v4?

erajanraja24 on Mar 27, 2022

@fabioselau077 @jorenvandeweyer https://www.digitalocean.com/community/questions/idle-redis-connection-reconnects-every-five-minutes try to set pingInterval to a value lower than 5 minutes (4 * 60 * 1000 should do it)

leibale on Dec 20, 2022

@leibale i just tested 4.1.0 and it seems this issue still persists, reconnecting still doesn’t work like it used to like in v3. after “Socket closed unexpectedly”, it just doesn’t reconnect without restarting the client application. We simulate this by restarting redis (docker) while the application is running.

rpong on May 4, 2022

Just had this occur for us with 4.1.0 running…

Code:

const pubClient = createClient(config);
    const subClient = pubClient.duplicate();

    pubClient.on('connect', () => { chatlogger.error('Redis pub adapter connect'); });
    pubClient.on('ready', () => { chatlogger.error('Redis pub adapter ready'); });
    pubClient.on('end', () => { chatlogger.error('Redis pub adapter end'); });
    pubClient.on('reconnecting', () => { chatlogger.error('Redis pub adapter reconnecting'); });
    pubClient.on('error', (err) => {
        chatlogger.error('Redis pub adapter error', err);
    });

    subClient.on('connect', () => { chatlogger.error('Redis sub adapter connect'); });
    subClient.on('ready', () => { chatlogger.error('Redis sub adapter ready'); });
    subClient.on('end', () => { chatlogger.error('Redis sub adapter end'); });
    subClient.on('reconnecting', () => { chatlogger.error('Redis sub adapter reconnecting'); });
    subClient.on('error', (err) => {
        chatlogger.error('Redis sub adapter error', err);
    });

    await pubClient.connect();
    await subClient.connect();

Our logging:

2022-05-17 13:58:01 error: Redis sub adapter error: SocketClosedUnexpectedlyError: Socket closed unexpectedly
2022-05-17 13:58:01 error: Redis sub adapter reconnecting: ''
2022-05-17 13:58:01 error: Redis sub adapter connect: ''

The ready event didn’t ever happen. When I restarted node it did connect/ready.

Any other ideas/tips? Something I’m missing? This is for a socket.io adaptor so the redis connection going down really messes with the rest of our app.

derikb on May 17, 2022

    client.on('error', err => console.error('client error', err));
    client.on('connect', () => console.log('client is connect'));
    client.on('reconnecting', () => console.log('client is reconnecting'));
    client.on('ready', () => console.log('client is ready'));

Thanks @leibale , i have discovered that this IS working if we do following before the “await client.connect();”

client.on('error', err => console.log('client error', err));
client.on('connect', () => console.log('client is connect'));
client.on('reconnecting', () => console.log('client is reconnecting'));
client.on('ready', () => console.log('client is ready'));

but without the above, it doesn’t reconnect like v3 does. We can upgrade now and just pass debug info for the above, but would like to confirm if this is the intended behavior.

rpong on May 4, 2022

resolved my issue… after switching to ioredis the error message got more specific and I was able to track down the underlying problem (redis required TLS). Disabled TLS on memorystore and it was able to work now.

redis error: ParserError: Protocol error, got "\u0015" as reply type byte. Please report this

djdabs on Jun 6, 2023

I’m facing the same issue. I use Redis version 6.0.

flashtheman on Nov 10, 2022

@CaCaBlocker the only known bug that can cause this issue is the one @ryanmeador pointed out in this comment, and I’m already working to fix it (see this branch).

Other than that, this error could be caused by some kind of a proxy or a load balancer between the redis-server and the node server.

Without a reproduction I won’t be able to debug and hopefully solve it.

leibale on Aug 5, 2022

Getting the same error at v4.6.12 `const redis = require(‘redis’); require(‘dotenv’).config(); const client = redis.createClient({ url: process.env.KV_URL }); (async () => { await client.connect(); })(); client.on(‘connect’, () => { console.log(‘Connected to Redis’); });

client.on(‘error’, (err) => { console.error(Redis Error: ${err}); });

client.on(‘reconnecting’, () => console.log(‘client is reconnecting’)); client.on(‘ready’, () => console.log(‘client is ready’));

module.exports = client;`

This is the error im gettting

ERROR Redis Error: Error: Socket closed unexpectedly INFO client is reconnecting INFO Connected to Redis ERROR Redis Error: Error: Socket closed unexpectedly ERROR Redis Error: Error: Socket closed unexpectedly INFO client is reconnecting INFO Connected to Redis ERROR Redis Error: Error: Socket closed unexpectedly ERROR Redis Error: Error: Socket closed unexpectedly Task timed out after 10.02 seconds

It runs well on local but not when deployed on vercel

shobhitbehl98 on Dec 29, 2023

@niksy https://github.com/redis/node-redis#events

“⚠️ You MUST listen to error events. If a client doesn’t have at least one error listener registered and an error occurs, that error will be thrown and the Node.js process will exit. See the EventEmitter docs for more details.”

leibale on May 21, 2023

@hieuhuynhh a simple setInterval should fix it:

const timer = setInterval(async () => {
  try {
    await client.ping();
  } catch (err) {
    console.error('Ping Interval Error', err);
  }
}, 1000 * 60 * 4);

// make sure to `clearInterval` when you close the client
clearInterval(timer);
client.disconnect();

leibale on May 10, 2023

@HenonoaH try to add listeners to all available events like ready reconnect error etc.

polearnik on Apr 24, 2023

I can confirm that the code above that @leibale provided fixes the issue! My issue in particular was that I had deployed my app using Redis to Heroku and upon the socket closing unexpectedly, the Heroku-deployed app would crash until I redeployed it. The problem was driving me nuts and this was the only solution on the web that fixed it. Thank you!

cheesecakeassassin on May 16, 2022

This code does not reproduce the issue (I’m running docker restart while the code is running)

import { createClient } from '@node-redis/client';

async function client() {
    const client = createClient();

    client.on('error', err => console.error('client error', err));
    client.on('connect', () => console.log('client is connect'));
    client.on('reconnecting', () => console.log('client is reconnecting'));
    client.on('ready', () => console.log('client is ready'));

    await client.connect();

    return client;
}

const [subscriber, publisher] = await Promise.all([
    client(),
    client()
]);

let counter = 0;
subscriber.pSubscribe('*', () => counter++);
setInterval(() => console.log(`Got ${counter} messages!`), 1000);

setInterval(async () => {
    try {
        await publisher.publish('channel', 'message');
    } catch (err) {
        console.error('publish error', err);
    }
}, 5);

client error Error: connect ECONNREFUSED 127.0.0.1:6379
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1146:16) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 6379
}
client is reconnecting
client is reconnecting
client is connect
client is connect
client is ready
Got 484 messages!
Got 484 messages!
client is ready
Got 677 messages!
Got 868 messages!

leibale on Apr 14, 2022