sequelize: ETIMEDOUT exception brings down connection to database

We’ve struggled with this issue for quite long, but worked around with the below at the end:

let sequelize = new Sequelize(dbName, dbUsername, dbPassword, {
	host: dbHost,
	...
	retry  : {
		match: [
			/ETIMEDOUT/,
			/EHOSTUNREACH/,
			/ECONNRESET/,
			/ECONNREFUSED/,
			/ETIMEDOUT/,
			/ESOCKETTIMEDOUT/,
			/EHOSTUNREACH/,
			/EPIPE/,
			/EAI_AGAIN/,
			/SequelizeConnectionError/,
			/SequelizeConnectionRefusedError/,
			/SequelizeHostNotFoundError/,
			/SequelizeHostNotReachableError/,
			/SequelizeInvalidConnectionError/,
			/SequelizeConnectionTimedOutError/
		],
		max  : 5
	}
});

I get this error also. I tried to set the min: 1 and idle: 30000 (30 seconds) in the pool settings but it didn’t solve my issue.

Unhandled rejection SequelizeDatabaseError: read ETIMEDOUT
    at Query.formatError (/service/node_modules/sequelize/lib/dialects/postgres/query.js:350:16)
    at Result.query.on.err (/service/node_modules/sequelize/lib/dialects/postgres/query.js:93:21)
    at emitOne (events.js:96:13)
    at Result.emit (events.js:188:7)
    at Result.Query.handleError (/service/node_modules/pg/lib/query.js:163:8)
    at Client.<anonymous> (/service/node_modules/pg/lib/client.js:188:26)
    at emitOne (events.js:96:13)
    at Connection.emit (events.js:188:7)
    at Socket.<anonymous> (/service/node_modules/pg/lib/connection.js:86:10)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at emitErrorNT (net.js:1278:8)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)

@jared-evari You need to update your inbound connections. Go to the databases list in your AWS Console, select the database that you need to update, then do a search on the page for ‘VPC security groups’ (it’s under the ‘Security’ label). You will see the link for the security group under the ‘VPC security groups’. Open that link. In the bottom of the page, select ‘Inbound’ tab. Press ‘Edit’, then ‘Add Rule’. Enter the following values: Type: Custom TCP, Protocol: TCP, Port Range: 0, Source: Custom, IP addresses field: 0.0.0.0/0, ::/0, Description: you can leave this empty. Press ‘Save’.

As @rsshilli said, something was blocking the connection, in my case, i had to allow inbound connections from heroku ip to my rds database in AWS. Also, im not sure if setting DATABASE_URL in Heroku config variable helped

If any of you are seeing this error consistently, you have a firewall problem. I guarantee it. Something, somewhere is stopping the connection. It mostly likely isn’t your Sequelize settings.

With that said, we see this error very rarely, but maybe once every 500 times an AWS Lambda starts up. It only happens when the Lambda is fresh (ie. hasn’t been run in over 20 mins) and needs to make an initial connection. Once the Lambda has started it’ll re-use the old connection and this problem will never happen.

We’re running on Sequelize 3.30.4, Node 6.10.3 and MySQL ^2.13.0. My connection pool settings look like this:

    let sequelize = new Sequelize(dbName, dbUsername, dbPassword, {
      host: dbHost,
      dialect: "mysql",

      pool: {
        max: 5,
        min: 0,
        idle: 1 // Keep this very low or it'll make all Lambda requests take longer
      },
    });

What’s even stranger, is that I have other Lambdas that share the exact same startup code (I mean exactly the same zip, different method) that startup a second or two before I see this error and they run fine. They’re all connecting to the same RDS database, which shows about 5 DB connections at the time (not high at all) nor a high load on the DB. It’s very strange.

Here’s the full stack trace:

2017-11-22T06:57:47.709Z	66946aa0-cf52-11e7-b612-6d9d4421107e	Error received: SequelizeConnectionError: connect ETIMEDOUT SequelizeConnectionError: connect ETIMEDOUT
at Handshake._callback (/var/task/node_modules/sequelize/lib/dialects/mysql/connection-manager.js:95:20)
at Handshake.Sequence.end (/var/task/node_modules/mysql/lib/protocol/sequences/Sequence.js:88:24)
at Protocol.handleNetworkError (/var/task/node_modules/mysql/lib/protocol/Protocol.js:363:14)
at Connection._handleNetworkError (/var/task/node_modules/mysql/lib/Connection.js:428:18)
at Connection._handleConnectTimeout (/var/task/node_modules/mysql/lib/Connection.js:424:8)
at Socket.g (events.js:292:16)
at emitNone (events.js:86:13)
at Socket.emit (events.js:185:7)
at Socket._onTimeout (net.js:338:8)
at ontimeout (timers.js:386:14)
at tryOnTimeout (timers.js:250:5)
at Timer.listOnTimeout (timers.js:214:5)

I suspect there’s something wrong at the connection pool level, but I’m not sure how to diagnose this further. Any ideas?

For people experiencing issues in AWS Lambda, I have created a pull request that documents how to properly configure sequelize for Lambda. Hopefully you’ll be able to see it in https://sequelize.org/master/manual/aws-lambda.html once it gets merged. In the meantime, please check #12642 out and provide feedback if you find any inaccuracies or space for improvement.

In addition to @sharq88 's options above, this one helps too:

  dialectOptions: {
    connectTimeout: 20000, // default is 10s which causes occasional ETIMEDOUT errors (see https://stackoverflow.com/a/52465919/491553)
  },

Here’s the link to the SO page, so you don’t have to copy & paste: https://stackoverflow.com/a/52465919/491553

+1

SequelizeConnectionError: connect ETIMEDOUT
...

My situation was like what @dts-peter said, but with Google Cloud SQL.

same error, on aws ec2. anyone can help?

You can fiddle around here https://github.com/sequelize/sequelize/blob/master/lib/dialects/postgres/connection-manager.js and here: https://github.com/sequelize/sequelize/blob/master/lib/dialects/abstract/connection-manager.js

Specifically you might want to tjek if $destroy is ever called: https://github.com/sequelize/sequelize/blob/master/lib/dialects/abstract/connection-manager.js#L79