mysql: Quit inactivity timeout : Lambda / node 8.10

After looking around, I see this was a bug others have bumped into, but a claim that it was the fault of node 4 and all you needed to do was to upgrade. This however does not help.

I am running on AWS Lambda with node 8.10 and every so often I get the following in the log files. It is intermittent with no specific reason. I am opening a connection and closing it down on a ‘finally’ clause if that helps.

2018-08-21T09:40:45.331Z	46fbd76e-a526-11e8-b235-4de84692bdfc	Unhandled rejection Error: Quit inactivity timeout
at Quit.<anonymous> (/var/task/node_modules/promise-mysql/node_modules/mysql/lib/protocol/Protocol.js:163:17)
at emitNone (events.js:106:13)
at Quit.emit (events.js:208:7)
at Quit._onTimeout (/var/task/node_modules/promise-mysql/node_modules/mysql/lib/protocol/sequences/Sequence.js:124:8)
at Timer._onTimeout (/var/task/node_modules/promise-mysql/node_modules/mysql/lib/protocol/Timer.js:32:23)
at ontimeout (timers.js:482:11)
at tryOnTimeout (timers.js:317:5)
at Timer.listOnTimeout (timers.js:277:5)
--------------------
at Protocol._enqueue (/var/task/node_modules/promise-mysql/node_modules/mysql/lib/protocol/Protocol.js:144:48)
at Protocol.quit (/var/task/node_modules/promise-mysql/node_modules/mysql/lib/protocol/Protocol.js:91:23)
at Connection.end (/var/task/node_modules/promise-mysql/node_modules/mysql/lib/Connection.js:241:18)
at /var/task/node_modules/promise-mysql/lib/helper.js:16:24
at Promise._execute (/var/task/node_modules/bluebird/js/release/debuggability.js:303:9)
at Promise._resolveFromExecutor (/var/task/node_modules/bluebird/js/release/promise.js:483:18)
at new Promise (/var/task/node_modules/bluebird/js/release/promise.js:79:10)
at Connection.promiseCallback (/var/task/node_modules/promise-mysql/lib/helper.js:6:10)
at connection.end (/var/task/node_modules/promise-mysql/lib/connection.js:96:28)
at Object.list [as GET] (/var/task/api/console/solicitation.js:167:14)
at <anonymous>

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 3
  • Comments: 19 (3 by maintainers)

Most upvoted comments

So for anyone who runs into this issue, later on, I think I may have figured it out?

So a few things I noticed. The setTimout timer for conn.end is never cleared. So as long as the lambda is warm the timer is still going as apparently lambda doesn’t clear its execution context unless the lambda is destroyed/not warm. So when that lambda is reused if X amount of seconds (30 is the default or whatever you passed into the timeout option for conn.end) has passed since that lambda has been hit, when it is reused it emits that error first THEN starts executing your handler code.

This article explains it better (number 19 in particular): https://medium.com/radient-tech-blog/aws-lambda-and-the-node-js-event-loop-864e48fba49

The amazon documentation states: https://docs.aws.amazon.com/lambda/latest/dg/running-lambda-code.html “Background processes or callbacks initiated by your Lambda function that did not complete when the function ended resume if AWS Lambda chooses to reuse the execution context. You should make sure any background processes or callbacks in your code are complete before the code exits.”

Since the lambda function exited before the callback for setTimeout was called I’m guessing it resumes once that same lambda is executed again if it decided to reuse the execution context.

Hopefully, this helps? I’m currently trying to figure out the best solution for this as how long lambda’s stay alive varies…

I’m on AWS Lambda with Node 8.10 and I’ve randomly been having this error, but prompted by @wips I did some experimenting, and I think I solved the issue simply by including a function(err){…} in .end(). Like:

connection.end(function(err) {console.log("Database connection end error is " + err )});

It seems that simply handling the error in that way is all the was needed- otherwise, my Lambda function would randomly crash no matter how I tried to catch the error. With this statement included, the queries run fine, even if I also pass in {timeout : 1}, which by itself consistently causes the error to appear.

@a1anw2, as @dougwilson said .end() is non-blocking, but this may lead your code to the same issue that was in mine - I was just calling .end() w/o waiting the operation to actually complete.

To actually wait for the connection end you can’t just await dbConn.end(), because .end() doesn’t return a promise. What you need is to create a promise and return it. Smth like the following:

return new Promise((resolve, reject) => {
    dbConn.end(error => error ? reject(error) : resolve());
});

Btw, the same thing is with await dbConn.query(sqlStat.sql, sqlStat.params) from your code snippet - .query() doesn’t return a promise, so there’s no point in await-ing it.

Had this same issue because I was working with a slow internet connection. So it frequently timed out. Increasing the timeout value worked for me.

...
const mysql = require('mysql');

mysql.createConnection({
        host: dbParams.HOST,
        user: dbParams.USER,
        password: dbParams.PASSWORD,
        database: dbParams.DATABASE,
        timeout: 600000  // <-------- HERE---------
});
...

I didn’t include the require modules … i am using https://www.npmjs.com/package/promise-mysql that wraps all the calls in a promise.

Thanks @dougwilson .

Looks like .destroy() is probably worth calling instead of .end() as I re-read this in the cold light of day.